2024-02-05T04:40:07.591 INFO:root:teuthology version: 0.0.1.dev256+gd9fdb22 2024-02-05T04:40:07.591 DEBUG:teuthology.run:Teuthology command: teuthology --description rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} --owner scheduled_nmordech@teuthology --name nmordech-2024-01-31_07:15:30-rados-wip-nitzan-max-pglog-size-distro-default-smithi --archive /home/teuthworker/archive/nmordech-2024-01-31_07:15:30-rados-wip-nitzan-max-pglog-size-distro-default-smithi/7539667 --verbose -- /home/teuthworker/archive/nmordech-2024-01-31_07:15:30-rados-wip-nitzan-max-pglog-size-distro-default-smithi/7539667/orig.config.yaml 2024-02-05T04:40:07.630 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-05T04:40:07.725 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/nmordech-2024-01-31_07:15:30-rados-wip-nitzan-max-pglog-size-distro-default-smithi/7539667 branch: wip-nitzan-max-pglog-size description: rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} email: null first_in_suite: false job_id: '7539667' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: nmordech-2024-01-31_07:15:30-rados-wip-nitzan-max-pglog-size-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-nitzan-max-pglog-size ceph: conf: global: mon client directed command retry: 5 mon election default strategy: 3 ms inject socket failures: 2500 ms tcp read timeout: 5 osd_async_recovery_min_cost: 1 osd_object_clean_region_max_num_intervals: 1000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon scrub interval: 300 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd debug reject backfill probability: 0.1 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd max backfills: 3 osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd memory target: 1610612736 osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - slow request - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from sha1: d4ed20cfb238ef2c47d64ca4533eca00b8968b98 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: d4ed20cfb238ef2c47d64ca4533eca00b8968b98 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-nitzan-max-pglog-size sha1: d4ed20cfb238ef2c47d64ca4533eca00b8968b98 owner: scheduled_nmordech@teuthology priority: 1000 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.y - osd.0 - osd.4 - osd.8 - osd.12 - - mon.b - osd.1 - osd.5 - osd.9 - osd.13 - - mon.c - osd.2 - osd.6 - osd.10 - osd.14 - - mgr.x - osd.3 - osd.7 - osd.11 - osd.15 - client.0 seed: 8695 sha1: d4ed20cfb238ef2c47d64ca4533eca00b8968b98 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-nitzan-max-pglog-size suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: d4ed20cfb238ef2c47d64ca4533eca00b8968b98 targets: smithi062.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLrCXX9FGrvpb2YY//9WUXmiMbUrI6W2c1RxGH7NTG8iNneJEkvdg41pWixfxXaNkgRiaumItanuQ+E6EfZtsuQ= smithi091.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKACiH2/ZV+aC13a4lw3tv6KdmbuvnXSsmKFzugeTLDBQHhxcPZCUScqyjU02uLKZMsAiDzBiR/NEjXhmt//V1Q= smithi150.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDkiF69tNRQpFmCKo4Q2yleP7Jc1Pgc2YXFX0TcAZ3/oJIAxt4oc0AJkq+b2psJRwqrnXI9Z/uxNWwNPz7ZO4Fo= smithi196.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOOMfC5VqynyKF0c2ZrwRiL5Aavhag9Mt0amEBga9yehMArhs/XBirD15Vfk+fhSedG28dlp3w/xd36KPqL4jPg= tasks: - install: null - ceph: conf: osd: debug monc: 20 - thrashosds: bluestore_new_sharding: random chance_bluestore_reshard: 1 chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 min_in: 8 timeout: 1200 - rados: clients: - client.0 ec_pool: true erasure_code_profile: c: 2 crush-failure-domain: osd k: 4 m: 3 name: shecprofile plugin: shec objects: 50 op_weights: append: 100 copy_from: 50 delete: 50 read: 100 rmattr: 25 rollback: 50 setattr: 25 snap_create: 50 snap_remove: 50 write: 0 ops: 400 write_append_excl: false teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: d9fdb2209e15b39d9f061fd85399f352ce0f0894 timestamp: 2024-01-31_07:15:30 tube: smithi user: nmordech verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-02-05T04:40:07.725 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa; will attempt to use it 2024-02-05T04:40:07.726 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks 2024-02-05T04:40:07.726 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-02-05T04:40:07.727 INFO:teuthology.task.internal:Checking packages... 2024-02-05T04:40:07.751 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'd4ed20cfb238ef2c47d64ca4533eca00b8968b98' 2024-02-05T04:40:07.751 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-05T04:40:07.751 INFO:teuthology.packaging:ref: None 2024-02-05T04:40:07.751 INFO:teuthology.packaging:tag: None 2024-02-05T04:40:07.751 INFO:teuthology.packaging:branch: wip-nitzan-max-pglog-size 2024-02-05T04:40:07.751 INFO:teuthology.packaging:sha1: d4ed20cfb238ef2c47d64ca4533eca00b8968b98 2024-02-05T04:40:07.752 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-nitzan-max-pglog-size 2024-02-05T04:40:07.892 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-658.gd4ed20cf 2024-02-05T04:40:07.893 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-02-05T04:40:07.899 INFO:teuthology.task.internal:no buildpackages task found 2024-02-05T04:40:07.900 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-02-05T04:40:07.906 INFO:teuthology.task.internal:Saving configuration 2024-02-05T04:40:07.921 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-02-05T04:40:07.927 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-05T04:40:07.948 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi062.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/nmordech-2024-01-31_07:15:30-rados-wip-nitzan-max-pglog-size-distro-default-smithi/7539667', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-05 04:34:49.111665', 'locked_by': 'scheduled_nmordech@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLrCXX9FGrvpb2YY//9WUXmiMbUrI6W2c1RxGH7NTG8iNneJEkvdg41pWixfxXaNkgRiaumItanuQ+E6EfZtsuQ='} 2024-02-05T04:40:07.969 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi091.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/nmordech-2024-01-31_07:15:30-rados-wip-nitzan-max-pglog-size-distro-default-smithi/7539667', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-05 04:34:49.106744', 'locked_by': 'scheduled_nmordech@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKACiH2/ZV+aC13a4lw3tv6KdmbuvnXSsmKFzugeTLDBQHhxcPZCUScqyjU02uLKZMsAiDzBiR/NEjXhmt//V1Q='} 2024-02-05T04:40:07.993 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi150.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/nmordech-2024-01-31_07:15:30-rados-wip-nitzan-max-pglog-size-distro-default-smithi/7539667', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-05 04:34:49.113890', 'locked_by': 'scheduled_nmordech@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDkiF69tNRQpFmCKo4Q2yleP7Jc1Pgc2YXFX0TcAZ3/oJIAxt4oc0AJkq+b2psJRwqrnXI9Z/uxNWwNPz7ZO4Fo='} 2024-02-05T04:40:08.019 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi196.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/nmordech-2024-01-31_07:15:30-rados-wip-nitzan-max-pglog-size-distro-default-smithi/7539667', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-05 04:34:49.112812', 'locked_by': 'scheduled_nmordech@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOOMfC5VqynyKF0c2ZrwRiL5Aavhag9Mt0amEBga9yehMArhs/XBirD15Vfk+fhSedG28dlp3w/xd36KPqL4jPg='} 2024-02-05T04:40:08.019 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-02-05T04:40:08.024 INFO:teuthology.task.internal:roles: ubuntu@smithi062.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'osd.4', 'osd.8', 'osd.12'] 2024-02-05T04:40:08.025 INFO:teuthology.task.internal:roles: ubuntu@smithi091.front.sepia.ceph.com - ['mon.b', 'osd.1', 'osd.5', 'osd.9', 'osd.13'] 2024-02-05T04:40:08.025 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.c', 'osd.2', 'osd.6', 'osd.10', 'osd.14'] 2024-02-05T04:40:08.025 INFO:teuthology.task.internal:roles: ubuntu@smithi196.front.sepia.ceph.com - ['mgr.x', 'osd.3', 'osd.7', 'osd.11', 'osd.15', 'client.0'] 2024-02-05T04:40:08.025 INFO:teuthology.run_tasks:Running task console_log... 2024-02-05T04:40:08.167 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fc8ab9b4160>, signals=[15]) 2024-02-05T04:40:08.167 INFO:teuthology.run_tasks:Running task internal.connect... 2024-02-05T04:40:08.174 INFO:teuthology.task.internal:Opening connections... 2024-02-05T04:40:08.174 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi062.front.sepia.ceph.com 2024-02-05T04:40:08.176 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi062.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-05T04:40:08.280 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi091.front.sepia.ceph.com 2024-02-05T04:40:08.281 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi091.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-05T04:40:08.369 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi150.front.sepia.ceph.com 2024-02-05T04:40:08.370 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-05T04:40:08.459 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi196.front.sepia.ceph.com 2024-02-05T04:40:08.460 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-05T04:40:08.549 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-02-05T04:40:08.555 DEBUG:teuthology.orchestra.run.smithi062:> uname -m 2024-02-05T04:40:08.574 INFO:teuthology.orchestra.run.smithi062.stdout:x86_64 2024-02-05T04:40:08.574 DEBUG:teuthology.orchestra.run.smithi062:> cat /etc/os-release 2024-02-05T04:40:08.632 INFO:teuthology.orchestra.run.smithi062.stdout:NAME="CentOS Stream" 2024-02-05T04:40:08.632 INFO:teuthology.orchestra.run.smithi062.stdout:VERSION="9" 2024-02-05T04:40:08.632 INFO:teuthology.orchestra.run.smithi062.stdout:ID="centos" 2024-02-05T04:40:08.632 INFO:teuthology.orchestra.run.smithi062.stdout:ID_LIKE="rhel fedora" 2024-02-05T04:40:08.633 INFO:teuthology.orchestra.run.smithi062.stdout:VERSION_ID="9" 2024-02-05T04:40:08.633 INFO:teuthology.orchestra.run.smithi062.stdout:PLATFORM_ID="platform:el9" 2024-02-05T04:40:08.633 INFO:teuthology.orchestra.run.smithi062.stdout:PRETTY_NAME="CentOS Stream 9" 2024-02-05T04:40:08.633 INFO:teuthology.orchestra.run.smithi062.stdout:ANSI_COLOR="0;31" 2024-02-05T04:40:08.633 INFO:teuthology.orchestra.run.smithi062.stdout:LOGO="fedora-logo-icon" 2024-02-05T04:40:08.633 INFO:teuthology.orchestra.run.smithi062.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-02-05T04:40:08.633 INFO:teuthology.orchestra.run.smithi062.stdout:HOME_URL="https://centos.org/" 2024-02-05T04:40:08.633 INFO:teuthology.orchestra.run.smithi062.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-05T04:40:08.633 INFO:teuthology.orchestra.run.smithi062.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-02-05T04:40:08.633 INFO:teuthology.orchestra.run.smithi062.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-05T04:40:08.634 INFO:teuthology.lock.ops:Updating smithi062.front.sepia.ceph.com on lock server 2024-02-05T04:40:08.655 DEBUG:teuthology.orchestra.run.smithi091:> uname -m 2024-02-05T04:40:08.672 INFO:teuthology.orchestra.run.smithi091.stdout:x86_64 2024-02-05T04:40:08.672 DEBUG:teuthology.orchestra.run.smithi091:> cat /etc/os-release 2024-02-05T04:40:08.730 INFO:teuthology.orchestra.run.smithi091.stdout:NAME="CentOS Stream" 2024-02-05T04:40:08.730 INFO:teuthology.orchestra.run.smithi091.stdout:VERSION="9" 2024-02-05T04:40:08.730 INFO:teuthology.orchestra.run.smithi091.stdout:ID="centos" 2024-02-05T04:40:08.730 INFO:teuthology.orchestra.run.smithi091.stdout:ID_LIKE="rhel fedora" 2024-02-05T04:40:08.730 INFO:teuthology.orchestra.run.smithi091.stdout:VERSION_ID="9" 2024-02-05T04:40:08.730 INFO:teuthology.orchestra.run.smithi091.stdout:PLATFORM_ID="platform:el9" 2024-02-05T04:40:08.730 INFO:teuthology.orchestra.run.smithi091.stdout:PRETTY_NAME="CentOS Stream 9" 2024-02-05T04:40:08.730 INFO:teuthology.orchestra.run.smithi091.stdout:ANSI_COLOR="0;31" 2024-02-05T04:40:08.730 INFO:teuthology.orchestra.run.smithi091.stdout:LOGO="fedora-logo-icon" 2024-02-05T04:40:08.730 INFO:teuthology.orchestra.run.smithi091.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-02-05T04:40:08.730 INFO:teuthology.orchestra.run.smithi091.stdout:HOME_URL="https://centos.org/" 2024-02-05T04:40:08.730 INFO:teuthology.orchestra.run.smithi091.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-05T04:40:08.730 INFO:teuthology.orchestra.run.smithi091.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-02-05T04:40:08.731 INFO:teuthology.orchestra.run.smithi091.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-05T04:40:08.731 INFO:teuthology.lock.ops:Updating smithi091.front.sepia.ceph.com on lock server 2024-02-05T04:40:08.755 DEBUG:teuthology.orchestra.run.smithi150:> uname -m 2024-02-05T04:40:08.773 INFO:teuthology.orchestra.run.smithi150.stdout:x86_64 2024-02-05T04:40:08.773 DEBUG:teuthology.orchestra.run.smithi150:> cat /etc/os-release 2024-02-05T04:40:08.833 INFO:teuthology.orchestra.run.smithi150.stdout:NAME="CentOS Stream" 2024-02-05T04:40:08.833 INFO:teuthology.orchestra.run.smithi150.stdout:VERSION="9" 2024-02-05T04:40:08.833 INFO:teuthology.orchestra.run.smithi150.stdout:ID="centos" 2024-02-05T04:40:08.833 INFO:teuthology.orchestra.run.smithi150.stdout:ID_LIKE="rhel fedora" 2024-02-05T04:40:08.833 INFO:teuthology.orchestra.run.smithi150.stdout:VERSION_ID="9" 2024-02-05T04:40:08.833 INFO:teuthology.orchestra.run.smithi150.stdout:PLATFORM_ID="platform:el9" 2024-02-05T04:40:08.833 INFO:teuthology.orchestra.run.smithi150.stdout:PRETTY_NAME="CentOS Stream 9" 2024-02-05T04:40:08.833 INFO:teuthology.orchestra.run.smithi150.stdout:ANSI_COLOR="0;31" 2024-02-05T04:40:08.834 INFO:teuthology.orchestra.run.smithi150.stdout:LOGO="fedora-logo-icon" 2024-02-05T04:40:08.834 INFO:teuthology.orchestra.run.smithi150.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-02-05T04:40:08.834 INFO:teuthology.orchestra.run.smithi150.stdout:HOME_URL="https://centos.org/" 2024-02-05T04:40:08.834 INFO:teuthology.orchestra.run.smithi150.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-05T04:40:08.834 INFO:teuthology.orchestra.run.smithi150.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-02-05T04:40:08.834 INFO:teuthology.orchestra.run.smithi150.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-05T04:40:08.834 INFO:teuthology.lock.ops:Updating smithi150.front.sepia.ceph.com on lock server 2024-02-05T04:40:08.855 DEBUG:teuthology.orchestra.run.smithi196:> uname -m 2024-02-05T04:40:08.873 INFO:teuthology.orchestra.run.smithi196.stdout:x86_64 2024-02-05T04:40:08.873 DEBUG:teuthology.orchestra.run.smithi196:> cat /etc/os-release 2024-02-05T04:40:08.931 INFO:teuthology.orchestra.run.smithi196.stdout:NAME="CentOS Stream" 2024-02-05T04:40:08.931 INFO:teuthology.orchestra.run.smithi196.stdout:VERSION="9" 2024-02-05T04:40:08.931 INFO:teuthology.orchestra.run.smithi196.stdout:ID="centos" 2024-02-05T04:40:08.931 INFO:teuthology.orchestra.run.smithi196.stdout:ID_LIKE="rhel fedora" 2024-02-05T04:40:08.931 INFO:teuthology.orchestra.run.smithi196.stdout:VERSION_ID="9" 2024-02-05T04:40:08.931 INFO:teuthology.orchestra.run.smithi196.stdout:PLATFORM_ID="platform:el9" 2024-02-05T04:40:08.931 INFO:teuthology.orchestra.run.smithi196.stdout:PRETTY_NAME="CentOS Stream 9" 2024-02-05T04:40:08.931 INFO:teuthology.orchestra.run.smithi196.stdout:ANSI_COLOR="0;31" 2024-02-05T04:40:08.931 INFO:teuthology.orchestra.run.smithi196.stdout:LOGO="fedora-logo-icon" 2024-02-05T04:40:08.931 INFO:teuthology.orchestra.run.smithi196.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-02-05T04:40:08.931 INFO:teuthology.orchestra.run.smithi196.stdout:HOME_URL="https://centos.org/" 2024-02-05T04:40:08.931 INFO:teuthology.orchestra.run.smithi196.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-05T04:40:08.932 INFO:teuthology.orchestra.run.smithi196.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-02-05T04:40:08.932 INFO:teuthology.orchestra.run.smithi196.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-05T04:40:08.932 INFO:teuthology.lock.ops:Updating smithi196.front.sepia.ceph.com on lock server 2024-02-05T04:40:08.955 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-02-05T04:40:08.970 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-02-05T04:40:08.976 INFO:teuthology.task.internal:Checking for old test directory... 2024-02-05T04:40:08.976 DEBUG:teuthology.orchestra.run.smithi062:> test '!' -e /home/ubuntu/cephtest 2024-02-05T04:40:08.979 DEBUG:teuthology.orchestra.run.smithi091:> test '!' -e /home/ubuntu/cephtest 2024-02-05T04:40:08.981 DEBUG:teuthology.orchestra.run.smithi150:> test '!' -e /home/ubuntu/cephtest 2024-02-05T04:40:08.984 DEBUG:teuthology.orchestra.run.smithi196:> test '!' -e /home/ubuntu/cephtest 2024-02-05T04:40:09.000 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-02-05T04:40:09.006 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-02-05T04:40:09.006 DEBUG:teuthology.orchestra.run.smithi062:> test -z $(ls -A /var/lib/ceph) 2024-02-05T04:40:09.037 DEBUG:teuthology.orchestra.run.smithi091:> test -z $(ls -A /var/lib/ceph) 2024-02-05T04:40:09.039 DEBUG:teuthology.orchestra.run.smithi150:> test -z $(ls -A /var/lib/ceph) 2024-02-05T04:40:09.044 DEBUG:teuthology.orchestra.run.smithi196:> test -z $(ls -A /var/lib/ceph) 2024-02-05T04:40:09.069 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-02-05T04:40:09.168 INFO:teuthology.run_tasks:Running task kernel... 2024-02-05T04:40:09.182 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-02-05T04:40:09.182 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}, 'mgr.x': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-02-05T04:40:09.182 DEBUG:teuthology.orchestra.run.smithi062:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-05T04:40:09.183 DEBUG:teuthology.orchestra.run.smithi091:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-05T04:40:09.183 DEBUG:teuthology.orchestra.run.smithi150:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-05T04:40:09.183 DEBUG:teuthology.orchestra.run.smithi196:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-05T04:40:09.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T04:40:09.200 DEBUG:teuthology.orchestra.run.smithi150:> uname -r 2024-02-05T04:40:09.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T04:40:09.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T04:40:09.201 DEBUG:teuthology.orchestra.run.smithi196:> uname -r 2024-02-05T04:40:09.202 DEBUG:teuthology.orchestra.run.smithi091:> uname -r 2024-02-05T04:40:09.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T04:40:09.202 DEBUG:teuthology.orchestra.run.smithi062:> uname -r 2024-02-05T04:40:09.258 INFO:teuthology.orchestra.run.smithi196.stdout:5.14.0-361.el9.x86_64 2024-02-05T04:40:09.258 INFO:teuthology.orchestra.run.smithi091.stdout:5.14.0-361.el9.x86_64 2024-02-05T04:40:09.259 INFO:teuthology.task.kernel:Running kernel on smithi196: 5.14.0-361.el9.x86_64 2024-02-05T04:40:09.259 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum install -y kernel 2024-02-05T04:40:09.259 INFO:teuthology.task.kernel:Running kernel on smithi091: 5.14.0-361.el9.x86_64 2024-02-05T04:40:09.259 DEBUG:teuthology.orchestra.run.smithi091:> sudo yum install -y kernel 2024-02-05T04:40:09.260 INFO:teuthology.orchestra.run.smithi150.stdout:5.14.0-361.el9.x86_64 2024-02-05T04:40:09.260 INFO:teuthology.task.kernel:Running kernel on smithi150: 5.14.0-361.el9.x86_64 2024-02-05T04:40:09.260 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum install -y kernel 2024-02-05T04:40:09.261 INFO:teuthology.orchestra.run.smithi062.stdout:5.14.0-361.el9.x86_64 2024-02-05T04:40:09.261 INFO:teuthology.task.kernel:Running kernel on smithi062: 5.14.0-361.el9.x86_64 2024-02-05T04:40:09.262 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum install -y kernel 2024-02-05T04:40:12.032 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS-9 - AppStream 5.7 MB/s | 6.6 MB 00:01 2024-02-05T04:40:12.058 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS-9 - AppStream 5.5 MB/s | 6.6 MB 00:01 2024-02-05T04:40:12.108 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS-9 - AppStream 5.9 MB/s | 6.6 MB 00:01 2024-02-05T04:40:12.120 INFO:teuthology.orchestra.run.smithi091.stdout:CentOS-9 - AppStream 5.7 MB/s | 6.6 MB 00:01 2024-02-05T04:40:13.600 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS-9 - BaseOS 4.3 MB/s | 2.0 MB 00:00 2024-02-05T04:40:13.725 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS-9 - BaseOS 4.2 MB/s | 2.0 MB 00:00 2024-02-05T04:40:13.845 INFO:teuthology.orchestra.run.smithi091.stdout:CentOS-9 - BaseOS 3.1 MB/s | 2.0 MB 00:00 2024-02-05T04:40:14.215 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS-9 - BaseOS 1.8 MB/s | 2.0 MB 00:01 2024-02-05T04:40:14.682 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.0 MB 00:00 2024-02-05T04:40:14.690 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.0 MB 00:00 2024-02-05T04:40:15.085 INFO:teuthology.orchestra.run.smithi091.stdout:CentOS Stream 9 - BaseOS 9.2 MB/s | 8.0 MB 00:00 2024-02-05T04:40:16.281 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - BaseOS 4.7 MB/s | 8.0 MB 00:01 2024-02-05T04:40:18.922 INFO:teuthology.orchestra.run.smithi091.stdout:CentOS Stream 9 - AppStream 6.8 MB/s | 19 MB 00:02 2024-02-05T04:40:19.133 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - AppStream 5.5 MB/s | 19 MB 00:03 2024-02-05T04:40:22.139 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - AppStream 3.9 MB/s | 19 MB 00:04 2024-02-05T04:40:24.019 INFO:teuthology.orchestra.run.smithi091.stdout:CentOS Stream 9 - CRB 3.6 MB/s | 5.9 MB 00:01 2024-02-05T04:40:24.657 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS Stream 9 - AppStream 2.1 MB/s | 19 MB 00:08 2024-02-05T04:40:25.445 INFO:teuthology.orchestra.run.smithi091.stdout:CentOS Stream 9 - Extras packages 47 kB/s | 15 kB 00:00 2024-02-05T04:40:26.225 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - CRB 1.6 MB/s | 5.9 MB 00:03 2024-02-05T04:40:26.488 INFO:teuthology.orchestra.run.smithi091.stdout:Extra Packages for Enterprise Linux 22 MB/s | 20 MB 00:00 2024-02-05T04:40:27.692 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - CRB 2.9 MB/s | 5.9 MB 00:02 2024-02-05T04:40:27.834 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - Extras packages 30 kB/s | 15 kB 00:00 2024-02-05T04:40:29.398 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - Extras packages 25 kB/s | 15 kB 00:00 2024-02-05T04:40:30.599 INFO:teuthology.orchestra.run.smithi196.stdout:Extra Packages for Enterprise Linux 19 MB/s | 20 MB 00:01 2024-02-05T04:40:31.149 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS Stream 9 - CRB 2.0 MB/s | 5.9 MB 00:02 2024-02-05T04:40:31.462 INFO:teuthology.orchestra.run.smithi091.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-02-05T04:40:33.694 INFO:teuthology.orchestra.run.smithi091.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-02-05T04:40:33.694 INFO:teuthology.orchestra.run.smithi091.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-02-05T04:40:33.788 INFO:teuthology.orchestra.run.smithi091.stdout:Dependencies resolved. 2024-02-05T04:40:33.798 INFO:teuthology.orchestra.run.smithi091.stdout:================================================================================ 2024-02-05T04:40:33.798 INFO:teuthology.orchestra.run.smithi091.stdout: Package Arch Version Repository Size 2024-02-05T04:40:33.798 INFO:teuthology.orchestra.run.smithi091.stdout:================================================================================ 2024-02-05T04:40:33.798 INFO:teuthology.orchestra.run.smithi091.stdout:Installing: 2024-02-05T04:40:33.798 INFO:teuthology.orchestra.run.smithi091.stdout: kernel x86_64 5.14.0-416.el9 CentOS-BaseOS 6.0 M 2024-02-05T04:40:33.798 INFO:teuthology.orchestra.run.smithi091.stdout:Installing dependencies: 2024-02-05T04:40:33.798 INFO:teuthology.orchestra.run.smithi091.stdout: kernel-core x86_64 5.14.0-416.el9 CentOS-BaseOS 21 M 2024-02-05T04:40:33.798 INFO:teuthology.orchestra.run.smithi091.stdout: kernel-modules x86_64 5.14.0-416.el9 CentOS-BaseOS 39 M 2024-02-05T04:40:33.798 INFO:teuthology.orchestra.run.smithi091.stdout: kernel-modules-core x86_64 5.14.0-416.el9 CentOS-BaseOS 33 M 2024-02-05T04:40:33.798 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:40:33.799 INFO:teuthology.orchestra.run.smithi091.stdout:Transaction Summary 2024-02-05T04:40:33.799 INFO:teuthology.orchestra.run.smithi091.stdout:================================================================================ 2024-02-05T04:40:33.799 INFO:teuthology.orchestra.run.smithi091.stdout:Install 4 Packages 2024-02-05T04:40:33.799 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:40:33.799 INFO:teuthology.orchestra.run.smithi091.stdout:Total download size: 99 M 2024-02-05T04:40:33.800 INFO:teuthology.orchestra.run.smithi091.stdout:Installed size: 122 M 2024-02-05T04:40:33.800 INFO:teuthology.orchestra.run.smithi091.stdout:Downloading Packages: 2024-02-05T04:40:34.697 INFO:teuthology.orchestra.run.smithi091.stdout:(1/4): kernel-5.14.0-416.el9.x86_64.rpm 6.7 MB/s | 6.0 MB 00:00 2024-02-05T04:40:34.941 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS Stream 9 - Extras packages 5.6 kB/s | 15 kB 00:02 2024-02-05T04:40:35.524 INFO:teuthology.orchestra.run.smithi196.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-02-05T04:40:35.838 INFO:teuthology.orchestra.run.smithi091.stdout:(2/4): kernel-core-5.14.0-416.el9.x86_64.rpm 10 MB/s | 21 MB 00:02 2024-02-05T04:40:36.614 INFO:teuthology.orchestra.run.smithi091.stdout:(3/4): kernel-modules-core-5.14.0-416.el9.x86_6 17 MB/s | 33 MB 00:01 2024-02-05T04:40:36.980 INFO:teuthology.orchestra.run.smithi091.stdout:(4/4): kernel-modules-5.14.0-416.el9.x86_64.rpm 12 MB/s | 39 MB 00:03 2024-02-05T04:40:36.983 INFO:teuthology.orchestra.run.smithi091.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:40:36.983 INFO:teuthology.orchestra.run.smithi091.stdout:Total 31 MB/s | 99 MB 00:03 2024-02-05T04:40:36.984 INFO:teuthology.orchestra.run.smithi091.stdout:Running transaction check 2024-02-05T04:40:37.042 INFO:teuthology.orchestra.run.smithi062.stdout:Extra Packages for Enterprise Linux 10 MB/s | 20 MB 00:01 2024-02-05T04:40:37.361 INFO:teuthology.orchestra.run.smithi091.stdout:Transaction check succeeded. 2024-02-05T04:40:37.361 INFO:teuthology.orchestra.run.smithi091.stdout:Running transaction test 2024-02-05T04:40:37.756 INFO:teuthology.orchestra.run.smithi196.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-02-05T04:40:37.756 INFO:teuthology.orchestra.run.smithi196.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-02-05T04:40:37.851 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-02-05T04:40:37.861 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-02-05T04:40:37.861 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-02-05T04:40:37.861 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-02-05T04:40:37.861 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-02-05T04:40:37.862 INFO:teuthology.orchestra.run.smithi196.stdout: kernel x86_64 5.14.0-416.el9 CentOS-BaseOS 6.0 M 2024-02-05T04:40:37.862 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-02-05T04:40:37.862 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-core x86_64 5.14.0-416.el9 CentOS-BaseOS 21 M 2024-02-05T04:40:37.862 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-modules x86_64 5.14.0-416.el9 CentOS-BaseOS 39 M 2024-02-05T04:40:37.862 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-modules-core x86_64 5.14.0-416.el9 CentOS-BaseOS 33 M 2024-02-05T04:40:37.862 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:40:37.862 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-02-05T04:40:37.862 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-02-05T04:40:37.862 INFO:teuthology.orchestra.run.smithi196.stdout:Install 4 Packages 2024-02-05T04:40:37.862 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:40:37.863 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 99 M 2024-02-05T04:40:37.863 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 122 M 2024-02-05T04:40:37.863 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-02-05T04:40:38.405 INFO:teuthology.orchestra.run.smithi091.stdout:Transaction test succeeded. 2024-02-05T04:40:38.406 INFO:teuthology.orchestra.run.smithi091.stdout:Running transaction 2024-02-05T04:40:38.575 INFO:teuthology.orchestra.run.smithi196.stdout:(1/4): kernel-5.14.0-416.el9.x86_64.rpm 8.4 MB/s | 6.0 MB 00:00 2024-02-05T04:40:39.484 INFO:teuthology.orchestra.run.smithi196.stdout:(2/4): kernel-core-5.14.0-416.el9.x86_64.rpm 13 MB/s | 21 MB 00:01 2024-02-05T04:40:40.000 INFO:teuthology.orchestra.run.smithi196.stdout:(3/4): kernel-modules-core-5.14.0-416.el9.x86_6 23 MB/s | 33 MB 00:01 2024-02-05T04:40:40.342 INFO:teuthology.orchestra.run.smithi196.stdout:(4/4): kernel-modules-5.14.0-416.el9.x86_64.rpm 16 MB/s | 39 MB 00:02 2024-02-05T04:40:40.345 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:40:40.345 INFO:teuthology.orchestra.run.smithi196.stdout:Total 40 MB/s | 99 MB 00:02 2024-02-05T04:40:40.346 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-02-05T04:40:40.706 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-02-05T04:40:40.706 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-02-05T04:40:40.774 INFO:teuthology.orchestra.run.smithi091.stdout: Preparing : 1/1 2024-02-05T04:40:41.525 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : kernel-modules-core-5.14.0-416.el9.x86_64 1/4 2024-02-05T04:40:41.655 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : kernel-core-5.14.0-416.el9.x86_64 2/4 2024-02-05T04:40:41.728 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-02-05T04:40:41.728 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-02-05T04:40:42.033 INFO:teuthology.orchestra.run.smithi062.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-02-05T04:40:44.126 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-02-05T04:40:44.397 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: kernel-core-5.14.0-416.el9.x86_64 2/4 2024-02-05T04:40:44.491 INFO:teuthology.orchestra.run.smithi062.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-02-05T04:40:44.491 INFO:teuthology.orchestra.run.smithi062.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-02-05T04:40:44.566 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : kernel-modules-5.14.0-416.el9.x86_64 3/4 2024-02-05T04:40:44.580 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-02-05T04:40:44.585 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-02-05T04:40:44.586 INFO:teuthology.orchestra.run.smithi062.stdout: Package Arch Version Repository Size 2024-02-05T04:40:44.586 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-02-05T04:40:44.586 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2024-02-05T04:40:44.586 INFO:teuthology.orchestra.run.smithi062.stdout: kernel x86_64 5.14.0-416.el9 CentOS-BaseOS 6.0 M 2024-02-05T04:40:44.586 INFO:teuthology.orchestra.run.smithi062.stdout:Installing dependencies: 2024-02-05T04:40:44.586 INFO:teuthology.orchestra.run.smithi062.stdout: kernel-core x86_64 5.14.0-416.el9 CentOS-BaseOS 21 M 2024-02-05T04:40:44.586 INFO:teuthology.orchestra.run.smithi062.stdout: kernel-modules x86_64 5.14.0-416.el9 CentOS-BaseOS 39 M 2024-02-05T04:40:44.586 INFO:teuthology.orchestra.run.smithi062.stdout: kernel-modules-core x86_64 5.14.0-416.el9 CentOS-BaseOS 33 M 2024-02-05T04:40:44.586 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:40:44.586 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2024-02-05T04:40:44.586 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-02-05T04:40:44.586 INFO:teuthology.orchestra.run.smithi062.stdout:Install 4 Packages 2024-02-05T04:40:44.586 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:40:44.587 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 99 M 2024-02-05T04:40:44.587 INFO:teuthology.orchestra.run.smithi062.stdout:Installed size: 122 M 2024-02-05T04:40:44.587 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2024-02-05T04:40:44.796 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : kernel-modules-core-5.14.0-416.el9.x86_64 1/4 2024-02-05T04:40:44.951 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : kernel-core-5.14.0-416.el9.x86_64 2/4 2024-02-05T04:40:45.402 INFO:teuthology.orchestra.run.smithi062.stdout:(1/4): kernel-5.14.0-416.el9.x86_64.rpm 7.4 MB/s | 6.0 MB 00:00 2024-02-05T04:40:46.160 INFO:teuthology.orchestra.run.smithi062.stdout:(2/4): kernel-core-5.14.0-416.el9.x86_64.rpm 13 MB/s | 21 MB 00:01 2024-02-05T04:40:47.011 INFO:teuthology.orchestra.run.smithi062.stdout:(3/4): kernel-modules-5.14.0-416.el9.x86_64.rpm 16 MB/s | 39 MB 00:02 2024-02-05T04:40:47.527 INFO:teuthology.orchestra.run.smithi062.stdout:(4/4): kernel-modules-core-5.14.0-416.el9.x86_6 16 MB/s | 33 MB 00:02 2024-02-05T04:40:47.531 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:40:47.531 INFO:teuthology.orchestra.run.smithi062.stdout:Total 34 MB/s | 99 MB 00:02 2024-02-05T04:40:47.531 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2024-02-05T04:40:47.751 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-core-5.14.0-416.el9.x86_64 2/4 2024-02-05T04:40:47.936 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : kernel-modules-5.14.0-416.el9.x86_64 3/4 2024-02-05T04:40:48.007 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2024-02-05T04:40:48.007 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2024-02-05T04:40:49.078 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2024-02-05T04:40:49.079 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2024-02-05T04:40:49.258 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: kernel-modules-5.14.0-416.el9.x86_64 3/4 2024-02-05T04:40:49.361 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : kernel-5.14.0-416.el9.x86_64 4/4 2024-02-05T04:40:51.634 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2024-02-05T04:40:52.334 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : kernel-modules-core-5.14.0-416.el9.x86_64 1/4 2024-02-05T04:40:52.460 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : kernel-core-5.14.0-416.el9.x86_64 2/4 2024-02-05T04:40:52.633 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-modules-5.14.0-416.el9.x86_64 3/4 2024-02-05T04:40:52.831 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : kernel-5.14.0-416.el9.x86_64 4/4 2024-02-05T04:40:53.903 INFO:teuthology.orchestra.run.smithi150.stdout:Extra Packages for Enterprise Linux 802 kB/s | 20 MB 00:25 2024-02-05T04:40:54.076 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: kernel-modules-core-5.14.0-416.el9.x86_64 4/4 2024-02-05T04:40:55.348 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: kernel-core-5.14.0-416.el9.x86_64 2/4 2024-02-05T04:40:55.529 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : kernel-modules-5.14.0-416.el9.x86_64 3/4 2024-02-05T04:40:57.548 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-modules-core-5.14.0-416.el9.x86_64 4/4 2024-02-05T04:40:58.725 INFO:teuthology.orchestra.run.smithi150.stdout:lab-extras 18 kB/s | 1.7 kB 00:00 2024-02-05T04:41:00.596 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: kernel-modules-5.14.0-416.el9.x86_64 3/4 2024-02-05T04:41:00.709 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : kernel-5.14.0-416.el9.x86_64 4/4 2024-02-05T04:41:00.890 INFO:teuthology.orchestra.run.smithi150.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-02-05T04:41:00.890 INFO:teuthology.orchestra.run.smithi150.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-02-05T04:41:00.985 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-02-05T04:41:00.992 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-02-05T04:41:00.992 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2024-02-05T04:41:00.992 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-02-05T04:41:00.992 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2024-02-05T04:41:00.992 INFO:teuthology.orchestra.run.smithi150.stdout: kernel x86_64 5.14.0-416.el9 CentOS-BaseOS 6.0 M 2024-02-05T04:41:00.992 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2024-02-05T04:41:00.992 INFO:teuthology.orchestra.run.smithi150.stdout: kernel-core x86_64 5.14.0-416.el9 CentOS-BaseOS 21 M 2024-02-05T04:41:00.992 INFO:teuthology.orchestra.run.smithi150.stdout: kernel-modules x86_64 5.14.0-416.el9 CentOS-BaseOS 39 M 2024-02-05T04:41:00.992 INFO:teuthology.orchestra.run.smithi150.stdout: kernel-modules-core x86_64 5.14.0-416.el9 CentOS-BaseOS 33 M 2024-02-05T04:41:00.992 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:41:00.993 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-02-05T04:41:00.993 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-02-05T04:41:00.993 INFO:teuthology.orchestra.run.smithi150.stdout:Install 4 Packages 2024-02-05T04:41:00.993 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:41:00.994 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 99 M 2024-02-05T04:41:00.994 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 122 M 2024-02-05T04:41:00.994 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-02-05T04:41:01.734 INFO:teuthology.orchestra.run.smithi150.stdout:(1/4): kernel-5.14.0-416.el9.x86_64.rpm 8.1 MB/s | 6.0 MB 00:00 2024-02-05T04:41:02.401 INFO:teuthology.orchestra.run.smithi150.stdout:(2/4): kernel-core-5.14.0-416.el9.x86_64.rpm 15 MB/s | 21 MB 00:01 2024-02-05T04:41:02.993 INFO:teuthology.orchestra.run.smithi150.stdout:(3/4): kernel-modules-5.14.0-416.el9.x86_64.rpm 20 MB/s | 39 MB 00:01 2024-02-05T04:41:03.685 INFO:teuthology.orchestra.run.smithi150.stdout:(4/4): kernel-modules-core-5.14.0-416.el9.x86_6 17 MB/s | 33 MB 00:01 2024-02-05T04:41:03.688 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:41:03.688 INFO:teuthology.orchestra.run.smithi150.stdout:Total 37 MB/s | 99 MB 00:02 2024-02-05T04:41:03.689 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-02-05T04:41:04.074 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-02-05T04:41:04.074 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-02-05T04:41:05.131 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-02-05T04:41:05.132 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-02-05T04:41:05.795 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: kernel-modules-core-5.14.0-416.el9.x86_64 4/4 2024-02-05T04:41:07.555 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-02-05T04:41:08.409 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : kernel-modules-core-5.14.0-416.el9.x86_64 1/4 2024-02-05T04:41:08.552 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : kernel-core-5.14.0-416.el9.x86_64 2/4 2024-02-05T04:41:11.285 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: kernel-core-5.14.0-416.el9.x86_64 2/4 2024-02-05T04:41:11.470 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : kernel-modules-5.14.0-416.el9.x86_64 3/4 2024-02-05T04:41:16.222 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: kernel-modules-5.14.0-416.el9.x86_64 3/4 2024-02-05T04:41:16.332 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : kernel-5.14.0-416.el9.x86_64 4/4 2024-02-05T04:41:21.071 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: kernel-modules-core-5.14.0-416.el9.x86_64 4/4 2024-02-05T04:41:23.173 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: kernel-core-5.14.0-416.el9.x86_64 4/4 2024-02-05T04:41:23.174 INFO:teuthology.orchestra.run.smithi091.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-02-05T04:41:23.174 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:41:23.241 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: kernel-modules-5.14.0-416.el9.x86_64 4/4 2024-02-05T04:41:23.614 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: kernel-5.14.0-416.el9.x86_64 4/4 2024-02-05T04:41:23.614 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : kernel-5.14.0-416.el9.x86_64 1/4 2024-02-05T04:41:23.614 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : kernel-core-5.14.0-416.el9.x86_64 2/4 2024-02-05T04:41:23.615 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : kernel-modules-5.14.0-416.el9.x86_64 3/4 2024-02-05T04:41:24.046 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : kernel-modules-core-5.14.0-416.el9.x86_64 4/4 2024-02-05T04:41:24.046 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:41:24.046 INFO:teuthology.orchestra.run.smithi091.stdout:Installed: 2024-02-05T04:41:24.046 INFO:teuthology.orchestra.run.smithi091.stdout: kernel-5.14.0-416.el9.x86_64 2024-02-05T04:41:24.046 INFO:teuthology.orchestra.run.smithi091.stdout: kernel-core-5.14.0-416.el9.x86_64 2024-02-05T04:41:24.046 INFO:teuthology.orchestra.run.smithi091.stdout: kernel-modules-5.14.0-416.el9.x86_64 2024-02-05T04:41:24.046 INFO:teuthology.orchestra.run.smithi091.stdout: kernel-modules-core-5.14.0-416.el9.x86_64 2024-02-05T04:41:24.046 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:41:24.046 INFO:teuthology.orchestra.run.smithi091.stdout:Complete! 2024-02-05T04:41:24.577 DEBUG:teuthology.orchestra.run.smithi091:> rpm -q kernel | sort -rV | head -n 1 2024-02-05T04:41:24.710 INFO:teuthology.orchestra.run.smithi091.stdout:kernel-5.14.0-416.el9.x86_64 2024-02-05T04:41:24.711 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-416.el9.x86_64 2024-02-05T04:41:24.711 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-416.el9.x86_64 2024-02-05T04:41:24.711 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-05T04:41:24.711 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-05T04:41:24.711 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-02-05T04:41:24.711 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi091.front.sepia.ceph.com, path=None, version=distro) 2024-02-05T04:41:24.711 DEBUG:teuthology.orchestra.run.smithi091:> rpm -q kernel | sort -rV | head -n 1 2024-02-05T04:41:24.818 INFO:teuthology.orchestra.run.smithi091.stdout:kernel-5.14.0-416.el9.x86_64 2024-02-05T04:41:24.819 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-416.el9.x86_64 2024-02-05T04:41:24.819 DEBUG:teuthology.orchestra.run.smithi091:> sudo rpm -qi grub2-tools 2024-02-05T04:41:24.853 INFO:teuthology.orchestra.run.smithi091.stdout:Name : grub2-tools 2024-02-05T04:41:24.853 INFO:teuthology.orchestra.run.smithi091.stdout:Epoch : 1 2024-02-05T04:41:24.853 INFO:teuthology.orchestra.run.smithi091.stdout:Version : 2.06 2024-02-05T04:41:24.853 INFO:teuthology.orchestra.run.smithi091.stdout:Release : 68.el9 2024-02-05T04:41:24.853 INFO:teuthology.orchestra.run.smithi091.stdout:Architecture: x86_64 2024-02-05T04:41:24.853 INFO:teuthology.orchestra.run.smithi091.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-02-05T04:41:24.853 INFO:teuthology.orchestra.run.smithi091.stdout:Group : Unspecified 2024-02-05T04:41:24.853 INFO:teuthology.orchestra.run.smithi091.stdout:Size : 8264254 2024-02-05T04:41:24.853 INFO:teuthology.orchestra.run.smithi091.stdout:License : GPLv3+ 2024-02-05T04:41:24.853 INFO:teuthology.orchestra.run.smithi091.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-02-05T04:41:24.854 INFO:teuthology.orchestra.run.smithi091.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-02-05T04:41:24.854 INFO:teuthology.orchestra.run.smithi091.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-02-05T04:41:24.854 INFO:teuthology.orchestra.run.smithi091.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-02-05T04:41:24.854 INFO:teuthology.orchestra.run.smithi091.stdout:Packager : builder@centos.org 2024-02-05T04:41:24.854 INFO:teuthology.orchestra.run.smithi091.stdout:Vendor : CentOS 2024-02-05T04:41:24.854 INFO:teuthology.orchestra.run.smithi091.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-05T04:41:24.854 INFO:teuthology.orchestra.run.smithi091.stdout:Summary : Support tools for GRUB. 2024-02-05T04:41:24.854 INFO:teuthology.orchestra.run.smithi091.stdout:Description : 2024-02-05T04:41:24.854 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:41:24.854 INFO:teuthology.orchestra.run.smithi091.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-05T04:41:24.854 INFO:teuthology.orchestra.run.smithi091.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-05T04:41:24.854 INFO:teuthology.orchestra.run.smithi091.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-05T04:41:24.854 INFO:teuthology.orchestra.run.smithi091.stdout:hardware devices. 2024-02-05T04:41:24.854 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:41:24.855 INFO:teuthology.orchestra.run.smithi091.stdout:This subpackage provides tools for support of all platforms. 2024-02-05T04:41:24.855 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-05T04:41:24.856 INFO:teuthology.task.kernel:Updating grub on smithi091 to boot 5.14.0-416.el9.x86_64 2024-02-05T04:41:24.856 DEBUG:teuthology.orchestra.run.smithi091:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-05T04:41:25.494 INFO:teuthology.orchestra.run.smithi091.stderr:Generating grub configuration file ... 2024-02-05T04:41:26.630 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-core-5.14.0-416.el9.x86_64 4/4 2024-02-05T04:41:26.630 INFO:teuthology.orchestra.run.smithi196.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-02-05T04:41:26.630 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:41:26.700 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-modules-5.14.0-416.el9.x86_64 4/4 2024-02-05T04:41:26.942 INFO:teuthology.orchestra.run.smithi091.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-02-05T04:41:26.981 INFO:teuthology.orchestra.run.smithi091.stderr:done 2024-02-05T04:41:26.984 DEBUG:teuthology.orchestra.run.smithi091:> mktemp 2024-02-05T04:41:26.999 INFO:teuthology.orchestra.run.smithi091.stdout:/tmp/tmp.At7G0HySWc 2024-02-05T04:41:26.999 DEBUG:teuthology.orchestra.run.smithi091:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.At7G0HySWc 2024-02-05T04:41:27.075 DEBUG:teuthology.orchestra.run.smithi091:> sudo chmod 0666 /tmp/tmp.At7G0HySWc 2024-02-05T04:41:27.118 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-5.14.0-416.el9.x86_64 4/4 2024-02-05T04:41:27.118 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : kernel-5.14.0-416.el9.x86_64 1/4 2024-02-05T04:41:27.118 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : kernel-core-5.14.0-416.el9.x86_64 2/4 2024-02-05T04:41:27.118 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : kernel-modules-5.14.0-416.el9.x86_64 3/4 2024-02-05T04:41:27.217 DEBUG:teuthology.orchestra.remote:smithi091:/tmp/tmp.At7G0HySWc is 6KB 2024-02-05T04:41:27.266 DEBUG:teuthology.orchestra.run.smithi091:> rm -fr /tmp/tmp.At7G0HySWc 2024-02-05T04:41:27.280 DEBUG:teuthology.orchestra.run.smithi091:> sudo /bin/ls /boot/loader/entries || true 2024-02-05T04:41:27.344 INFO:teuthology.orchestra.run.smithi091.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-02-05T04:41:27.344 INFO:teuthology.orchestra.run.smithi091.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-02-05T04:41:27.344 INFO:teuthology.orchestra.run.smithi091.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-02-05T04:41:27.344 INFO:teuthology.orchestra.run.smithi091.stdout:15949e3c88704300882482eddfd7baae-5.14.0-416.el9.x86_64.conf 2024-02-05T04:41:27.346 DEBUG:teuthology.orchestra.run.smithi091:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-416.el9.x86_64 2024-02-05T04:41:27.478 DEBUG:teuthology.orchestra.run.smithi091:> sudo shutdown -r now 2024-02-05T04:41:27.532 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : kernel-modules-core-5.14.0-416.el9.x86_64 4/4 2024-02-05T04:41:27.532 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:41:27.532 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-02-05T04:41:27.532 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-5.14.0-416.el9.x86_64 2024-02-05T04:41:27.532 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-core-5.14.0-416.el9.x86_64 2024-02-05T04:41:27.532 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-modules-5.14.0-416.el9.x86_64 2024-02-05T04:41:27.532 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-modules-core-5.14.0-416.el9.x86_64 2024-02-05T04:41:27.532 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:41:27.532 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-02-05T04:41:28.043 DEBUG:teuthology.orchestra.run.smithi196:> rpm -q kernel | sort -rV | head -n 1 2024-02-05T04:41:28.169 INFO:teuthology.orchestra.run.smithi196.stdout:kernel-5.14.0-416.el9.x86_64 2024-02-05T04:41:28.169 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-416.el9.x86_64 2024-02-05T04:41:28.169 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-416.el9.x86_64 2024-02-05T04:41:28.169 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-05T04:41:28.169 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-05T04:41:28.169 INFO:teuthology.task.kernel:Installing distro kernel on mgr.x... 2024-02-05T04:41:28.170 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi196.front.sepia.ceph.com, path=None, version=distro) 2024-02-05T04:41:28.170 DEBUG:teuthology.orchestra.run.smithi196:> rpm -q kernel | sort -rV | head -n 1 2024-02-05T04:41:28.276 INFO:teuthology.orchestra.run.smithi196.stdout:kernel-5.14.0-416.el9.x86_64 2024-02-05T04:41:28.276 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-416.el9.x86_64 2024-02-05T04:41:28.276 DEBUG:teuthology.orchestra.run.smithi196:> sudo rpm -qi grub2-tools 2024-02-05T04:41:28.319 INFO:teuthology.orchestra.run.smithi196.stdout:Name : grub2-tools 2024-02-05T04:41:28.319 INFO:teuthology.orchestra.run.smithi196.stdout:Epoch : 1 2024-02-05T04:41:28.319 INFO:teuthology.orchestra.run.smithi196.stdout:Version : 2.06 2024-02-05T04:41:28.319 INFO:teuthology.orchestra.run.smithi196.stdout:Release : 68.el9 2024-02-05T04:41:28.319 INFO:teuthology.orchestra.run.smithi196.stdout:Architecture: x86_64 2024-02-05T04:41:28.319 INFO:teuthology.orchestra.run.smithi196.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-02-05T04:41:28.319 INFO:teuthology.orchestra.run.smithi196.stdout:Group : Unspecified 2024-02-05T04:41:28.319 INFO:teuthology.orchestra.run.smithi196.stdout:Size : 8264254 2024-02-05T04:41:28.320 INFO:teuthology.orchestra.run.smithi196.stdout:License : GPLv3+ 2024-02-05T04:41:28.320 INFO:teuthology.orchestra.run.smithi196.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-02-05T04:41:28.320 INFO:teuthology.orchestra.run.smithi196.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-02-05T04:41:28.320 INFO:teuthology.orchestra.run.smithi196.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-02-05T04:41:28.320 INFO:teuthology.orchestra.run.smithi196.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-02-05T04:41:28.320 INFO:teuthology.orchestra.run.smithi196.stdout:Packager : builder@centos.org 2024-02-05T04:41:28.320 INFO:teuthology.orchestra.run.smithi196.stdout:Vendor : CentOS 2024-02-05T04:41:28.320 INFO:teuthology.orchestra.run.smithi196.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-05T04:41:28.320 INFO:teuthology.orchestra.run.smithi196.stdout:Summary : Support tools for GRUB. 2024-02-05T04:41:28.320 INFO:teuthology.orchestra.run.smithi196.stdout:Description : 2024-02-05T04:41:28.320 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:41:28.320 INFO:teuthology.orchestra.run.smithi196.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-05T04:41:28.321 INFO:teuthology.orchestra.run.smithi196.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-05T04:41:28.321 INFO:teuthology.orchestra.run.smithi196.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-05T04:41:28.321 INFO:teuthology.orchestra.run.smithi196.stdout:hardware devices. 2024-02-05T04:41:28.321 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:41:28.321 INFO:teuthology.orchestra.run.smithi196.stdout:This subpackage provides tools for support of all platforms. 2024-02-05T04:41:28.322 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-05T04:41:28.322 INFO:teuthology.task.kernel:Updating grub on smithi196 to boot 5.14.0-416.el9.x86_64 2024-02-05T04:41:28.322 DEBUG:teuthology.orchestra.run.smithi196:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-05T04:41:28.901 INFO:teuthology.orchestra.run.smithi196.stderr:Generating grub configuration file ... 2024-02-05T04:41:30.301 INFO:teuthology.orchestra.run.smithi196.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-02-05T04:41:30.336 INFO:teuthology.orchestra.run.smithi196.stderr:done 2024-02-05T04:41:30.338 DEBUG:teuthology.orchestra.run.smithi196:> mktemp 2024-02-05T04:41:30.353 INFO:teuthology.orchestra.run.smithi196.stdout:/tmp/tmp.4hM0ySaJbZ 2024-02-05T04:41:30.353 DEBUG:teuthology.orchestra.run.smithi196:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.4hM0ySaJbZ 2024-02-05T04:41:30.425 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod 0666 /tmp/tmp.4hM0ySaJbZ 2024-02-05T04:41:30.559 DEBUG:teuthology.orchestra.remote:smithi196:/tmp/tmp.4hM0ySaJbZ is 6KB 2024-02-05T04:41:30.607 DEBUG:teuthology.orchestra.run.smithi196:> rm -fr /tmp/tmp.4hM0ySaJbZ 2024-02-05T04:41:30.622 DEBUG:teuthology.orchestra.run.smithi196:> sudo /bin/ls /boot/loader/entries || true 2024-02-05T04:41:30.686 INFO:teuthology.orchestra.run.smithi196.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-02-05T04:41:30.686 INFO:teuthology.orchestra.run.smithi196.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-02-05T04:41:30.686 INFO:teuthology.orchestra.run.smithi196.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-02-05T04:41:30.686 INFO:teuthology.orchestra.run.smithi196.stdout:15949e3c88704300882482eddfd7baae-5.14.0-416.el9.x86_64.conf 2024-02-05T04:41:30.687 DEBUG:teuthology.orchestra.run.smithi196:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-416.el9.x86_64 2024-02-05T04:41:30.830 DEBUG:teuthology.orchestra.run.smithi196:> sudo shutdown -r now 2024-02-05T04:41:36.441 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: kernel-core-5.14.0-416.el9.x86_64 4/4 2024-02-05T04:41:36.442 INFO:teuthology.orchestra.run.smithi062.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-02-05T04:41:36.442 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:41:36.513 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: kernel-modules-5.14.0-416.el9.x86_64 4/4 2024-02-05T04:41:36.934 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: kernel-5.14.0-416.el9.x86_64 4/4 2024-02-05T04:41:36.934 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : kernel-5.14.0-416.el9.x86_64 1/4 2024-02-05T04:41:36.934 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : kernel-core-5.14.0-416.el9.x86_64 2/4 2024-02-05T04:41:36.934 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : kernel-modules-5.14.0-416.el9.x86_64 3/4 2024-02-05T04:41:37.365 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : kernel-modules-core-5.14.0-416.el9.x86_64 4/4 2024-02-05T04:41:37.365 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:41:37.366 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2024-02-05T04:41:37.366 INFO:teuthology.orchestra.run.smithi062.stdout: kernel-5.14.0-416.el9.x86_64 2024-02-05T04:41:37.366 INFO:teuthology.orchestra.run.smithi062.stdout: kernel-core-5.14.0-416.el9.x86_64 2024-02-05T04:41:37.366 INFO:teuthology.orchestra.run.smithi062.stdout: kernel-modules-5.14.0-416.el9.x86_64 2024-02-05T04:41:37.366 INFO:teuthology.orchestra.run.smithi062.stdout: kernel-modules-core-5.14.0-416.el9.x86_64 2024-02-05T04:41:37.366 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:41:37.366 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-02-05T04:41:37.869 DEBUG:teuthology.orchestra.run.smithi062:> rpm -q kernel | sort -rV | head -n 1 2024-02-05T04:41:37.994 INFO:teuthology.orchestra.run.smithi062.stdout:kernel-5.14.0-416.el9.x86_64 2024-02-05T04:41:37.994 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-416.el9.x86_64 2024-02-05T04:41:37.994 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-416.el9.x86_64 2024-02-05T04:41:37.994 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-05T04:41:37.994 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-05T04:41:37.994 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-02-05T04:41:37.994 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi062.front.sepia.ceph.com, path=None, version=distro) 2024-02-05T04:41:37.995 DEBUG:teuthology.orchestra.run.smithi062:> rpm -q kernel | sort -rV | head -n 1 2024-02-05T04:41:38.107 INFO:teuthology.orchestra.run.smithi062.stdout:kernel-5.14.0-416.el9.x86_64 2024-02-05T04:41:38.107 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-416.el9.x86_64 2024-02-05T04:41:38.107 DEBUG:teuthology.orchestra.run.smithi062:> sudo rpm -qi grub2-tools 2024-02-05T04:41:38.145 INFO:teuthology.orchestra.run.smithi062.stdout:Name : grub2-tools 2024-02-05T04:41:38.145 INFO:teuthology.orchestra.run.smithi062.stdout:Epoch : 1 2024-02-05T04:41:38.145 INFO:teuthology.orchestra.run.smithi062.stdout:Version : 2.06 2024-02-05T04:41:38.145 INFO:teuthology.orchestra.run.smithi062.stdout:Release : 68.el9 2024-02-05T04:41:38.145 INFO:teuthology.orchestra.run.smithi062.stdout:Architecture: x86_64 2024-02-05T04:41:38.145 INFO:teuthology.orchestra.run.smithi062.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-02-05T04:41:38.145 INFO:teuthology.orchestra.run.smithi062.stdout:Group : Unspecified 2024-02-05T04:41:38.145 INFO:teuthology.orchestra.run.smithi062.stdout:Size : 8264254 2024-02-05T04:41:38.145 INFO:teuthology.orchestra.run.smithi062.stdout:License : GPLv3+ 2024-02-05T04:41:38.145 INFO:teuthology.orchestra.run.smithi062.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-02-05T04:41:38.146 INFO:teuthology.orchestra.run.smithi062.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-02-05T04:41:38.146 INFO:teuthology.orchestra.run.smithi062.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-02-05T04:41:38.146 INFO:teuthology.orchestra.run.smithi062.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-02-05T04:41:38.146 INFO:teuthology.orchestra.run.smithi062.stdout:Packager : builder@centos.org 2024-02-05T04:41:38.146 INFO:teuthology.orchestra.run.smithi062.stdout:Vendor : CentOS 2024-02-05T04:41:38.146 INFO:teuthology.orchestra.run.smithi062.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-05T04:41:38.146 INFO:teuthology.orchestra.run.smithi062.stdout:Summary : Support tools for GRUB. 2024-02-05T04:41:38.146 INFO:teuthology.orchestra.run.smithi062.stdout:Description : 2024-02-05T04:41:38.146 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:41:38.146 INFO:teuthology.orchestra.run.smithi062.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-05T04:41:38.146 INFO:teuthology.orchestra.run.smithi062.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-05T04:41:38.146 INFO:teuthology.orchestra.run.smithi062.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-05T04:41:38.146 INFO:teuthology.orchestra.run.smithi062.stdout:hardware devices. 2024-02-05T04:41:38.147 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:41:38.147 INFO:teuthology.orchestra.run.smithi062.stdout:This subpackage provides tools for support of all platforms. 2024-02-05T04:41:38.147 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-05T04:41:38.148 INFO:teuthology.task.kernel:Updating grub on smithi062 to boot 5.14.0-416.el9.x86_64 2024-02-05T04:41:38.148 DEBUG:teuthology.orchestra.run.smithi062:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-05T04:41:38.720 INFO:teuthology.orchestra.run.smithi062.stderr:Generating grub configuration file ... 2024-02-05T04:41:40.178 INFO:teuthology.orchestra.run.smithi062.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-02-05T04:41:40.207 INFO:teuthology.orchestra.run.smithi062.stderr:done 2024-02-05T04:41:40.210 DEBUG:teuthology.orchestra.run.smithi062:> mktemp 2024-02-05T04:41:40.225 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/tmp.xQieR1qGW2 2024-02-05T04:41:40.225 DEBUG:teuthology.orchestra.run.smithi062:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.xQieR1qGW2 2024-02-05T04:41:40.299 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 0666 /tmp/tmp.xQieR1qGW2 2024-02-05T04:41:40.444 DEBUG:teuthology.orchestra.remote:smithi062:/tmp/tmp.xQieR1qGW2 is 6KB 2024-02-05T04:41:40.493 DEBUG:teuthology.orchestra.run.smithi062:> rm -fr /tmp/tmp.xQieR1qGW2 2024-02-05T04:41:40.508 DEBUG:teuthology.orchestra.run.smithi062:> sudo /bin/ls /boot/loader/entries || true 2024-02-05T04:41:40.576 INFO:teuthology.orchestra.run.smithi062.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-02-05T04:41:40.576 INFO:teuthology.orchestra.run.smithi062.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-02-05T04:41:40.576 INFO:teuthology.orchestra.run.smithi062.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-02-05T04:41:40.576 INFO:teuthology.orchestra.run.smithi062.stdout:15949e3c88704300882482eddfd7baae-5.14.0-416.el9.x86_64.conf 2024-02-05T04:41:40.577 DEBUG:teuthology.orchestra.run.smithi062:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-416.el9.x86_64 2024-02-05T04:41:40.735 DEBUG:teuthology.orchestra.run.smithi062:> sudo shutdown -r now 2024-02-05T04:41:50.574 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: kernel-core-5.14.0-416.el9.x86_64 4/4 2024-02-05T04:41:50.575 INFO:teuthology.orchestra.run.smithi150.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-02-05T04:41:50.575 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:41:50.642 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: kernel-modules-5.14.0-416.el9.x86_64 4/4 2024-02-05T04:41:51.053 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: kernel-5.14.0-416.el9.x86_64 4/4 2024-02-05T04:41:51.053 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : kernel-5.14.0-416.el9.x86_64 1/4 2024-02-05T04:41:51.053 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : kernel-core-5.14.0-416.el9.x86_64 2/4 2024-02-05T04:41:51.053 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : kernel-modules-5.14.0-416.el9.x86_64 3/4 2024-02-05T04:41:51.471 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : kernel-modules-core-5.14.0-416.el9.x86_64 4/4 2024-02-05T04:41:51.471 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:41:51.472 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2024-02-05T04:41:51.472 INFO:teuthology.orchestra.run.smithi150.stdout: kernel-5.14.0-416.el9.x86_64 2024-02-05T04:41:51.472 INFO:teuthology.orchestra.run.smithi150.stdout: kernel-core-5.14.0-416.el9.x86_64 2024-02-05T04:41:51.472 INFO:teuthology.orchestra.run.smithi150.stdout: kernel-modules-5.14.0-416.el9.x86_64 2024-02-05T04:41:51.472 INFO:teuthology.orchestra.run.smithi150.stdout: kernel-modules-core-5.14.0-416.el9.x86_64 2024-02-05T04:41:51.472 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:41:51.472 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-02-05T04:41:51.961 DEBUG:teuthology.orchestra.run.smithi150:> rpm -q kernel | sort -rV | head -n 1 2024-02-05T04:41:52.093 INFO:teuthology.orchestra.run.smithi150.stdout:kernel-5.14.0-416.el9.x86_64 2024-02-05T04:41:52.094 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-416.el9.x86_64 2024-02-05T04:41:52.094 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-416.el9.x86_64 2024-02-05T04:41:52.094 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-05T04:41:52.094 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-05T04:41:52.094 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2024-02-05T04:41:52.094 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi150.front.sepia.ceph.com, path=None, version=distro) 2024-02-05T04:41:52.094 DEBUG:teuthology.orchestra.run.smithi150:> rpm -q kernel | sort -rV | head -n 1 2024-02-05T04:41:52.203 INFO:teuthology.orchestra.run.smithi150.stdout:kernel-5.14.0-416.el9.x86_64 2024-02-05T04:41:52.203 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-416.el9.x86_64 2024-02-05T04:41:52.203 DEBUG:teuthology.orchestra.run.smithi150:> sudo rpm -qi grub2-tools 2024-02-05T04:41:52.237 INFO:teuthology.orchestra.run.smithi150.stdout:Name : grub2-tools 2024-02-05T04:41:52.237 INFO:teuthology.orchestra.run.smithi150.stdout:Epoch : 1 2024-02-05T04:41:52.238 INFO:teuthology.orchestra.run.smithi150.stdout:Version : 2.06 2024-02-05T04:41:52.238 INFO:teuthology.orchestra.run.smithi150.stdout:Release : 68.el9 2024-02-05T04:41:52.238 INFO:teuthology.orchestra.run.smithi150.stdout:Architecture: x86_64 2024-02-05T04:41:52.238 INFO:teuthology.orchestra.run.smithi150.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-02-05T04:41:52.238 INFO:teuthology.orchestra.run.smithi150.stdout:Group : Unspecified 2024-02-05T04:41:52.238 INFO:teuthology.orchestra.run.smithi150.stdout:Size : 8264254 2024-02-05T04:41:52.238 INFO:teuthology.orchestra.run.smithi150.stdout:License : GPLv3+ 2024-02-05T04:41:52.238 INFO:teuthology.orchestra.run.smithi150.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-02-05T04:41:52.238 INFO:teuthology.orchestra.run.smithi150.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-02-05T04:41:52.238 INFO:teuthology.orchestra.run.smithi150.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-02-05T04:41:52.238 INFO:teuthology.orchestra.run.smithi150.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-02-05T04:41:52.238 INFO:teuthology.orchestra.run.smithi150.stdout:Packager : builder@centos.org 2024-02-05T04:41:52.238 INFO:teuthology.orchestra.run.smithi150.stdout:Vendor : CentOS 2024-02-05T04:41:52.238 INFO:teuthology.orchestra.run.smithi150.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-05T04:41:52.238 INFO:teuthology.orchestra.run.smithi150.stdout:Summary : Support tools for GRUB. 2024-02-05T04:41:52.239 INFO:teuthology.orchestra.run.smithi150.stdout:Description : 2024-02-05T04:41:52.239 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:41:52.239 INFO:teuthology.orchestra.run.smithi150.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-05T04:41:52.239 INFO:teuthology.orchestra.run.smithi150.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-05T04:41:52.239 INFO:teuthology.orchestra.run.smithi150.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-05T04:41:52.239 INFO:teuthology.orchestra.run.smithi150.stdout:hardware devices. 2024-02-05T04:41:52.239 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:41:52.239 INFO:teuthology.orchestra.run.smithi150.stdout:This subpackage provides tools for support of all platforms. 2024-02-05T04:41:52.240 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-05T04:41:52.240 INFO:teuthology.task.kernel:Updating grub on smithi150 to boot 5.14.0-416.el9.x86_64 2024-02-05T04:41:52.240 DEBUG:teuthology.orchestra.run.smithi150:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-05T04:41:52.857 INFO:teuthology.orchestra.run.smithi150.stderr:Generating grub configuration file ... 2024-02-05T04:41:54.178 INFO:teuthology.orchestra.run.smithi150.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-02-05T04:41:54.210 INFO:teuthology.orchestra.run.smithi150.stderr:done 2024-02-05T04:41:54.213 DEBUG:teuthology.orchestra.run.smithi150:> mktemp 2024-02-05T04:41:54.227 INFO:teuthology.orchestra.run.smithi150.stdout:/tmp/tmp.UEyZLWIi52 2024-02-05T04:41:54.227 DEBUG:teuthology.orchestra.run.smithi150:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.UEyZLWIi52 2024-02-05T04:41:54.297 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 0666 /tmp/tmp.UEyZLWIi52 2024-02-05T04:41:54.438 DEBUG:teuthology.orchestra.remote:smithi150:/tmp/tmp.UEyZLWIi52 is 6KB 2024-02-05T04:41:54.487 DEBUG:teuthology.orchestra.run.smithi150:> rm -fr /tmp/tmp.UEyZLWIi52 2024-02-05T04:41:54.501 DEBUG:teuthology.orchestra.run.smithi150:> sudo /bin/ls /boot/loader/entries || true 2024-02-05T04:41:54.565 INFO:teuthology.orchestra.run.smithi150.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-02-05T04:41:54.565 INFO:teuthology.orchestra.run.smithi150.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-02-05T04:41:54.565 INFO:teuthology.orchestra.run.smithi150.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-02-05T04:41:54.565 INFO:teuthology.orchestra.run.smithi150.stdout:15949e3c88704300882482eddfd7baae-5.14.0-416.el9.x86_64.conf 2024-02-05T04:41:54.566 DEBUG:teuthology.orchestra.run.smithi150:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-416.el9.x86_64 2024-02-05T04:41:54.696 DEBUG:teuthology.orchestra.run.smithi150:> sudo shutdown -r now 2024-02-05T04:41:57.484 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-02-05T04:41:57.484 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi091.front.sepia.ceph.com' 2024-02-05T04:41:57.485 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi091.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-05T04:42:00.837 INFO:teuthology.task.kernel:Checking client mgr.x for new kernel version... 2024-02-05T04:42:00.837 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-02-05T04:42:00.838 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-05T04:42:10.748 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-02-05T04:42:10.748 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi062.front.sepia.ceph.com' 2024-02-05T04:42:10.749 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi062.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-05T04:42:15.996 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.91 2024-02-05T04:42:24.707 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-02-05T04:42:24.708 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-02-05T04:42:24.709 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-05T04:42:24.998 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi091.front.sepia.ceph.com' 2024-02-05T04:42:24.998 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi091.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-05T04:42:28.060 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.91 2024-02-05T04:42:29.312 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.62 2024-02-05T04:42:38.320 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi062.front.sepia.ceph.com' 2024-02-05T04:42:38.320 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi062.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-05T04:42:40.063 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi091.front.sepia.ceph.com' 2024-02-05T04:42:40.063 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi091.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-05T04:42:41.376 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.62 2024-02-05T04:42:43.132 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2024-02-05T04:42:52.140 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-02-05T04:42:52.141 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-05T04:42:53.378 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi062.front.sepia.ceph.com' 2024-02-05T04:42:53.379 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi062.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-05T04:42:55.196 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2024-02-05T04:43:00.846 DEBUG:teuthology.orchestra.remote:timed out 2024-02-05T04:43:07.204 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-02-05T04:43:07.205 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-05T04:43:09.850 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-02-05T04:43:09.851 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-05T04:43:10.335 DEBUG:teuthology.orchestra.run.smithi196:> true 2024-02-05T04:43:10.876 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-02-05T04:43:10.876 INFO:teuthology.task.kernel:Checking kernel version of mgr.x, want "5.14.0-416.el9.x86_64"... 2024-02-05T04:43:10.877 DEBUG:teuthology.orchestra.run.smithi196:> uname -r 2024-02-05T04:43:10.891 INFO:teuthology.orchestra.run.smithi196.stdout:5.14.0-416.el9.x86_64 2024-02-05T04:43:10.891 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-416.el9.x86_64 vs 5.14.0-416.el9.x86_64 2024-02-05T04:43:10.891 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-05T04:43:10.891 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-05T04:43:11.804 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.62 2024-02-05T04:43:11.892 INFO:teuthology.task.kernel:Enabling kdb on mgr.x... 2024-02-05T04:43:11.892 DEBUG:teuthology.orchestra.run.smithi196:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-05T04:43:11.924 DEBUG:teuthology.orchestra.run.smithi091:> true 2024-02-05T04:43:11.978 INFO:teuthology.orchestra.run.smithi196.stdout:ttyS1 2024-02-05T04:43:12.023 DEBUG:teuthology.parallel:result is None 2024-02-05T04:43:12.344 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi091.front.sepia.ceph.com' 2024-02-05T04:43:12.344 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-416.el9.x86_64"... 2024-02-05T04:43:12.344 DEBUG:teuthology.orchestra.run.smithi091:> uname -r 2024-02-05T04:43:12.357 INFO:teuthology.orchestra.run.smithi091.stdout:5.14.0-416.el9.x86_64 2024-02-05T04:43:12.358 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-416.el9.x86_64 vs 5.14.0-416.el9.x86_64 2024-02-05T04:43:12.358 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-05T04:43:12.358 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-05T04:43:13.358 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-02-05T04:43:13.359 DEBUG:teuthology.orchestra.run.smithi091:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-05T04:43:13.422 INFO:teuthology.orchestra.run.smithi091.stdout:ttyS1 2024-02-05T04:43:13.464 DEBUG:teuthology.parallel:result is None 2024-02-05T04:43:26.805 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi062.front.sepia.ceph.com' 2024-02-05T04:43:26.806 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi062.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-05T04:43:27.267 DEBUG:teuthology.orchestra.run.smithi062:> true 2024-02-05T04:43:27.801 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi062.front.sepia.ceph.com' 2024-02-05T04:43:27.801 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-416.el9.x86_64"... 2024-02-05T04:43:27.802 DEBUG:teuthology.orchestra.run.smithi062:> uname -r 2024-02-05T04:43:27.815 INFO:teuthology.orchestra.run.smithi062.stdout:5.14.0-416.el9.x86_64 2024-02-05T04:43:27.816 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-416.el9.x86_64 vs 5.14.0-416.el9.x86_64 2024-02-05T04:43:27.816 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-05T04:43:27.816 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-05T04:43:28.817 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-02-05T04:43:28.817 DEBUG:teuthology.orchestra.run.smithi062:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-05T04:43:28.922 INFO:teuthology.orchestra.run.smithi062.stdout:ttyS1 2024-02-05T04:43:28.974 DEBUG:teuthology.parallel:result is None 2024-02-05T04:44:07.207 DEBUG:teuthology.orchestra.remote:timed out 2024-02-05T04:44:22.209 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-02-05T04:44:22.210 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-05T04:44:22.503 DEBUG:teuthology.orchestra.run.smithi150:> true 2024-02-05T04:44:22.902 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-02-05T04:44:22.902 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "5.14.0-416.el9.x86_64"... 2024-02-05T04:44:22.902 DEBUG:teuthology.orchestra.run.smithi150:> uname -r 2024-02-05T04:44:22.918 INFO:teuthology.orchestra.run.smithi150.stdout:5.14.0-416.el9.x86_64 2024-02-05T04:44:22.918 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-416.el9.x86_64 vs 5.14.0-416.el9.x86_64 2024-02-05T04:44:22.918 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-05T04:44:22.918 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-05T04:44:23.919 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-02-05T04:44:23.920 DEBUG:teuthology.orchestra.run.smithi150:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-05T04:44:23.993 INFO:teuthology.orchestra.run.smithi150.stdout:ttyS1 2024-02-05T04:44:24.024 DEBUG:teuthology.parallel:result is None 2024-02-05T04:44:24.024 INFO:teuthology.run_tasks:Running task internal.base... 2024-02-05T04:44:24.032 INFO:teuthology.task.internal:Creating test directory... 2024-02-05T04:44:24.032 DEBUG:teuthology.orchestra.run.smithi062:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-05T04:44:24.035 DEBUG:teuthology.orchestra.run.smithi091:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-05T04:44:24.038 DEBUG:teuthology.orchestra.run.smithi150:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-05T04:44:24.040 DEBUG:teuthology.orchestra.run.smithi196:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-05T04:44:24.060 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-02-05T04:44:24.066 INFO:teuthology.run_tasks:Running task internal.archive... 2024-02-05T04:44:24.084 INFO:teuthology.task.internal:Creating archive directory... 2024-02-05T04:44:24.084 DEBUG:teuthology.orchestra.run.smithi062:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-05T04:44:24.092 DEBUG:teuthology.orchestra.run.smithi091:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-05T04:44:24.096 DEBUG:teuthology.orchestra.run.smithi150:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-05T04:44:24.099 DEBUG:teuthology.orchestra.run.smithi196:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-05T04:44:24.128 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-02-05T04:44:24.135 INFO:teuthology.task.internal:Enabling coredump saving... 2024-02-05T04:44:24.135 DEBUG:teuthology.orchestra.run.smithi062:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-05T04:44:24.155 DEBUG:teuthology.orchestra.run.smithi091:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-05T04:44:24.162 DEBUG:teuthology.orchestra.run.smithi150:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-05T04:44:24.164 DEBUG:teuthology.orchestra.run.smithi196:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-05T04:44:24.188 INFO:teuthology.orchestra.run.smithi062.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T04:44:24.197 INFO:teuthology.orchestra.run.smithi150.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T04:44:24.201 INFO:teuthology.orchestra.run.smithi091.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T04:44:24.202 INFO:teuthology.orchestra.run.smithi062.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T04:44:24.203 INFO:teuthology.orchestra.run.smithi196.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T04:44:24.211 INFO:teuthology.orchestra.run.smithi150.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T04:44:24.215 INFO:teuthology.orchestra.run.smithi091.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T04:44:24.217 INFO:teuthology.orchestra.run.smithi196.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T04:44:24.218 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-02-05T04:44:24.225 INFO:teuthology.task.internal:Configuring sudo... 2024-02-05T04:44:24.225 DEBUG:teuthology.orchestra.run.smithi062:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-05T04:44:24.247 DEBUG:teuthology.orchestra.run.smithi091:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-05T04:44:24.259 DEBUG:teuthology.orchestra.run.smithi150:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-05T04:44:24.261 DEBUG:teuthology.orchestra.run.smithi196:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-05T04:44:24.291 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-02-05T04:44:24.301 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-02-05T04:44:24.301 DEBUG:teuthology.orchestra.run.smithi062:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-05T04:44:24.316 DEBUG:teuthology.orchestra.run.smithi091:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-05T04:44:24.331 DEBUG:teuthology.orchestra.run.smithi150:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-05T04:44:24.333 DEBUG:teuthology.orchestra.run.smithi196:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-05T04:44:24.349 DEBUG:teuthology.orchestra.run.smithi062:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-05T04:44:24.411 DEBUG:teuthology.orchestra.run.smithi062:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-05T04:44:24.462 DEBUG:teuthology.orchestra.run.smithi062:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-05T04:44:24.545 DEBUG:teuthology.orchestra.run.smithi062:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-05T04:44:24.586 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-02-05T04:44:24.586 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-05T04:44:24.653 DEBUG:teuthology.orchestra.run.smithi091:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-05T04:44:24.696 DEBUG:teuthology.orchestra.run.smithi091:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-05T04:44:24.752 DEBUG:teuthology.orchestra.run.smithi091:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-05T04:44:24.833 DEBUG:teuthology.orchestra.run.smithi091:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-05T04:44:24.879 DEBUG:teuthology.orchestra.run.smithi091:> set -ex 2024-02-05T04:44:24.880 DEBUG:teuthology.orchestra.run.smithi091:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-05T04:44:24.948 DEBUG:teuthology.orchestra.run.smithi150:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-05T04:44:24.993 DEBUG:teuthology.orchestra.run.smithi150:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-05T04:44:25.048 DEBUG:teuthology.orchestra.run.smithi150:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-05T04:44:25.135 DEBUG:teuthology.orchestra.run.smithi150:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-05T04:44:25.179 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-02-05T04:44:25.179 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-05T04:44:25.248 DEBUG:teuthology.orchestra.run.smithi196:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-05T04:44:25.292 DEBUG:teuthology.orchestra.run.smithi196:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-05T04:44:25.354 DEBUG:teuthology.orchestra.run.smithi196:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-05T04:44:25.440 DEBUG:teuthology.orchestra.run.smithi196:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-05T04:44:25.484 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-02-05T04:44:25.485 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-05T04:44:25.554 DEBUG:teuthology.orchestra.run.smithi062:> sudo service rsyslog restart 2024-02-05T04:44:25.556 DEBUG:teuthology.orchestra.run.smithi091:> sudo service rsyslog restart 2024-02-05T04:44:25.559 DEBUG:teuthology.orchestra.run.smithi150:> sudo service rsyslog restart 2024-02-05T04:44:25.561 DEBUG:teuthology.orchestra.run.smithi196:> sudo service rsyslog restart 2024-02-05T04:44:25.608 INFO:teuthology.orchestra.run.smithi091.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-05T04:44:25.615 INFO:teuthology.orchestra.run.smithi062.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-05T04:44:25.632 INFO:teuthology.orchestra.run.smithi150.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-05T04:44:25.644 INFO:teuthology.orchestra.run.smithi196.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-05T04:44:25.996 INFO:teuthology.run_tasks:Running task internal.timer... 2024-02-05T04:44:26.003 INFO:teuthology.task.internal:Starting timer... 2024-02-05T04:44:26.003 INFO:teuthology.run_tasks:Running task pcp... 2024-02-05T04:44:26.050 INFO:teuthology.run_tasks:Running task selinux... 2024-02-05T04:44:26.062 DEBUG:teuthology.orchestra.run.smithi062:> sudo service auditd rotate 2024-02-05T04:44:26.106 INFO:teuthology.orchestra.run.smithi062.stdout:Rotating logs: 2024-02-05T04:44:26.108 DEBUG:teuthology.orchestra.run.smithi091:> sudo service auditd rotate 2024-02-05T04:44:26.155 INFO:teuthology.orchestra.run.smithi091.stdout:Rotating logs: 2024-02-05T04:44:26.157 DEBUG:teuthology.orchestra.run.smithi150:> sudo service auditd rotate 2024-02-05T04:44:26.226 INFO:teuthology.orchestra.run.smithi150.stdout:Rotating logs: 2024-02-05T04:44:26.228 DEBUG:teuthology.orchestra.run.smithi196:> sudo service auditd rotate 2024-02-05T04:44:26.287 INFO:teuthology.orchestra.run.smithi196.stdout:Rotating logs: 2024-02-05T04:44:26.289 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-02-05T04:44:26.289 DEBUG:teuthology.orchestra.run.smithi062:> /usr/sbin/getenforce 2024-02-05T04:44:26.310 INFO:teuthology.orchestra.run.smithi062.stdout:Permissive 2024-02-05T04:44:26.310 DEBUG:teuthology.orchestra.run.smithi091:> /usr/sbin/getenforce 2024-02-05T04:44:26.342 INFO:teuthology.orchestra.run.smithi091.stdout:Permissive 2024-02-05T04:44:26.343 DEBUG:teuthology.orchestra.run.smithi150:> /usr/sbin/getenforce 2024-02-05T04:44:26.374 INFO:teuthology.orchestra.run.smithi150.stdout:Permissive 2024-02-05T04:44:26.374 DEBUG:teuthology.orchestra.run.smithi196:> /usr/sbin/getenforce 2024-02-05T04:44:26.397 INFO:teuthology.orchestra.run.smithi196.stdout:Permissive 2024-02-05T04:44:26.397 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi062.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi091.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi150.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi196.front.sepia.ceph.com': 'permissive'} 2024-02-05T04:44:26.397 DEBUG:teuthology.orchestra.run.smithi062:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-05T04:44:26.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T04:44:26.426 DEBUG:teuthology.orchestra.run.smithi091:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-05T04:44:26.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T04:44:26.456 DEBUG:teuthology.orchestra.run.smithi150:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-05T04:44:26.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T04:44:26.489 DEBUG:teuthology.orchestra.run.smithi196:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-05T04:44:26.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T04:44:26.517 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-02-05T04:44:26.518 DEBUG:teuthology.orchestra.run.smithi062:> sudo /usr/sbin/setenforce permissive 2024-02-05T04:44:26.553 DEBUG:teuthology.orchestra.run.smithi091:> sudo /usr/sbin/setenforce permissive 2024-02-05T04:44:26.594 DEBUG:teuthology.orchestra.run.smithi150:> sudo /usr/sbin/setenforce permissive 2024-02-05T04:44:26.626 DEBUG:teuthology.orchestra.run.smithi196:> sudo /usr/sbin/setenforce permissive 2024-02-05T04:44:26.665 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-02-05T04:44:26.675 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-02-05T04:44:26.681 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-02-05T04:44:26.804 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-02-05T04:44:26.819 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-02-05T04:44:26.820 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi062.front.sepia.ceph.com,smithi091.front.sepia.ceph.com,smithi150.front.sepia.ceph.com,smithi196.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-02-05T04:48:38.903 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi062.front.sepia.ceph.com'), Remote(name='ubuntu@smithi091.front.sepia.ceph.com'), Remote(name='ubuntu@smithi150.front.sepia.ceph.com'), Remote(name='ubuntu@smithi196.front.sepia.ceph.com')] 2024-02-05T04:48:38.905 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi062.front.sepia.ceph.com' 2024-02-05T04:48:38.906 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi062.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-05T04:48:38.997 DEBUG:teuthology.orchestra.run.smithi062:> true 2024-02-05T04:48:39.072 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi062.front.sepia.ceph.com' 2024-02-05T04:48:39.072 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi091.front.sepia.ceph.com' 2024-02-05T04:48:39.073 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi091.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-05T04:48:39.164 DEBUG:teuthology.orchestra.run.smithi091:> true 2024-02-05T04:48:39.239 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi091.front.sepia.ceph.com' 2024-02-05T04:48:39.239 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-02-05T04:48:39.239 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-05T04:48:39.334 DEBUG:teuthology.orchestra.run.smithi150:> true 2024-02-05T04:48:39.417 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-02-05T04:48:39.418 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-02-05T04:48:39.418 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-05T04:48:39.520 DEBUG:teuthology.orchestra.run.smithi196:> true 2024-02-05T04:48:39.598 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-02-05T04:48:39.599 INFO:teuthology.run_tasks:Running task clock... 2024-02-05T04:48:39.616 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-02-05T04:48:39.616 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-05T04:48:39.617 DEBUG:teuthology.orchestra.run.smithi062:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-05T04:48:39.620 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-05T04:48:39.620 DEBUG:teuthology.orchestra.run.smithi091:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-05T04:48:39.624 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-05T04:48:39.624 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-05T04:48:39.627 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-05T04:48:39.627 DEBUG:teuthology.orchestra.run.smithi196:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-05T04:48:39.652 INFO:teuthology.orchestra.run.smithi062.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-05T04:48:39.655 INFO:teuthology.orchestra.run.smithi091.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-05T04:48:39.661 INFO:teuthology.orchestra.run.smithi150.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-05T04:48:39.670 INFO:teuthology.orchestra.run.smithi062.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-05T04:48:39.671 INFO:teuthology.orchestra.run.smithi196.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-05T04:48:39.672 INFO:teuthology.orchestra.run.smithi091.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-05T04:48:39.679 INFO:teuthology.orchestra.run.smithi150.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-05T04:48:39.689 INFO:teuthology.orchestra.run.smithi196.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-05T04:48:39.701 INFO:teuthology.orchestra.run.smithi062.stderr:sudo: ntpd: command not found 2024-02-05T04:48:39.702 INFO:teuthology.orchestra.run.smithi091.stderr:sudo: ntpd: command not found 2024-02-05T04:48:39.710 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: ntpd: command not found 2024-02-05T04:48:39.715 INFO:teuthology.orchestra.run.smithi062.stdout:506 Cannot talk to daemon 2024-02-05T04:48:39.717 INFO:teuthology.orchestra.run.smithi091.stdout:506 Cannot talk to daemon 2024-02-05T04:48:39.719 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: ntpd: command not found 2024-02-05T04:48:39.724 INFO:teuthology.orchestra.run.smithi150.stdout:506 Cannot talk to daemon 2024-02-05T04:48:39.732 INFO:teuthology.orchestra.run.smithi062.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-05T04:48:39.733 INFO:teuthology.orchestra.run.smithi196.stdout:506 Cannot talk to daemon 2024-02-05T04:48:39.734 INFO:teuthology.orchestra.run.smithi091.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-05T04:48:39.742 INFO:teuthology.orchestra.run.smithi150.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-05T04:48:39.749 INFO:teuthology.orchestra.run.smithi062.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-05T04:48:39.749 INFO:teuthology.orchestra.run.smithi091.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-05T04:48:39.750 INFO:teuthology.orchestra.run.smithi196.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-05T04:48:39.758 INFO:teuthology.orchestra.run.smithi150.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-05T04:48:39.766 INFO:teuthology.orchestra.run.smithi196.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-05T04:48:39.805 INFO:teuthology.orchestra.run.smithi091.stderr:bash: line 1: ntpq: command not found 2024-02-05T04:48:39.806 INFO:teuthology.orchestra.run.smithi062.stderr:bash: line 1: ntpq: command not found 2024-02-05T04:48:39.809 INFO:teuthology.orchestra.run.smithi091.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-05T04:48:39.809 INFO:teuthology.orchestra.run.smithi091.stdout:=============================================================================== 2024-02-05T04:48:39.810 INFO:teuthology.orchestra.run.smithi091.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-05T04:48:39.810 INFO:teuthology.orchestra.run.smithi091.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-05T04:48:39.810 INFO:teuthology.orchestra.run.smithi091.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-05T04:48:39.810 INFO:teuthology.orchestra.run.smithi091.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-05T04:48:39.811 INFO:teuthology.orchestra.run.smithi062.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-05T04:48:39.811 INFO:teuthology.orchestra.run.smithi062.stdout:=============================================================================== 2024-02-05T04:48:39.811 INFO:teuthology.orchestra.run.smithi062.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-05T04:48:39.811 INFO:teuthology.orchestra.run.smithi062.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-05T04:48:39.811 INFO:teuthology.orchestra.run.smithi062.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-05T04:48:39.811 INFO:teuthology.orchestra.run.smithi062.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-05T04:48:39.812 INFO:teuthology.orchestra.run.smithi150.stderr:bash: line 1: ntpq: command not found 2024-02-05T04:48:39.815 INFO:teuthology.orchestra.run.smithi150.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-05T04:48:39.815 INFO:teuthology.orchestra.run.smithi150.stdout:=============================================================================== 2024-02-05T04:48:39.816 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-05T04:48:39.816 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-05T04:48:39.816 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-05T04:48:39.816 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-05T04:48:39.817 INFO:teuthology.orchestra.run.smithi196.stderr:bash: line 1: ntpq: command not found 2024-02-05T04:48:39.820 INFO:teuthology.orchestra.run.smithi196.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-05T04:48:39.820 INFO:teuthology.orchestra.run.smithi196.stdout:=============================================================================== 2024-02-05T04:48:39.820 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-05T04:48:39.820 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-05T04:48:39.820 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-05T04:48:39.821 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-05T04:48:39.822 INFO:teuthology.run_tasks:Running task install... 2024-02-05T04:48:39.830 DEBUG:teuthology.task.install:project ceph 2024-02-05T04:48:39.830 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'd4ed20cfb238ef2c47d64ca4533eca00b8968b98'}} 2024-02-05T04:48:39.830 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'd4ed20cfb238ef2c47d64ca4533eca00b8968b98'} 2024-02-05T04:48:39.830 INFO:teuthology.task.install:Using flavor: default 2024-02-05T04:48:39.836 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-02-05T04:48:39.836 INFO:teuthology.task.install:extra packages: [] 2024-02-05T04:48:39.837 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd4ed20cfb238ef2c47d64ca4533eca00b8968b98', 'tag': None, 'wait_for_package': False} 2024-02-05T04:48:39.837 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d4ed20cfb238ef2c47d64ca4533eca00b8968b98 2024-02-05T04:48:39.839 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd4ed20cfb238ef2c47d64ca4533eca00b8968b98', 'tag': None, 'wait_for_package': False} 2024-02-05T04:48:39.840 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d4ed20cfb238ef2c47d64ca4533eca00b8968b98 2024-02-05T04:48:39.840 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd4ed20cfb238ef2c47d64ca4533eca00b8968b98', 'tag': None, 'wait_for_package': False} 2024-02-05T04:48:39.841 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d4ed20cfb238ef2c47d64ca4533eca00b8968b98 2024-02-05T04:48:39.842 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd4ed20cfb238ef2c47d64ca4533eca00b8968b98', 'tag': None, 'wait_for_package': False} 2024-02-05T04:48:39.842 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d4ed20cfb238ef2c47d64ca4533eca00b8968b98 2024-02-05T04:48:39.990 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/centos/9/flavors/default/ 2024-02-05T04:48:39.990 INFO:teuthology.task.install.rpm:Package version is 19.0.0-658.gd4ed20cf 2024-02-05T04:48:40.000 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/centos/9/flavors/default/ 2024-02-05T04:48:40.000 INFO:teuthology.task.install.rpm:Package version is 19.0.0-658.gd4ed20cf 2024-02-05T04:48:40.004 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/centos/9/flavors/default/ 2024-02-05T04:48:40.004 INFO:teuthology.task.install.rpm:Package version is 19.0.0-658.gd4ed20cf 2024-02-05T04:48:40.082 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/centos/9/flavors/default/ 2024-02-05T04:48:40.083 INFO:teuthology.task.install.rpm:Package version is 19.0.0-658.gd4ed20cf 2024-02-05T04:48:40.106 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-05T04:48:40.106 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-02-05T04:48:40.106 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-05T04:48:40.125 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-05T04:48:40.125 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-02-05T04:48:40.125 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-05T04:48:40.150 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-05T04:48:40.151 DEBUG:teuthology.orchestra.run.smithi062:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-05T04:48:40.154 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-05T04:48:40.155 DEBUG:teuthology.orchestra.run.smithi150:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-05T04:48:40.174 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-05T04:48:40.174 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-02-05T04:48:40.174 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-05T04:48:40.196 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-05T04:48:40.196 DEBUG:teuthology.orchestra.run.smithi091:> set -ex 2024-02-05T04:48:40.196 DEBUG:teuthology.orchestra.run.smithi091:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-05T04:48:40.203 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-05T04:48:40.203 DEBUG:teuthology.orchestra.run.smithi196:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-05T04:48:40.219 DEBUG:teuthology.orchestra.run.smithi062:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-05T04:48:40.224 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-05T04:48:40.224 DEBUG:teuthology.orchestra.run.smithi091:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-05T04:48:40.225 DEBUG:teuthology.orchestra.run.smithi150:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-05T04:48:40.272 DEBUG:teuthology.orchestra.run.smithi196:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-05T04:48:40.292 DEBUG:teuthology.orchestra.run.smithi091:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-05T04:48:40.299 DEBUG:teuthology.orchestra.run.smithi062:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-05T04:48:40.306 DEBUG:teuthology.orchestra.run.smithi150:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-05T04:48:40.351 DEBUG:teuthology.orchestra.run.smithi196:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-05T04:48:40.364 INFO:teuthology.orchestra.run.smithi062.stdout:check_obsoletes = 1 2024-02-05T04:48:40.366 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum clean all 2024-02-05T04:48:40.371 DEBUG:teuthology.orchestra.run.smithi091:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-05T04:48:40.373 INFO:teuthology.orchestra.run.smithi150.stdout:check_obsoletes = 1 2024-02-05T04:48:40.375 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum clean all 2024-02-05T04:48:40.421 INFO:teuthology.orchestra.run.smithi196.stdout:check_obsoletes = 1 2024-02-05T04:48:40.422 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum clean all 2024-02-05T04:48:40.438 INFO:teuthology.orchestra.run.smithi091.stdout:check_obsoletes = 1 2024-02-05T04:48:40.440 DEBUG:teuthology.orchestra.run.smithi091:> sudo yum clean all 2024-02-05T04:48:40.609 INFO:teuthology.orchestra.run.smithi150.stdout:74 files removed 2024-02-05T04:48:40.619 INFO:teuthology.orchestra.run.smithi062.stdout:74 files removed 2024-02-05T04:48:40.637 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-radosgw 2024-02-05T04:48:40.646 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install ceph-radosgw 2024-02-05T04:48:40.669 INFO:teuthology.orchestra.run.smithi196.stdout:74 files removed 2024-02-05T04:48:40.682 INFO:teuthology.orchestra.run.smithi091.stdout:74 files removed 2024-02-05T04:48:40.696 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-radosgw 2024-02-05T04:48:40.709 DEBUG:teuthology.orchestra.run.smithi091:> sudo yum -y install ceph-radosgw 2024-02-05T04:48:41.332 INFO:teuthology.orchestra.run.smithi062.stdout:ceph packages for x86_64 218 kB/s | 80 kB 00:00 2024-02-05T04:48:41.431 INFO:teuthology.orchestra.run.smithi150.stdout:ceph packages for x86_64 168 kB/s | 80 kB 00:00 2024-02-05T04:48:41.452 INFO:teuthology.orchestra.run.smithi196.stdout:ceph packages for x86_64 182 kB/s | 80 kB 00:00 2024-02-05T04:48:41.458 INFO:teuthology.orchestra.run.smithi091.stdout:ceph packages for x86_64 184 kB/s | 80 kB 00:00 2024-02-05T04:48:41.816 INFO:teuthology.orchestra.run.smithi062.stdout:ceph noarch packages 22 kB/s | 9.5 kB 00:00 2024-02-05T04:48:41.869 INFO:teuthology.orchestra.run.smithi196.stdout:ceph noarch packages 26 kB/s | 9.5 kB 00:00 2024-02-05T04:48:41.931 INFO:teuthology.orchestra.run.smithi150.stdout:ceph noarch packages 21 kB/s | 9.5 kB 00:00 2024-02-05T04:48:41.950 INFO:teuthology.orchestra.run.smithi091.stdout:ceph noarch packages 21 kB/s | 9.5 kB 00:00 2024-02-05T04:48:42.191 INFO:teuthology.orchestra.run.smithi062.stdout:ceph source packages 5.6 kB/s | 1.9 kB 00:00 2024-02-05T04:48:42.220 INFO:teuthology.orchestra.run.smithi196.stdout:ceph source packages 6.0 kB/s | 1.9 kB 00:00 2024-02-05T04:48:42.257 INFO:teuthology.orchestra.run.smithi150.stdout:ceph source packages 6.5 kB/s | 1.9 kB 00:00 2024-02-05T04:48:42.367 INFO:teuthology.orchestra.run.smithi091.stdout:ceph source packages 4.9 kB/s | 1.9 kB 00:00 2024-02-05T04:48:42.619 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS-9 - AppStream 17 MB/s | 6.6 MB 00:00 2024-02-05T04:48:42.656 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS-9 - AppStream 17 MB/s | 6.6 MB 00:00 2024-02-05T04:48:42.759 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS-9 - AppStream 14 MB/s | 6.6 MB 00:00 2024-02-05T04:48:42.820 INFO:teuthology.orchestra.run.smithi091.stdout:CentOS-9 - AppStream 16 MB/s | 6.6 MB 00:00 2024-02-05T04:48:44.052 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS-9 - BaseOS 7.2 MB/s | 2.0 MB 00:00 2024-02-05T04:48:44.110 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS-9 - BaseOS 7.3 MB/s | 2.0 MB 00:00 2024-02-05T04:48:44.179 INFO:teuthology.orchestra.run.smithi091.stdout:CentOS-9 - BaseOS 7.3 MB/s | 2.0 MB 00:00 2024-02-05T04:48:44.197 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS-9 - BaseOS 4.6 MB/s | 2.0 MB 00:00 2024-02-05T04:48:45.313 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.0 MB 00:00 2024-02-05T04:48:45.383 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - BaseOS 8.4 MB/s | 8.0 MB 00:00 2024-02-05T04:48:46.038 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS Stream 9 - BaseOS 4.8 MB/s | 8.0 MB 00:01 2024-02-05T04:48:47.468 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - AppStream 18 MB/s | 19 MB 00:01 2024-02-05T04:48:47.840 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - AppStream 13 MB/s | 19 MB 00:01 2024-02-05T04:48:49.213 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS Stream 9 - AppStream 9.2 MB/s | 19 MB 00:02 2024-02-05T04:48:51.859 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - CRB 10 MB/s | 5.9 MB 00:00 2024-02-05T04:48:52.126 INFO:teuthology.orchestra.run.smithi091.stdout:CentOS Stream 9 - BaseOS 1.0 MB/s | 8.0 MB 00:07 2024-02-05T04:48:53.390 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - Extras packages 36 kB/s | 15 kB 00:00 2024-02-05T04:48:53.803 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS Stream 9 - CRB 6.3 MB/s | 5.9 MB 00:00 2024-02-05T04:48:54.267 INFO:teuthology.orchestra.run.smithi196.stdout:Extra Packages for Enterprise Linux 27 MB/s | 20 MB 00:00 2024-02-05T04:48:56.878 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS Stream 9 - Extras packages 7.8 kB/s | 15 kB 00:01 2024-02-05T04:48:57.426 INFO:teuthology.orchestra.run.smithi091.stdout:CentOS Stream 9 - AppStream 4.4 MB/s | 19 MB 00:04 2024-02-05T04:48:59.142 INFO:teuthology.orchestra.run.smithi196.stdout:lab-extras 15 kB/s | 1.7 kB 00:00 2024-02-05T04:49:01.235 INFO:teuthology.orchestra.run.smithi091.stdout:CentOS Stream 9 - CRB 16 MB/s | 5.9 MB 00:00 2024-02-05T04:49:01.727 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-02-05T04:49:01.733 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-02-05T04:49:01.733 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-02-05T04:49:01.733 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-02-05T04:49:01.733 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-02-05T04:49:01.733 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-radosgw x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 8.6 M 2024-02-05T04:49:01.733 INFO:teuthology.orchestra.run.smithi196.stdout:Upgrading: 2024-02-05T04:49:01.733 INFO:teuthology.orchestra.run.smithi196.stdout: librados2 x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 3.4 M 2024-02-05T04:49:01.733 INFO:teuthology.orchestra.run.smithi196.stdout: librbd1 x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 3.1 M 2024-02-05T04:49:01.733 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-02-05T04:49:01.733 INFO:teuthology.orchestra.run.smithi196.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-02-05T04:49:01.733 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-base x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 5.1 M 2024-02-05T04:49:01.734 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-common x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 19 M 2024-02-05T04:49:01.734 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-selinux x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 25 k 2024-02-05T04:49:01.734 INFO:teuthology.orchestra.run.smithi196.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-02-05T04:49:01.734 INFO:teuthology.orchestra.run.smithi196.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-02-05T04:49:01.734 INFO:teuthology.orchestra.run.smithi196.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-02-05T04:49:01.734 INFO:teuthology.orchestra.run.smithi196.stdout: libcephfs2 x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 742 k 2024-02-05T04:49:01.734 INFO:teuthology.orchestra.run.smithi196.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-02-05T04:49:01.734 INFO:teuthology.orchestra.run.smithi196.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-02-05T04:49:01.734 INFO:teuthology.orchestra.run.smithi196.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-02-05T04:49:01.734 INFO:teuthology.orchestra.run.smithi196.stdout: libradosstriper1 x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 499 k 2024-02-05T04:49:01.734 INFO:teuthology.orchestra.run.smithi196.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-02-05T04:49:01.734 INFO:teuthology.orchestra.run.smithi196.stdout: librgw2 x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 4.8 M 2024-02-05T04:49:01.734 INFO:teuthology.orchestra.run.smithi196.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-02-05T04:49:01.735 INFO:teuthology.orchestra.run.smithi196.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-02-05T04:49:01.735 INFO:teuthology.orchestra.run.smithi196.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-02-05T04:49:01.735 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ceph-argparse x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 45 k 2024-02-05T04:49:01.735 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ceph-common x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 130 k 2024-02-05T04:49:01.735 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cephfs x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 162 k 2024-02-05T04:49:01.735 INFO:teuthology.orchestra.run.smithi196.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-02-05T04:49:01.735 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rados x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 320 k 2024-02-05T04:49:01.735 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rbd x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 298 k 2024-02-05T04:49:01.735 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rgw x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 100 k 2024-02-05T04:49:01.735 INFO:teuthology.orchestra.run.smithi196.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-02-05T04:49:01.735 INFO:teuthology.orchestra.run.smithi196.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-02-05T04:49:01.735 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:49:01.735 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-02-05T04:49:01.735 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-02-05T04:49:01.735 INFO:teuthology.orchestra.run.smithi196.stdout:Install 27 Packages 2024-02-05T04:49:01.736 INFO:teuthology.orchestra.run.smithi196.stdout:Upgrade 2 Packages 2024-02-05T04:49:01.736 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:49:01.736 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 55 M 2024-02-05T04:49:01.736 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-02-05T04:49:02.083 INFO:teuthology.orchestra.run.smithi062.stdout:Extra Packages for Enterprise Linux 4.0 MB/s | 20 MB 00:05 2024-02-05T04:49:02.752 INFO:teuthology.orchestra.run.smithi091.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 15 kB 00:00 2024-02-05T04:49:03.351 INFO:teuthology.orchestra.run.smithi196.stdout:(1/29): ceph-base-19.0.0-658.gd4ed20cf.el9.x86_ 3.4 MB/s | 5.1 MB 00:01 2024-02-05T04:49:03.410 INFO:teuthology.orchestra.run.smithi196.stdout:(2/29): ceph-selinux-19.0.0-658.gd4ed20cf.el9.x 432 kB/s | 25 kB 00:00 2024-02-05T04:49:03.601 INFO:teuthology.orchestra.run.smithi196.stdout:(3/29): ceph-common-19.0.0-658.gd4ed20cf.el9.x8 11 MB/s | 19 MB 00:01 2024-02-05T04:49:03.643 INFO:teuthology.orchestra.run.smithi196.stdout:(4/29): libcephfs2-19.0.0-658.gd4ed20cf.el9.x86 3.1 MB/s | 742 kB 00:00 2024-02-05T04:49:03.669 INFO:teuthology.orchestra.run.smithi196.stdout:(5/29): libradosstriper1-19.0.0-658.gd4ed20cf.e 7.3 MB/s | 499 kB 00:00 2024-02-05T04:49:03.769 INFO:teuthology.orchestra.run.smithi196.stdout:(6/29): ceph-radosgw-19.0.0-658.gd4ed20cf.el9.x 4.5 MB/s | 8.6 MB 00:01 2024-02-05T04:49:03.794 INFO:teuthology.orchestra.run.smithi196.stdout:(7/29): python3-ceph-argparse-19.0.0-658.gd4ed2 360 kB/s | 45 kB 00:00 2024-02-05T04:49:03.844 INFO:teuthology.orchestra.run.smithi196.stdout:(8/29): python3-ceph-common-19.0.0-658.gd4ed20c 1.7 MB/s | 130 kB 00:00 2024-02-05T04:49:03.869 INFO:teuthology.orchestra.run.smithi196.stdout:(9/29): python3-cephfs-19.0.0-658.gd4ed20cf.el9 2.1 MB/s | 162 kB 00:00 2024-02-05T04:49:03.911 INFO:teuthology.orchestra.run.smithi196.stdout:(10/29): python3-rados-19.0.0-658.gd4ed20cf.el9 4.7 MB/s | 320 kB 00:00 2024-02-05T04:49:03.937 INFO:teuthology.orchestra.run.smithi196.stdout:(11/29): python3-rbd-19.0.0-658.gd4ed20cf.el9.x 4.4 MB/s | 298 kB 00:00 2024-02-05T04:49:03.987 INFO:teuthology.orchestra.run.smithi196.stdout:(12/29): python3-rgw-19.0.0-658.gd4ed20cf.el9.x 1.3 MB/s | 100 kB 00:00 2024-02-05T04:49:04.046 INFO:teuthology.orchestra.run.smithi196.stdout:(13/29): boost-program-options-1.75.0-8.el9.x86 976 kB/s | 106 kB 00:00 2024-02-05T04:49:04.071 INFO:teuthology.orchestra.run.smithi196.stdout:(14/29): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.9 MB/s | 159 kB 00:00 2024-02-05T04:49:04.096 INFO:teuthology.orchestra.run.smithi196.stdout:(15/29): librabbitmq-0.11.0-7.el9.x86_64.rpm 885 kB/s | 44 kB 00:00 2024-02-05T04:49:04.171 INFO:teuthology.orchestra.run.smithi196.stdout:(16/29): librgw2-19.0.0-658.gd4ed20cf.el9.x86_6 9.1 MB/s | 4.8 MB 00:00 2024-02-05T04:49:04.205 INFO:teuthology.orchestra.run.smithi196.stdout:(17/29): librdkafka-1.6.1-102.el9.x86_64.rpm 4.8 MB/s | 661 kB 00:00 2024-02-05T04:49:04.230 INFO:teuthology.orchestra.run.smithi196.stdout:(18/29): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.1 MB/s | 282 kB 00:00 2024-02-05T04:49:04.256 INFO:teuthology.orchestra.run.smithi196.stdout:(19/29): python3-prettytable-0.7.2-27.el9.noarc 497 kB/s | 41 kB 00:00 2024-02-05T04:49:04.756 INFO:teuthology.orchestra.run.smithi196.stdout:(20/29): libarrow-doc-9.0.0-10.el9.noarch.rpm 52 kB/s | 26 kB 00:00 2024-02-05T04:49:04.873 INFO:teuthology.orchestra.run.smithi091.stdout:Extra Packages for Enterprise Linux 10 MB/s | 20 MB 00:01 2024-02-05T04:49:04.939 INFO:teuthology.orchestra.run.smithi196.stdout:(21/29): liboath-2.6.7-2.el9.x86_64.rpm 265 kB/s | 49 kB 00:00 2024-02-05T04:49:05.056 INFO:teuthology.orchestra.run.smithi196.stdout:(22/29): gperftools-libs-2.9.1-2.el9.x86_64.rpm 374 kB/s | 309 kB 00:00 2024-02-05T04:49:05.123 INFO:teuthology.orchestra.run.smithi196.stdout:(23/29): libunwind-1.6.2-1.el9.x86_64.rpm 366 kB/s | 67 kB 00:00 2024-02-05T04:49:05.407 INFO:teuthology.orchestra.run.smithi196.stdout:(24/29): re2-20211101-3.el9.x86_64.rpm 676 kB/s | 192 kB 00:00 2024-02-05T04:49:06.190 INFO:teuthology.orchestra.run.smithi196.stdout:(25/29): parquet-libs-9.0.0-10.el9.x86_64.rpm 739 kB/s | 838 kB 00:01 2024-02-05T04:49:06.293 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - CRB 396 kB/s | 5.9 MB 00:15 2024-02-05T04:49:06.732 INFO:teuthology.orchestra.run.smithi196.stdout:(26/29): librados2-19.0.0-658.gd4ed20cf.el9.x86 6.2 MB/s | 3.4 MB 00:00 2024-02-05T04:49:07.099 INFO:teuthology.orchestra.run.smithi196.stdout:(27/29): librbd1-19.0.0-658.gd4ed20cf.el9.x86_6 8.5 MB/s | 3.1 MB 00:00 2024-02-05T04:49:07.222 INFO:teuthology.orchestra.run.smithi062.stdout:lab-extras 37 kB/s | 1.7 kB 00:00 2024-02-05T04:49:07.557 INFO:teuthology.orchestra.run.smithi196.stdout:(28/29): thrift-0.15.0-2.el9.x86_64.rpm 753 kB/s | 1.6 MB 00:02 2024-02-05T04:49:07.718 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - Extras packages 50 kB/s | 15 kB 00:00 2024-02-05T04:49:09.687 INFO:teuthology.orchestra.run.smithi091.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-02-05T04:49:09.937 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-02-05T04:49:09.944 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-02-05T04:49:09.944 INFO:teuthology.orchestra.run.smithi062.stdout: Package Arch Version Repository Size 2024-02-05T04:49:09.944 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-02-05T04:49:09.945 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2024-02-05T04:49:09.945 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-radosgw x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 8.6 M 2024-02-05T04:49:09.945 INFO:teuthology.orchestra.run.smithi062.stdout:Upgrading: 2024-02-05T04:49:09.945 INFO:teuthology.orchestra.run.smithi062.stdout: librados2 x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 3.4 M 2024-02-05T04:49:09.945 INFO:teuthology.orchestra.run.smithi062.stdout: librbd1 x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 3.1 M 2024-02-05T04:49:09.945 INFO:teuthology.orchestra.run.smithi062.stdout:Installing dependencies: 2024-02-05T04:49:09.945 INFO:teuthology.orchestra.run.smithi062.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-02-05T04:49:09.945 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-base x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 5.1 M 2024-02-05T04:49:09.945 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-common x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 19 M 2024-02-05T04:49:09.945 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-selinux x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 25 k 2024-02-05T04:49:09.945 INFO:teuthology.orchestra.run.smithi062.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-02-05T04:49:09.945 INFO:teuthology.orchestra.run.smithi062.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-02-05T04:49:09.945 INFO:teuthology.orchestra.run.smithi062.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-02-05T04:49:09.946 INFO:teuthology.orchestra.run.smithi062.stdout: libcephfs2 x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 742 k 2024-02-05T04:49:09.946 INFO:teuthology.orchestra.run.smithi062.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-02-05T04:49:09.946 INFO:teuthology.orchestra.run.smithi062.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-02-05T04:49:09.946 INFO:teuthology.orchestra.run.smithi062.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-02-05T04:49:09.946 INFO:teuthology.orchestra.run.smithi062.stdout: libradosstriper1 x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 499 k 2024-02-05T04:49:09.946 INFO:teuthology.orchestra.run.smithi062.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-02-05T04:49:09.946 INFO:teuthology.orchestra.run.smithi062.stdout: librgw2 x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 4.8 M 2024-02-05T04:49:09.946 INFO:teuthology.orchestra.run.smithi062.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-02-05T04:49:09.946 INFO:teuthology.orchestra.run.smithi062.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-02-05T04:49:09.946 INFO:teuthology.orchestra.run.smithi062.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-02-05T04:49:09.946 INFO:teuthology.orchestra.run.smithi062.stdout: python3-ceph-argparse x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 45 k 2024-02-05T04:49:09.946 INFO:teuthology.orchestra.run.smithi062.stdout: python3-ceph-common x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 130 k 2024-02-05T04:49:09.946 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cephfs x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 162 k 2024-02-05T04:49:09.947 INFO:teuthology.orchestra.run.smithi062.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-02-05T04:49:09.947 INFO:teuthology.orchestra.run.smithi062.stdout: python3-rados x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 320 k 2024-02-05T04:49:09.947 INFO:teuthology.orchestra.run.smithi062.stdout: python3-rbd x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 298 k 2024-02-05T04:49:09.947 INFO:teuthology.orchestra.run.smithi062.stdout: python3-rgw x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 100 k 2024-02-05T04:49:09.947 INFO:teuthology.orchestra.run.smithi062.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-02-05T04:49:09.947 INFO:teuthology.orchestra.run.smithi062.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-02-05T04:49:09.947 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:49:09.947 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2024-02-05T04:49:09.947 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-02-05T04:49:09.947 INFO:teuthology.orchestra.run.smithi062.stdout:Install 27 Packages 2024-02-05T04:49:09.947 INFO:teuthology.orchestra.run.smithi062.stdout:Upgrade 2 Packages 2024-02-05T04:49:09.947 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:49:09.948 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 55 M 2024-02-05T04:49:09.948 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2024-02-05T04:49:10.224 INFO:teuthology.orchestra.run.smithi196.stdout:(29/29): libarrow-9.0.0-10.el9.x86_64.rpm 755 kB/s | 4.4 MB 00:05 2024-02-05T04:49:10.231 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:49:10.231 INFO:teuthology.orchestra.run.smithi196.stdout:Total 6.5 MB/s | 55 MB 00:08 2024-02-05T04:49:10.231 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-02-05T04:49:10.266 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-02-05T04:49:10.266 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-02-05T04:49:10.739 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-02-05T04:49:10.740 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-02-05T04:49:11.031 INFO:teuthology.orchestra.run.smithi062.stdout:(1/29): ceph-radosgw-19.0.0-658.gd4ed20cf.el9.x 8.3 MB/s | 8.6 MB 00:01 2024-02-05T04:49:11.098 INFO:teuthology.orchestra.run.smithi062.stdout:(2/29): ceph-selinux-19.0.0-658.gd4ed20cf.el9.x 380 kB/s | 25 kB 00:00 2024-02-05T04:49:11.199 INFO:teuthology.orchestra.run.smithi062.stdout:(3/29): libcephfs2-19.0.0-658.gd4ed20cf.el9.x86 7.2 MB/s | 742 kB 00:00 2024-02-05T04:49:11.290 INFO:teuthology.orchestra.run.smithi062.stdout:(4/29): libradosstriper1-19.0.0-658.gd4ed20cf.e 5.3 MB/s | 499 kB 00:00 2024-02-05T04:49:11.916 INFO:teuthology.orchestra.run.smithi062.stdout:(5/29): ceph-common-19.0.0-658.gd4ed20cf.el9.x8 10 MB/s | 19 MB 00:01 2024-02-05T04:49:11.982 INFO:teuthology.orchestra.run.smithi062.stdout:(6/29): librgw2-19.0.0-658.gd4ed20cf.el9.x86_64 6.9 MB/s | 4.8 MB 00:00 2024-02-05T04:49:11.999 INFO:teuthology.orchestra.run.smithi062.stdout:(7/29): python3-ceph-argparse-19.0.0-658.gd4ed2 541 kB/s | 45 kB 00:00 2024-02-05T04:49:12.025 INFO:teuthology.orchestra.run.smithi062.stdout:(8/29): python3-ceph-common-19.0.0-658.gd4ed20c 3.0 MB/s | 130 kB 00:00 2024-02-05T04:49:12.050 INFO:teuthology.orchestra.run.smithi062.stdout:(9/29): python3-cephfs-19.0.0-658.gd4ed20cf.el9 3.2 MB/s | 162 kB 00:00 2024-02-05T04:49:12.075 INFO:teuthology.orchestra.run.smithi062.stdout:(10/29): python3-rados-19.0.0-658.gd4ed20cf.el9 6.2 MB/s | 320 kB 00:00 2024-02-05T04:49:12.108 INFO:teuthology.orchestra.run.smithi062.stdout:(11/29): python3-rbd-19.0.0-658.gd4ed20cf.el9.x 5.0 MB/s | 298 kB 00:00 2024-02-05T04:49:12.134 INFO:teuthology.orchestra.run.smithi062.stdout:(12/29): python3-rgw-19.0.0-658.gd4ed20cf.el9.x 1.7 MB/s | 100 kB 00:00 2024-02-05T04:49:12.139 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-02-05T04:49:12.219 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/31 2024-02-05T04:49:12.253 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/31 2024-02-05T04:49:12.256 INFO:teuthology.orchestra.run.smithi091.stdout:Dependencies resolved. 2024-02-05T04:49:12.259 INFO:teuthology.orchestra.run.smithi062.stdout:(13/29): boost-program-options-1.75.0-8.el9.x86 705 kB/s | 106 kB 00:00 2024-02-05T04:49:12.270 INFO:teuthology.orchestra.run.smithi091.stdout:================================================================================ 2024-02-05T04:49:12.270 INFO:teuthology.orchestra.run.smithi091.stdout: Package Arch Version Repository Size 2024-02-05T04:49:12.270 INFO:teuthology.orchestra.run.smithi091.stdout:================================================================================ 2024-02-05T04:49:12.270 INFO:teuthology.orchestra.run.smithi091.stdout:Installing: 2024-02-05T04:49:12.270 INFO:teuthology.orchestra.run.smithi091.stdout: ceph-radosgw x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 8.6 M 2024-02-05T04:49:12.270 INFO:teuthology.orchestra.run.smithi091.stdout:Upgrading: 2024-02-05T04:49:12.271 INFO:teuthology.orchestra.run.smithi091.stdout: librados2 x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 3.4 M 2024-02-05T04:49:12.271 INFO:teuthology.orchestra.run.smithi091.stdout: librbd1 x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 3.1 M 2024-02-05T04:49:12.271 INFO:teuthology.orchestra.run.smithi091.stdout:Installing dependencies: 2024-02-05T04:49:12.271 INFO:teuthology.orchestra.run.smithi091.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-02-05T04:49:12.271 INFO:teuthology.orchestra.run.smithi091.stdout: ceph-base x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 5.1 M 2024-02-05T04:49:12.271 INFO:teuthology.orchestra.run.smithi091.stdout: ceph-common x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 19 M 2024-02-05T04:49:12.271 INFO:teuthology.orchestra.run.smithi091.stdout: ceph-selinux x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 25 k 2024-02-05T04:49:12.272 INFO:teuthology.orchestra.run.smithi091.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-02-05T04:49:12.272 INFO:teuthology.orchestra.run.smithi091.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-02-05T04:49:12.272 INFO:teuthology.orchestra.run.smithi091.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-02-05T04:49:12.272 INFO:teuthology.orchestra.run.smithi091.stdout: libcephfs2 x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 742 k 2024-02-05T04:49:12.272 INFO:teuthology.orchestra.run.smithi091.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-02-05T04:49:12.272 INFO:teuthology.orchestra.run.smithi091.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-02-05T04:49:12.272 INFO:teuthology.orchestra.run.smithi091.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-02-05T04:49:12.273 INFO:teuthology.orchestra.run.smithi091.stdout: libradosstriper1 x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 499 k 2024-02-05T04:49:12.273 INFO:teuthology.orchestra.run.smithi091.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-02-05T04:49:12.273 INFO:teuthology.orchestra.run.smithi091.stdout: librgw2 x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 4.8 M 2024-02-05T04:49:12.273 INFO:teuthology.orchestra.run.smithi091.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-02-05T04:49:12.273 INFO:teuthology.orchestra.run.smithi091.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-02-05T04:49:12.273 INFO:teuthology.orchestra.run.smithi091.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-02-05T04:49:12.273 INFO:teuthology.orchestra.run.smithi091.stdout: python3-ceph-argparse x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 45 k 2024-02-05T04:49:12.273 INFO:teuthology.orchestra.run.smithi091.stdout: python3-ceph-common x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 130 k 2024-02-05T04:49:12.274 INFO:teuthology.orchestra.run.smithi091.stdout: python3-cephfs x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 162 k 2024-02-05T04:49:12.274 INFO:teuthology.orchestra.run.smithi091.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-02-05T04:49:12.274 INFO:teuthology.orchestra.run.smithi091.stdout: python3-rados x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 320 k 2024-02-05T04:49:12.274 INFO:teuthology.orchestra.run.smithi091.stdout: python3-rbd x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 298 k 2024-02-05T04:49:12.274 INFO:teuthology.orchestra.run.smithi091.stdout: python3-rgw x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 100 k 2024-02-05T04:49:12.274 INFO:teuthology.orchestra.run.smithi091.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-02-05T04:49:12.274 INFO:teuthology.orchestra.run.smithi091.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-02-05T04:49:12.274 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:49:12.275 INFO:teuthology.orchestra.run.smithi091.stdout:Transaction Summary 2024-02-05T04:49:12.275 INFO:teuthology.orchestra.run.smithi091.stdout:================================================================================ 2024-02-05T04:49:12.275 INFO:teuthology.orchestra.run.smithi091.stdout:Install 27 Packages 2024-02-05T04:49:12.275 INFO:teuthology.orchestra.run.smithi091.stdout:Upgrade 2 Packages 2024-02-05T04:49:12.275 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:49:12.276 INFO:teuthology.orchestra.run.smithi091.stdout:Total download size: 55 M 2024-02-05T04:49:12.276 INFO:teuthology.orchestra.run.smithi091.stdout:Downloading Packages: 2024-02-05T04:49:12.276 INFO:teuthology.orchestra.run.smithi062.stdout:(14/29): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.1 MB/s | 159 kB 00:00 2024-02-05T04:49:12.301 INFO:teuthology.orchestra.run.smithi062.stdout:(15/29): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-02-05T04:49:12.311 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/31 2024-02-05T04:49:12.372 INFO:teuthology.orchestra.run.smithi062.stdout:(16/29): librdkafka-1.6.1-102.el9.x86_64.rpm 6.9 MB/s | 661 kB 00:00 2024-02-05T04:49:12.402 INFO:teuthology.orchestra.run.smithi062.stdout:(17/29): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.8 MB/s | 282 kB 00:00 2024-02-05T04:49:12.427 INFO:teuthology.orchestra.run.smithi062.stdout:(18/29): python3-prettytable-0.7.2-27.el9.noarc 727 kB/s | 41 kB 00:00 2024-02-05T04:49:12.656 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/31 2024-02-05T04:49:12.719 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : librados2-2:19.0.0-658.gd4ed20cf.el9.x86_64 5/31 2024-02-05T04:49:12.802 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librados2-2:19.0.0-658.gd4ed20cf.el9.x86_64 5/31 2024-02-05T04:49:12.890 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-rados-2:19.0.0-658.gd4ed20cf.el9.x86_64 6/31 2024-02-05T04:49:12.894 INFO:teuthology.orchestra.run.smithi062.stdout:(19/29): gperftools-libs-2.9.1-2.el9.x86_64.rpm 628 kB/s | 309 kB 00:00 2024-02-05T04:49:12.920 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libcephfs2-2:19.0.0-658.gd4ed20cf.el9.x86_64 7/31 2024-02-05T04:49:12.937 INFO:teuthology.orchestra.run.smithi062.stdout:(20/29): libarrow-doc-9.0.0-10.el9.noarch.rpm 618 kB/s | 26 kB 00:00 2024-02-05T04:49:12.966 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: libcephfs2-2:19.0.0-658.gd4ed20cf.el9.x86_64 7/31 2024-02-05T04:49:12.987 INFO:teuthology.orchestra.run.smithi062.stdout:(21/29): liboath-2.6.7-2.el9.x86_64.rpm 975 kB/s | 49 kB 00:00 2024-02-05T04:49:12.994 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/31 2024-02-05T04:49:13.045 INFO:teuthology.orchestra.run.smithi062.stdout:(22/29): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-02-05T04:49:13.047 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-ceph-argparse-2:19.0.0-658.gd4ed20cf.el9.x 9/31 2024-02-05T04:49:13.115 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cephfs-2:19.0.0-658.gd4ed20cf.el9.x86_64 10/31 2024-02-05T04:49:13.139 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libradosstriper1-2:19.0.0-658.gd4ed20cf.el9.x86_64 11/31 2024-02-05T04:49:13.188 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: libradosstriper1-2:19.0.0-658.gd4ed20cf.el9.x86_64 11/31 2024-02-05T04:49:13.296 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/31 2024-02-05T04:49:13.330 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/31 2024-02-05T04:49:13.363 INFO:teuthology.orchestra.run.smithi091.stdout:(1/29): ceph-radosgw-19.0.0-658.gd4ed20cf.el9.x 8.5 MB/s | 8.6 MB 00:01 2024-02-05T04:49:13.370 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/31 2024-02-05T04:49:13.403 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/31 2024-02-05T04:49:13.430 INFO:teuthology.orchestra.run.smithi091.stdout:(2/29): ceph-selinux-19.0.0-658.gd4ed20cf.el9.x 378 kB/s | 25 kB 00:00 2024-02-05T04:49:13.432 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/31 2024-02-05T04:49:13.454 INFO:teuthology.orchestra.run.smithi062.stdout:(23/29): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.0 MB/s | 838 kB 00:00 2024-02-05T04:49:13.464 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 17/31 2024-02-05T04:49:13.547 INFO:teuthology.orchestra.run.smithi091.stdout:(3/29): libcephfs2-19.0.0-658.gd4ed20cf.el9.x86 6.2 MB/s | 742 kB 00:00 2024-02-05T04:49:13.579 INFO:teuthology.orchestra.run.smithi062.stdout:(24/29): re2-20211101-3.el9.x86_64.rpm 1.5 MB/s | 192 kB 00:00 2024-02-05T04:49:13.630 INFO:teuthology.orchestra.run.smithi091.stdout:(4/29): libradosstriper1-19.0.0-658.gd4ed20cf.e 5.9 MB/s | 499 kB 00:00 2024-02-05T04:49:13.773 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 18/31 2024-02-05T04:49:13.803 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : librbd1-2:19.0.0-658.gd4ed20cf.el9.x86_64 19/31 2024-02-05T04:49:13.880 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librbd1-2:19.0.0-658.gd4ed20cf.el9.x86_64 19/31 2024-02-05T04:49:13.907 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-rbd-2:19.0.0-658.gd4ed20cf.el9.x86_64 20/31 2024-02-05T04:49:13.949 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 21/31 2024-02-05T04:49:14.164 INFO:teuthology.orchestra.run.smithi091.stdout:(5/29): ceph-common-19.0.0-658.gd4ed20cf.el9.x8 11 MB/s | 19 MB 00:01 2024-02-05T04:49:14.231 INFO:teuthology.orchestra.run.smithi091.stdout:(6/29): librgw2-19.0.0-658.gd4ed20cf.el9.x86_64 8.0 MB/s | 4.8 MB 00:00 2024-02-05T04:49:14.256 INFO:teuthology.orchestra.run.smithi091.stdout:(7/29): python3-ceph-argparse-19.0.0-658.gd4ed2 492 kB/s | 45 kB 00:00 2024-02-05T04:49:14.281 INFO:teuthology.orchestra.run.smithi091.stdout:(8/29): python3-ceph-common-19.0.0-658.gd4ed20c 2.5 MB/s | 130 kB 00:00 2024-02-05T04:49:14.306 INFO:teuthology.orchestra.run.smithi091.stdout:(9/29): python3-cephfs-19.0.0-658.gd4ed20cf.el9 3.2 MB/s | 162 kB 00:00 2024-02-05T04:49:14.313 INFO:teuthology.orchestra.run.smithi062.stdout:(25/29): thrift-0.15.0-2.el9.x86_64.rpm 2.2 MB/s | 1.6 MB 00:00 2024-02-05T04:49:14.332 INFO:teuthology.orchestra.run.smithi091.stdout:(10/29): python3-rados-19.0.0-658.gd4ed20cf.el9 6.2 MB/s | 320 kB 00:00 2024-02-05T04:49:14.357 INFO:teuthology.orchestra.run.smithi091.stdout:(11/29): python3-rbd-19.0.0-658.gd4ed20cf.el9.x 5.8 MB/s | 298 kB 00:00 2024-02-05T04:49:14.370 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 22/31 2024-02-05T04:49:14.397 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : librgw2-2:19.0.0-658.gd4ed20cf.el9.x86_64 23/31 2024-02-05T04:49:14.416 INFO:teuthology.orchestra.run.smithi091.stdout:(12/29): python3-rgw-19.0.0-658.gd4ed20cf.el9.x 1.2 MB/s | 100 kB 00:00 2024-02-05T04:49:14.451 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librgw2-2:19.0.0-658.gd4ed20cf.el9.x86_64 23/31 2024-02-05T04:49:14.518 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-rgw-2:19.0.0-658.gd4ed20cf.el9.x86_64 24/31 2024-02-05T04:49:14.549 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-ceph-common-2:19.0.0-658.gd4ed20cf.el9.x86 25/31 2024-02-05T04:49:14.605 INFO:teuthology.orchestra.run.smithi062.stdout:(26/29): librados2-19.0.0-658.gd4ed20cf.el9.x86 12 MB/s | 3.4 MB 00:00 2024-02-05T04:49:14.649 INFO:teuthology.orchestra.run.smithi091.stdout:(13/29): boost-program-options-1.75.0-8.el9.x86 362 kB/s | 106 kB 00:00 2024-02-05T04:49:14.663 INFO:teuthology.orchestra.run.smithi062.stdout:(27/29): libarrow-9.0.0-10.el9.x86_64.rpm 2.0 MB/s | 4.4 MB 00:02 2024-02-05T04:49:14.674 INFO:teuthology.orchestra.run.smithi091.stdout:(14/29): libpmemobj-1.12.1-1.el9.x86_64.rpm 616 kB/s | 159 kB 00:00 2024-02-05T04:49:14.708 INFO:teuthology.orchestra.run.smithi091.stdout:(15/29): librabbitmq-0.11.0-7.el9.x86_64.rpm 754 kB/s | 44 kB 00:00 2024-02-05T04:49:14.730 INFO:teuthology.orchestra.run.smithi062.stdout:(28/29): ceph-base-19.0.0-658.gd4ed20cf.el9.x86 1.1 MB/s | 5.1 MB 00:04 2024-02-05T04:49:14.800 INFO:teuthology.orchestra.run.smithi091.stdout:(16/29): librdkafka-1.6.1-102.el9.x86_64.rpm 5.2 MB/s | 661 kB 00:00 2024-02-05T04:49:14.827 INFO:teuthology.orchestra.run.smithi091.stdout:(17/29): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.3 MB/s | 282 kB 00:00 2024-02-05T04:49:14.853 INFO:teuthology.orchestra.run.smithi091.stdout:(18/29): python3-prettytable-0.7.2-27.el9.noarc 789 kB/s | 41 kB 00:00 2024-02-05T04:49:14.980 INFO:teuthology.orchestra.run.smithi062.stdout:(29/29): librbd1-19.0.0-658.gd4ed20cf.el9.x86_6 8.3 MB/s | 3.1 MB 00:00 2024-02-05T04:49:14.987 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:49:14.987 INFO:teuthology.orchestra.run.smithi062.stdout:Total 11 MB/s | 55 MB 00:05 2024-02-05T04:49:14.988 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2024-02-05T04:49:15.028 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2024-02-05T04:49:15.028 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2024-02-05T04:49:15.161 INFO:teuthology.orchestra.run.smithi091.stdout:(19/29): ceph-base-19.0.0-658.gd4ed20cf.el9.x86 1.8 MB/s | 5.1 MB 00:02 2024-02-05T04:49:15.512 INFO:teuthology.orchestra.run.smithi091.stdout:(20/29): gperftools-libs-2.9.1-2.el9.x86_64.rpm 452 kB/s | 309 kB 00:00 2024-02-05T04:49:15.521 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2024-02-05T04:49:15.521 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2024-02-05T04:49:15.537 INFO:teuthology.orchestra.run.smithi091.stdout:(21/29): libarrow-doc-9.0.0-10.el9.noarch.rpm 69 kB/s | 26 kB 00:00 2024-02-05T04:49:15.620 INFO:teuthology.orchestra.run.smithi091.stdout:(22/29): liboath-2.6.7-2.el9.x86_64.rpm 447 kB/s | 49 kB 00:00 2024-02-05T04:49:15.746 INFO:teuthology.orchestra.run.smithi091.stdout:(23/29): libunwind-1.6.2-1.el9.x86_64.rpm 323 kB/s | 67 kB 00:00 2024-02-05T04:49:15.829 INFO:teuthology.orchestra.run.smithi091.stdout:(24/29): libarrow-9.0.0-10.el9.x86_64.rpm 4.5 MB/s | 4.4 MB 00:00 2024-02-05T04:49:15.888 INFO:teuthology.orchestra.run.smithi091.stdout:(25/29): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.1 MB/s | 838 kB 00:00 2024-02-05T04:49:15.964 INFO:teuthology.orchestra.run.smithi091.stdout:(26/29): thrift-0.15.0-2.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-02-05T04:49:15.988 INFO:teuthology.orchestra.run.smithi091.stdout:(27/29): re2-20211101-3.el9.x86_64.rpm 790 kB/s | 192 kB 00:00 2024-02-05T04:49:16.239 INFO:teuthology.orchestra.run.smithi091.stdout:(28/29): librbd1-19.0.0-658.gd4ed20cf.el9.x86_6 11 MB/s | 3.1 MB 00:00 2024-02-05T04:49:16.522 INFO:teuthology.orchestra.run.smithi091.stdout:(29/29): librados2-19.0.0-658.gd4ed20cf.el9.x86 5.3 MB/s | 3.4 MB 00:00 2024-02-05T04:49:16.530 INFO:teuthology.orchestra.run.smithi091.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:49:16.530 INFO:teuthology.orchestra.run.smithi091.stdout:Total 13 MB/s | 55 MB 00:04 2024-02-05T04:49:16.530 INFO:teuthology.orchestra.run.smithi091.stdout:Running transaction check 2024-02-05T04:49:16.565 INFO:teuthology.orchestra.run.smithi091.stdout:Transaction check succeeded. 2024-02-05T04:49:16.565 INFO:teuthology.orchestra.run.smithi091.stdout:Running transaction test 2024-02-05T04:49:16.766 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-common-2:19.0.0-658.gd4ed20cf.el9.x86_64 26/31 2024-02-05T04:49:16.795 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-common-2:19.0.0-658.gd4ed20cf.el9.x86_64 26/31 2024-02-05T04:49:16.929 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2024-02-05T04:49:17.033 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/31 2024-02-05T04:49:17.035 INFO:teuthology.orchestra.run.smithi091.stdout:Transaction test succeeded. 2024-02-05T04:49:17.036 INFO:teuthology.orchestra.run.smithi091.stdout:Running transaction 2024-02-05T04:49:17.060 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/31 2024-02-05T04:49:17.111 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/31 2024-02-05T04:49:17.327 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-common-2:19.0.0-658.gd4ed20cf.el9.x86_64 26/31 2024-02-05T04:49:17.366 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-base-2:19.0.0-658.gd4ed20cf.el9.x86_64 27/31 2024-02-05T04:49:17.506 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/31 2024-02-05T04:49:17.542 INFO:teuthology.orchestra.run.smithi062.stdout: Upgrading : librados2-2:19.0.0-658.gd4ed20cf.el9.x86_64 5/31 2024-02-05T04:49:17.591 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-base-2:19.0.0-658.gd4ed20cf.el9.x86_64 27/31 2024-02-05T04:49:17.591 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-02-05T04:49:17.591 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-02-05T04:49:17.591 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:49:17.625 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-selinux-2:19.0.0-658.gd4ed20cf.el9.x86_64 28/31 2024-02-05T04:49:17.635 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: librados2-2:19.0.0-658.gd4ed20cf.el9.x86_64 5/31 2024-02-05T04:49:17.724 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-rados-2:19.0.0-658.gd4ed20cf.el9.x86_64 6/31 2024-02-05T04:49:17.752 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libcephfs2-2:19.0.0-658.gd4ed20cf.el9.x86_64 7/31 2024-02-05T04:49:17.823 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: libcephfs2-2:19.0.0-658.gd4ed20cf.el9.x86_64 7/31 2024-02-05T04:49:17.910 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/31 2024-02-05T04:49:18.030 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-ceph-argparse-2:19.0.0-658.gd4ed20cf.el9.x 9/31 2024-02-05T04:49:18.108 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-cephfs-2:19.0.0-658.gd4ed20cf.el9.x86_64 10/31 2024-02-05T04:49:18.129 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libradosstriper1-2:19.0.0-658.gd4ed20cf.el9.x86_64 11/31 2024-02-05T04:49:18.195 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: libradosstriper1-2:19.0.0-658.gd4ed20cf.el9.x86_64 11/31 2024-02-05T04:49:18.305 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/31 2024-02-05T04:49:18.337 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/31 2024-02-05T04:49:18.377 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/31 2024-02-05T04:49:18.392 INFO:teuthology.orchestra.run.smithi091.stdout: Preparing : 1/1 2024-02-05T04:49:18.410 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/31 2024-02-05T04:49:18.439 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/31 2024-02-05T04:49:18.472 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 17/31 2024-02-05T04:49:18.493 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/31 2024-02-05T04:49:18.518 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/31 2024-02-05T04:49:18.568 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/31 2024-02-05T04:49:18.789 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 18/31 2024-02-05T04:49:18.818 INFO:teuthology.orchestra.run.smithi062.stdout: Upgrading : librbd1-2:19.0.0-658.gd4ed20cf.el9.x86_64 19/31 2024-02-05T04:49:18.885 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/31 2024-02-05T04:49:18.917 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: librbd1-2:19.0.0-658.gd4ed20cf.el9.x86_64 19/31 2024-02-05T04:49:18.944 INFO:teuthology.orchestra.run.smithi091.stdout: Upgrading : librados2-2:19.0.0-658.gd4ed20cf.el9.x86_64 5/31 2024-02-05T04:49:18.947 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-rbd-2:19.0.0-658.gd4ed20cf.el9.x86_64 20/31 2024-02-05T04:49:18.987 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 21/31 2024-02-05T04:49:19.016 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: librados2-2:19.0.0-658.gd4ed20cf.el9.x86_64 5/31 2024-02-05T04:49:19.104 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-rados-2:19.0.0-658.gd4ed20cf.el9.x86_64 6/31 2024-02-05T04:49:19.135 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : libcephfs2-2:19.0.0-658.gd4ed20cf.el9.x86_64 7/31 2024-02-05T04:49:19.181 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: libcephfs2-2:19.0.0-658.gd4ed20cf.el9.x86_64 7/31 2024-02-05T04:49:19.210 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/31 2024-02-05T04:49:19.254 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-ceph-argparse-2:19.0.0-658.gd4ed20cf.el9.x 9/31 2024-02-05T04:49:19.322 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-cephfs-2:19.0.0-658.gd4ed20cf.el9.x86_64 10/31 2024-02-05T04:49:19.346 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : libradosstriper1-2:19.0.0-658.gd4ed20cf.el9.x86_64 11/31 2024-02-05T04:49:19.404 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: libradosstriper1-2:19.0.0-658.gd4ed20cf.el9.x86_64 11/31 2024-02-05T04:49:19.437 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 22/31 2024-02-05T04:49:19.472 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : librgw2-2:19.0.0-658.gd4ed20cf.el9.x86_64 23/31 2024-02-05T04:49:19.508 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/31 2024-02-05T04:49:19.544 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: librgw2-2:19.0.0-658.gd4ed20cf.el9.x86_64 23/31 2024-02-05T04:49:19.545 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/31 2024-02-05T04:49:19.586 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/31 2024-02-05T04:49:19.610 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/31 2024-02-05T04:49:19.623 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-rgw-2:19.0.0-658.gd4ed20cf.el9.x86_64 24/31 2024-02-05T04:49:19.639 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/31 2024-02-05T04:49:19.658 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-ceph-common-2:19.0.0-658.gd4ed20cf.el9.x86 25/31 2024-02-05T04:49:19.671 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 17/31 2024-02-05T04:49:19.982 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 18/31 2024-02-05T04:49:20.018 INFO:teuthology.orchestra.run.smithi091.stdout: Upgrading : librbd1-2:19.0.0-658.gd4ed20cf.el9.x86_64 19/31 2024-02-05T04:49:20.108 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: librbd1-2:19.0.0-658.gd4ed20cf.el9.x86_64 19/31 2024-02-05T04:49:20.148 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-rbd-2:19.0.0-658.gd4ed20cf.el9.x86_64 20/31 2024-02-05T04:49:20.188 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 21/31 2024-02-05T04:49:20.613 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 22/31 2024-02-05T04:49:20.637 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : librgw2-2:19.0.0-658.gd4ed20cf.el9.x86_64 23/31 2024-02-05T04:49:20.692 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: librgw2-2:19.0.0-658.gd4ed20cf.el9.x86_64 23/31 2024-02-05T04:49:20.768 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-rgw-2:19.0.0-658.gd4ed20cf.el9.x86_64 24/31 2024-02-05T04:49:20.798 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-ceph-common-2:19.0.0-658.gd4ed20cf.el9.x86 25/31 2024-02-05T04:49:21.845 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-common-2:19.0.0-658.gd4ed20cf.el9.x86_64 26/31 2024-02-05T04:49:21.879 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-common-2:19.0.0-658.gd4ed20cf.el9.x86_64 26/31 2024-02-05T04:49:22.439 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-common-2:19.0.0-658.gd4ed20cf.el9.x86_64 26/31 2024-02-05T04:49:22.483 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-base-2:19.0.0-658.gd4ed20cf.el9.x86_64 27/31 2024-02-05T04:49:22.571 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-base-2:19.0.0-658.gd4ed20cf.el9.x86_64 27/31 2024-02-05T04:49:22.571 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-02-05T04:49:22.572 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-02-05T04:49:22.572 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:49:22.609 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-selinux-2:19.0.0-658.gd4ed20cf.el9.x86_64 28/31 2024-02-05T04:49:22.939 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: ceph-common-2:19.0.0-658.gd4ed20cf.el9.x86_64 26/31 2024-02-05T04:49:22.977 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : ceph-common-2:19.0.0-658.gd4ed20cf.el9.x86_64 26/31 2024-02-05T04:49:23.510 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: ceph-common-2:19.0.0-658.gd4ed20cf.el9.x86_64 26/31 2024-02-05T04:49:23.547 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : ceph-base-2:19.0.0-658.gd4ed20cf.el9.x86_64 27/31 2024-02-05T04:49:23.635 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: ceph-base-2:19.0.0-658.gd4ed20cf.el9.x86_64 27/31 2024-02-05T04:49:23.635 INFO:teuthology.orchestra.run.smithi091.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-02-05T04:49:23.635 INFO:teuthology.orchestra.run.smithi091.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-02-05T04:49:23.635 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:49:23.673 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : ceph-selinux-2:19.0.0-658.gd4ed20cf.el9.x86_64 28/31 2024-02-05T04:49:26.965 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-selinux-2:19.0.0-658.gd4ed20cf.el9.x86_64 28/31 2024-02-05T04:49:26.965 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /sys 2024-02-05T04:49:26.965 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /proc 2024-02-05T04:49:26.965 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /mnt 2024-02-05T04:49:26.965 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /var/tmp 2024-02-05T04:49:26.966 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /home 2024-02-05T04:49:26.966 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /root 2024-02-05T04:49:26.966 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /tmp 2024-02-05T04:49:26.966 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:49:27.798 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-radosgw-2:19.0.0-658.gd4ed20cf.el9.x86_64 29/31 2024-02-05T04:49:27.823 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-658.gd4ed20cf.el9.x86_64 29/31 2024-02-05T04:49:27.823 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T04:49:27.823 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-02-05T04:49:27.823 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-05T04:49:27.823 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-05T04:49:27.823 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:49:27.825 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 30/31 2024-02-05T04:49:27.909 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 30/31 2024-02-05T04:49:27.910 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 31/31 2024-02-05T04:49:29.065 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 31/31 2024-02-05T04:49:29.065 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-base-2:19.0.0-658.gd4ed20cf.el9.x86_64 1/31 2024-02-05T04:49:29.065 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-common-2:19.0.0-658.gd4ed20cf.el9.x86_64 2/31 2024-02-05T04:49:29.065 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-radosgw-2:19.0.0-658.gd4ed20cf.el9.x86_64 3/31 2024-02-05T04:49:29.065 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-selinux-2:19.0.0-658.gd4ed20cf.el9.x86_64 4/31 2024-02-05T04:49:29.065 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libcephfs2-2:19.0.0-658.gd4ed20cf.el9.x86_64 5/31 2024-02-05T04:49:29.065 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libradosstriper1-2:19.0.0-658.gd4ed20cf.el9.x86_64 6/31 2024-02-05T04:49:29.066 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librgw2-2:19.0.0-658.gd4ed20cf.el9.x86_64 7/31 2024-02-05T04:49:29.066 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-ceph-argparse-2:19.0.0-658.gd4ed20cf.el9.x 8/31 2024-02-05T04:49:29.066 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-ceph-common-2:19.0.0-658.gd4ed20cf.el9.x86 9/31 2024-02-05T04:49:29.066 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cephfs-2:19.0.0-658.gd4ed20cf.el9.x86_64 10/31 2024-02-05T04:49:29.066 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-rados-2:19.0.0-658.gd4ed20cf.el9.x86_64 11/31 2024-02-05T04:49:29.066 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-rbd-2:19.0.0-658.gd4ed20cf.el9.x86_64 12/31 2024-02-05T04:49:29.066 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-rgw-2:19.0.0-658.gd4ed20cf.el9.x86_64 13/31 2024-02-05T04:49:29.066 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/31 2024-02-05T04:49:29.066 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/31 2024-02-05T04:49:29.066 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/31 2024-02-05T04:49:29.067 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/31 2024-02-05T04:49:29.067 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/31 2024-02-05T04:49:29.067 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/31 2024-02-05T04:49:29.067 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 20/31 2024-02-05T04:49:29.067 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 21/31 2024-02-05T04:49:29.067 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 22/31 2024-02-05T04:49:29.067 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 23/31 2024-02-05T04:49:29.067 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 24/31 2024-02-05T04:49:29.067 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 25/31 2024-02-05T04:49:29.067 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : re2-1:20211101-3.el9.x86_64 26/31 2024-02-05T04:49:29.067 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 27/31 2024-02-05T04:49:29.067 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librados2-2:19.0.0-658.gd4ed20cf.el9.x86_64 28/31 2024-02-05T04:49:29.068 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 29/31 2024-02-05T04:49:29.068 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librbd1-2:19.0.0-658.gd4ed20cf.el9.x86_64 30/31 2024-02-05T04:49:29.973 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 31/31 2024-02-05T04:49:29.973 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:49:29.973 INFO:teuthology.orchestra.run.smithi196.stdout:Upgraded: 2024-02-05T04:49:29.974 INFO:teuthology.orchestra.run.smithi196.stdout: librados2-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:49:29.974 INFO:teuthology.orchestra.run.smithi196.stdout: librbd1-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:49:29.974 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-02-05T04:49:29.974 INFO:teuthology.orchestra.run.smithi196.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-02-05T04:49:29.974 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-base-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:49:29.974 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-common-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:49:29.974 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-radosgw-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:49:29.974 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-selinux-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:49:29.974 INFO:teuthology.orchestra.run.smithi196.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-02-05T04:49:29.974 INFO:teuthology.orchestra.run.smithi196.stdout: libarrow-9.0.0-10.el9.x86_64 2024-02-05T04:49:29.974 INFO:teuthology.orchestra.run.smithi196.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-02-05T04:49:29.975 INFO:teuthology.orchestra.run.smithi196.stdout: libcephfs2-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:49:29.975 INFO:teuthology.orchestra.run.smithi196.stdout: liboath-2.6.7-2.el9.x86_64 2024-02-05T04:49:29.975 INFO:teuthology.orchestra.run.smithi196.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-02-05T04:49:29.975 INFO:teuthology.orchestra.run.smithi196.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-02-05T04:49:29.975 INFO:teuthology.orchestra.run.smithi196.stdout: libradosstriper1-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:49:29.975 INFO:teuthology.orchestra.run.smithi196.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-02-05T04:49:29.975 INFO:teuthology.orchestra.run.smithi196.stdout: librgw2-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:49:29.975 INFO:teuthology.orchestra.run.smithi196.stdout: libunwind-1.6.2-1.el9.x86_64 2024-02-05T04:49:29.975 INFO:teuthology.orchestra.run.smithi196.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-02-05T04:49:29.975 INFO:teuthology.orchestra.run.smithi196.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-02-05T04:49:29.975 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ceph-argparse-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:49:29.975 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ceph-common-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:49:29.975 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cephfs-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:49:29.976 INFO:teuthology.orchestra.run.smithi196.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-02-05T04:49:29.976 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rados-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:49:29.976 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rbd-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:49:29.976 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rgw-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:49:29.976 INFO:teuthology.orchestra.run.smithi196.stdout: re2-1:20211101-3.el9.x86_64 2024-02-05T04:49:29.976 INFO:teuthology.orchestra.run.smithi196.stdout: thrift-0.15.0-2.el9.x86_64 2024-02-05T04:49:29.976 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:49:29.976 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-02-05T04:49:30.355 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-test 2024-02-05T04:49:30.920 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:00:31 ago on Mon 05 Feb 2024 04:48:59 AM UTC. 2024-02-05T04:49:31.076 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-02-05T04:49:31.077 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-02-05T04:49:31.077 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-02-05T04:49:31.077 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-02-05T04:49:31.077 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-02-05T04:49:31.077 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-test x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 45 M 2024-02-05T04:49:31.077 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-02-05T04:49:31.077 INFO:teuthology.orchestra.run.smithi196.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-02-05T04:49:31.077 INFO:teuthology.orchestra.run.smithi196.stdout: libcephsqlite x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 163 k 2024-02-05T04:49:31.077 INFO:teuthology.orchestra.run.smithi196.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-02-05T04:49:31.077 INFO:teuthology.orchestra.run.smithi196.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-02-05T04:49:31.078 INFO:teuthology.orchestra.run.smithi196.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-02-05T04:49:31.078 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:49:31.078 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-02-05T04:49:31.078 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-02-05T04:49:31.078 INFO:teuthology.orchestra.run.smithi196.stdout:Install 6 Packages 2024-02-05T04:49:31.078 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:49:31.079 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 46 M 2024-02-05T04:49:31.079 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 196 M 2024-02-05T04:49:31.079 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-02-05T04:49:31.239 INFO:teuthology.orchestra.run.smithi196.stdout:(1/6): socat-1.7.4.1-5.el9.x86_64.rpm 1.8 MB/s | 300 kB 00:00 2024-02-05T04:49:31.272 INFO:teuthology.orchestra.run.smithi196.stdout:(2/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-02-05T04:49:31.314 INFO:teuthology.orchestra.run.smithi196.stdout:(3/6): jq-1.6-16.el9.x86_64.rpm 4.4 MB/s | 186 kB 00:00 2024-02-05T04:49:31.339 INFO:teuthology.orchestra.run.smithi196.stdout:(4/6): libcephsqlite-19.0.0-658.gd4ed20cf.el9.x 628 kB/s | 163 kB 00:00 2024-02-05T04:49:31.364 INFO:teuthology.orchestra.run.smithi196.stdout:(5/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.2 MB/s | 217 kB 00:00 2024-02-05T04:49:32.629 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-selinux-2:19.0.0-658.gd4ed20cf.el9.x86_64 28/31 2024-02-05T04:49:32.629 INFO:teuthology.orchestra.run.smithi062.stdout:skipping the directory /sys 2024-02-05T04:49:32.629 INFO:teuthology.orchestra.run.smithi062.stdout:skipping the directory /proc 2024-02-05T04:49:32.629 INFO:teuthology.orchestra.run.smithi062.stdout:skipping the directory /mnt 2024-02-05T04:49:32.629 INFO:teuthology.orchestra.run.smithi062.stdout:skipping the directory /var/tmp 2024-02-05T04:49:32.629 INFO:teuthology.orchestra.run.smithi062.stdout:skipping the directory /home 2024-02-05T04:49:32.629 INFO:teuthology.orchestra.run.smithi062.stdout:skipping the directory /root 2024-02-05T04:49:32.629 INFO:teuthology.orchestra.run.smithi062.stdout:skipping the directory /tmp 2024-02-05T04:49:32.629 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:49:33.127 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: ceph-selinux-2:19.0.0-658.gd4ed20cf.el9.x86_64 28/31 2024-02-05T04:49:33.127 INFO:teuthology.orchestra.run.smithi091.stdout:skipping the directory /sys 2024-02-05T04:49:33.127 INFO:teuthology.orchestra.run.smithi091.stdout:skipping the directory /proc 2024-02-05T04:49:33.127 INFO:teuthology.orchestra.run.smithi091.stdout:skipping the directory /mnt 2024-02-05T04:49:33.127 INFO:teuthology.orchestra.run.smithi091.stdout:skipping the directory /var/tmp 2024-02-05T04:49:33.128 INFO:teuthology.orchestra.run.smithi091.stdout:skipping the directory /home 2024-02-05T04:49:33.128 INFO:teuthology.orchestra.run.smithi091.stdout:skipping the directory /root 2024-02-05T04:49:33.128 INFO:teuthology.orchestra.run.smithi091.stdout:skipping the directory /tmp 2024-02-05T04:49:33.128 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:49:33.458 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-radosgw-2:19.0.0-658.gd4ed20cf.el9.x86_64 29/31 2024-02-05T04:49:33.488 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-658.gd4ed20cf.el9.x86_64 29/31 2024-02-05T04:49:33.488 INFO:teuthology.orchestra.run.smithi062.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T04:49:33.488 INFO:teuthology.orchestra.run.smithi062.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-02-05T04:49:33.489 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-05T04:49:33.489 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-05T04:49:33.489 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:49:33.490 INFO:teuthology.orchestra.run.smithi062.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 30/31 2024-02-05T04:49:33.560 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 30/31 2024-02-05T04:49:33.560 INFO:teuthology.orchestra.run.smithi062.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 31/31 2024-02-05T04:49:33.964 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : ceph-radosgw-2:19.0.0-658.gd4ed20cf.el9.x86_64 29/31 2024-02-05T04:49:33.988 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-658.gd4ed20cf.el9.x86_64 29/31 2024-02-05T04:49:33.988 INFO:teuthology.orchestra.run.smithi091.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T04:49:33.989 INFO:teuthology.orchestra.run.smithi091.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-02-05T04:49:33.989 INFO:teuthology.orchestra.run.smithi091.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-05T04:49:33.989 INFO:teuthology.orchestra.run.smithi091.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-05T04:49:33.989 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:49:33.990 INFO:teuthology.orchestra.run.smithi091.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 30/31 2024-02-05T04:49:34.099 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 30/31 2024-02-05T04:49:34.099 INFO:teuthology.orchestra.run.smithi091.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 31/31 2024-02-05T04:49:34.667 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 31/31 2024-02-05T04:49:34.667 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-base-2:19.0.0-658.gd4ed20cf.el9.x86_64 1/31 2024-02-05T04:49:34.667 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-common-2:19.0.0-658.gd4ed20cf.el9.x86_64 2/31 2024-02-05T04:49:34.667 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-radosgw-2:19.0.0-658.gd4ed20cf.el9.x86_64 3/31 2024-02-05T04:49:34.667 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-selinux-2:19.0.0-658.gd4ed20cf.el9.x86_64 4/31 2024-02-05T04:49:34.667 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libcephfs2-2:19.0.0-658.gd4ed20cf.el9.x86_64 5/31 2024-02-05T04:49:34.667 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libradosstriper1-2:19.0.0-658.gd4ed20cf.el9.x86_64 6/31 2024-02-05T04:49:34.667 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : librgw2-2:19.0.0-658.gd4ed20cf.el9.x86_64 7/31 2024-02-05T04:49:34.668 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-ceph-argparse-2:19.0.0-658.gd4ed20cf.el9.x 8/31 2024-02-05T04:49:34.668 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-ceph-common-2:19.0.0-658.gd4ed20cf.el9.x86 9/31 2024-02-05T04:49:34.668 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-cephfs-2:19.0.0-658.gd4ed20cf.el9.x86_64 10/31 2024-02-05T04:49:34.668 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-rados-2:19.0.0-658.gd4ed20cf.el9.x86_64 11/31 2024-02-05T04:49:34.668 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-rbd-2:19.0.0-658.gd4ed20cf.el9.x86_64 12/31 2024-02-05T04:49:34.668 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-rgw-2:19.0.0-658.gd4ed20cf.el9.x86_64 13/31 2024-02-05T04:49:34.668 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/31 2024-02-05T04:49:34.668 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/31 2024-02-05T04:49:34.668 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/31 2024-02-05T04:49:34.668 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/31 2024-02-05T04:49:34.668 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/31 2024-02-05T04:49:34.668 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/31 2024-02-05T04:49:34.668 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 20/31 2024-02-05T04:49:34.669 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 21/31 2024-02-05T04:49:34.669 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 22/31 2024-02-05T04:49:34.669 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 23/31 2024-02-05T04:49:34.669 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 24/31 2024-02-05T04:49:34.669 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 25/31 2024-02-05T04:49:34.669 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : re2-1:20211101-3.el9.x86_64 26/31 2024-02-05T04:49:34.669 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 27/31 2024-02-05T04:49:34.669 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : librados2-2:19.0.0-658.gd4ed20cf.el9.x86_64 28/31 2024-02-05T04:49:34.669 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 29/31 2024-02-05T04:49:34.669 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : librbd1-2:19.0.0-658.gd4ed20cf.el9.x86_64 30/31 2024-02-05T04:49:34.812 INFO:teuthology.orchestra.run.smithi150.stdout:Extra Packages for Enterprise Linux 771 kB/s | 20 MB 00:26 2024-02-05T04:49:35.187 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 31/31 2024-02-05T04:49:35.187 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : ceph-base-2:19.0.0-658.gd4ed20cf.el9.x86_64 1/31 2024-02-05T04:49:35.187 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : ceph-common-2:19.0.0-658.gd4ed20cf.el9.x86_64 2/31 2024-02-05T04:49:35.187 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : ceph-radosgw-2:19.0.0-658.gd4ed20cf.el9.x86_64 3/31 2024-02-05T04:49:35.187 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : ceph-selinux-2:19.0.0-658.gd4ed20cf.el9.x86_64 4/31 2024-02-05T04:49:35.188 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : libcephfs2-2:19.0.0-658.gd4ed20cf.el9.x86_64 5/31 2024-02-05T04:49:35.188 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : libradosstriper1-2:19.0.0-658.gd4ed20cf.el9.x86_64 6/31 2024-02-05T04:49:35.188 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : librgw2-2:19.0.0-658.gd4ed20cf.el9.x86_64 7/31 2024-02-05T04:49:35.188 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-ceph-argparse-2:19.0.0-658.gd4ed20cf.el9.x 8/31 2024-02-05T04:49:35.188 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-ceph-common-2:19.0.0-658.gd4ed20cf.el9.x86 9/31 2024-02-05T04:49:35.188 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-cephfs-2:19.0.0-658.gd4ed20cf.el9.x86_64 10/31 2024-02-05T04:49:35.188 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-rados-2:19.0.0-658.gd4ed20cf.el9.x86_64 11/31 2024-02-05T04:49:35.188 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-rbd-2:19.0.0-658.gd4ed20cf.el9.x86_64 12/31 2024-02-05T04:49:35.188 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-rgw-2:19.0.0-658.gd4ed20cf.el9.x86_64 13/31 2024-02-05T04:49:35.188 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/31 2024-02-05T04:49:35.188 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/31 2024-02-05T04:49:35.188 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/31 2024-02-05T04:49:35.188 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/31 2024-02-05T04:49:35.188 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/31 2024-02-05T04:49:35.189 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/31 2024-02-05T04:49:35.189 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 20/31 2024-02-05T04:49:35.189 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 21/31 2024-02-05T04:49:35.189 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 22/31 2024-02-05T04:49:35.189 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 23/31 2024-02-05T04:49:35.189 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 24/31 2024-02-05T04:49:35.189 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 25/31 2024-02-05T04:49:35.189 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : re2-1:20211101-3.el9.x86_64 26/31 2024-02-05T04:49:35.189 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 27/31 2024-02-05T04:49:35.190 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : librados2-2:19.0.0-658.gd4ed20cf.el9.x86_64 28/31 2024-02-05T04:49:35.190 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 29/31 2024-02-05T04:49:35.190 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : librbd1-2:19.0.0-658.gd4ed20cf.el9.x86_64 30/31 2024-02-05T04:49:35.791 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 31/31 2024-02-05T04:49:35.791 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:49:35.792 INFO:teuthology.orchestra.run.smithi062.stdout:Upgraded: 2024-02-05T04:49:35.792 INFO:teuthology.orchestra.run.smithi062.stdout: librados2-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:49:35.792 INFO:teuthology.orchestra.run.smithi062.stdout: librbd1-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:49:35.792 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2024-02-05T04:49:35.792 INFO:teuthology.orchestra.run.smithi062.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-02-05T04:49:35.792 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-base-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:49:35.792 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-common-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:49:35.792 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-radosgw-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:49:35.792 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-selinux-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:49:35.792 INFO:teuthology.orchestra.run.smithi062.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-02-05T04:49:35.792 INFO:teuthology.orchestra.run.smithi062.stdout: libarrow-9.0.0-10.el9.x86_64 2024-02-05T04:49:35.792 INFO:teuthology.orchestra.run.smithi062.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-02-05T04:49:35.792 INFO:teuthology.orchestra.run.smithi062.stdout: libcephfs2-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:49:35.793 INFO:teuthology.orchestra.run.smithi062.stdout: liboath-2.6.7-2.el9.x86_64 2024-02-05T04:49:35.793 INFO:teuthology.orchestra.run.smithi062.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-02-05T04:49:35.793 INFO:teuthology.orchestra.run.smithi062.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-02-05T04:49:35.793 INFO:teuthology.orchestra.run.smithi062.stdout: libradosstriper1-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:49:35.793 INFO:teuthology.orchestra.run.smithi062.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-02-05T04:49:35.793 INFO:teuthology.orchestra.run.smithi062.stdout: librgw2-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:49:35.793 INFO:teuthology.orchestra.run.smithi062.stdout: libunwind-1.6.2-1.el9.x86_64 2024-02-05T04:49:35.793 INFO:teuthology.orchestra.run.smithi062.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-02-05T04:49:35.793 INFO:teuthology.orchestra.run.smithi062.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-02-05T04:49:35.793 INFO:teuthology.orchestra.run.smithi062.stdout: python3-ceph-argparse-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:49:35.793 INFO:teuthology.orchestra.run.smithi062.stdout: python3-ceph-common-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:49:35.793 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cephfs-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:49:35.793 INFO:teuthology.orchestra.run.smithi062.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-02-05T04:49:35.793 INFO:teuthology.orchestra.run.smithi062.stdout: python3-rados-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:49:35.794 INFO:teuthology.orchestra.run.smithi062.stdout: python3-rbd-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:49:35.794 INFO:teuthology.orchestra.run.smithi062.stdout: python3-rgw-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:49:35.794 INFO:teuthology.orchestra.run.smithi062.stdout: re2-1:20211101-3.el9.x86_64 2024-02-05T04:49:35.794 INFO:teuthology.orchestra.run.smithi062.stdout: thrift-0.15.0-2.el9.x86_64 2024-02-05T04:49:35.794 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:49:35.794 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-02-05T04:49:36.144 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install ceph-test 2024-02-05T04:49:36.295 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 31/31 2024-02-05T04:49:36.295 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:49:36.295 INFO:teuthology.orchestra.run.smithi091.stdout:Upgraded: 2024-02-05T04:49:36.296 INFO:teuthology.orchestra.run.smithi091.stdout: librados2-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:49:36.296 INFO:teuthology.orchestra.run.smithi091.stdout: librbd1-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:49:36.296 INFO:teuthology.orchestra.run.smithi091.stdout:Installed: 2024-02-05T04:49:36.296 INFO:teuthology.orchestra.run.smithi091.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-02-05T04:49:36.296 INFO:teuthology.orchestra.run.smithi091.stdout: ceph-base-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:49:36.296 INFO:teuthology.orchestra.run.smithi091.stdout: ceph-common-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:49:36.297 INFO:teuthology.orchestra.run.smithi091.stdout: ceph-radosgw-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:49:36.297 INFO:teuthology.orchestra.run.smithi091.stdout: ceph-selinux-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:49:36.297 INFO:teuthology.orchestra.run.smithi091.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-02-05T04:49:36.297 INFO:teuthology.orchestra.run.smithi091.stdout: libarrow-9.0.0-10.el9.x86_64 2024-02-05T04:49:36.297 INFO:teuthology.orchestra.run.smithi091.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-02-05T04:49:36.297 INFO:teuthology.orchestra.run.smithi091.stdout: libcephfs2-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:49:36.297 INFO:teuthology.orchestra.run.smithi091.stdout: liboath-2.6.7-2.el9.x86_64 2024-02-05T04:49:36.298 INFO:teuthology.orchestra.run.smithi091.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-02-05T04:49:36.298 INFO:teuthology.orchestra.run.smithi091.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-02-05T04:49:36.298 INFO:teuthology.orchestra.run.smithi091.stdout: libradosstriper1-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:49:36.298 INFO:teuthology.orchestra.run.smithi091.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-02-05T04:49:36.298 INFO:teuthology.orchestra.run.smithi091.stdout: librgw2-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:49:36.298 INFO:teuthology.orchestra.run.smithi091.stdout: libunwind-1.6.2-1.el9.x86_64 2024-02-05T04:49:36.298 INFO:teuthology.orchestra.run.smithi091.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-02-05T04:49:36.298 INFO:teuthology.orchestra.run.smithi091.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-02-05T04:49:36.298 INFO:teuthology.orchestra.run.smithi091.stdout: python3-ceph-argparse-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:49:36.298 INFO:teuthology.orchestra.run.smithi091.stdout: python3-ceph-common-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:49:36.299 INFO:teuthology.orchestra.run.smithi091.stdout: python3-cephfs-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:49:36.299 INFO:teuthology.orchestra.run.smithi091.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-02-05T04:49:36.299 INFO:teuthology.orchestra.run.smithi091.stdout: python3-rados-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:49:36.299 INFO:teuthology.orchestra.run.smithi091.stdout: python3-rbd-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:49:36.299 INFO:teuthology.orchestra.run.smithi091.stdout: python3-rgw-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:49:36.299 INFO:teuthology.orchestra.run.smithi091.stdout: re2-1:20211101-3.el9.x86_64 2024-02-05T04:49:36.299 INFO:teuthology.orchestra.run.smithi091.stdout: thrift-0.15.0-2.el9.x86_64 2024-02-05T04:49:36.299 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:49:36.299 INFO:teuthology.orchestra.run.smithi091.stdout:Complete! 2024-02-05T04:49:36.683 DEBUG:teuthology.orchestra.run.smithi091:> sudo yum -y install ceph-test 2024-02-05T04:49:36.715 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:00:29 ago on Mon 05 Feb 2024 04:49:07 AM UTC. 2024-02-05T04:49:36.884 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-02-05T04:49:36.885 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-02-05T04:49:36.885 INFO:teuthology.orchestra.run.smithi062.stdout: Package Arch Version Repository Size 2024-02-05T04:49:36.885 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-02-05T04:49:36.885 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2024-02-05T04:49:36.885 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-test x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 45 M 2024-02-05T04:49:36.886 INFO:teuthology.orchestra.run.smithi062.stdout:Installing dependencies: 2024-02-05T04:49:36.886 INFO:teuthology.orchestra.run.smithi062.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-02-05T04:49:36.886 INFO:teuthology.orchestra.run.smithi062.stdout: libcephsqlite x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 163 k 2024-02-05T04:49:36.886 INFO:teuthology.orchestra.run.smithi062.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-02-05T04:49:36.886 INFO:teuthology.orchestra.run.smithi062.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-02-05T04:49:36.886 INFO:teuthology.orchestra.run.smithi062.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-02-05T04:49:36.886 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:49:36.886 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2024-02-05T04:49:36.886 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-02-05T04:49:36.886 INFO:teuthology.orchestra.run.smithi062.stdout:Install 6 Packages 2024-02-05T04:49:36.886 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:49:36.887 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 46 M 2024-02-05T04:49:36.887 INFO:teuthology.orchestra.run.smithi062.stdout:Installed size: 196 M 2024-02-05T04:49:36.887 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2024-02-05T04:49:37.042 INFO:teuthology.orchestra.run.smithi062.stdout:(1/6): socat-1.7.4.1-5.el9.x86_64.rpm 1.9 MB/s | 300 kB 00:00 2024-02-05T04:49:37.068 INFO:teuthology.orchestra.run.smithi062.stdout:(2/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.6 MB/s | 63 kB 00:00 2024-02-05T04:49:37.109 INFO:teuthology.orchestra.run.smithi062.stdout:(3/6): jq-1.6-16.el9.x86_64.rpm 4.4 MB/s | 186 kB 00:00 2024-02-05T04:49:37.151 INFO:teuthology.orchestra.run.smithi062.stdout:(4/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 5.1 MB/s | 217 kB 00:00 2024-02-05T04:49:37.245 INFO:teuthology.orchestra.run.smithi091.stdout:Last metadata expiration check: 0:00:28 ago on Mon 05 Feb 2024 04:49:09 AM UTC. 2024-02-05T04:49:37.281 INFO:teuthology.orchestra.run.smithi196.stdout:(6/6): ceph-test-19.0.0-658.gd4ed20cf.el9.x86_6 7.2 MB/s | 45 MB 00:06 2024-02-05T04:49:37.283 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:49:37.284 INFO:teuthology.orchestra.run.smithi196.stdout:Total 7.4 MB/s | 46 MB 00:06 2024-02-05T04:49:37.284 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-02-05T04:49:37.301 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-02-05T04:49:37.301 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-02-05T04:49:37.377 INFO:teuthology.orchestra.run.smithi062.stdout:(5/6): libcephsqlite-19.0.0-658.gd4ed20cf.el9.x 334 kB/s | 163 kB 00:00 2024-02-05T04:49:37.402 INFO:teuthology.orchestra.run.smithi091.stdout:Dependencies resolved. 2024-02-05T04:49:37.402 INFO:teuthology.orchestra.run.smithi091.stdout:================================================================================ 2024-02-05T04:49:37.402 INFO:teuthology.orchestra.run.smithi091.stdout: Package Arch Version Repository Size 2024-02-05T04:49:37.403 INFO:teuthology.orchestra.run.smithi091.stdout:================================================================================ 2024-02-05T04:49:37.403 INFO:teuthology.orchestra.run.smithi091.stdout:Installing: 2024-02-05T04:49:37.403 INFO:teuthology.orchestra.run.smithi091.stdout: ceph-test x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 45 M 2024-02-05T04:49:37.403 INFO:teuthology.orchestra.run.smithi091.stdout:Installing dependencies: 2024-02-05T04:49:37.403 INFO:teuthology.orchestra.run.smithi091.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-02-05T04:49:37.403 INFO:teuthology.orchestra.run.smithi091.stdout: libcephsqlite x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 163 k 2024-02-05T04:49:37.403 INFO:teuthology.orchestra.run.smithi091.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-02-05T04:49:37.403 INFO:teuthology.orchestra.run.smithi091.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-02-05T04:49:37.403 INFO:teuthology.orchestra.run.smithi091.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-02-05T04:49:37.403 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:49:37.403 INFO:teuthology.orchestra.run.smithi091.stdout:Transaction Summary 2024-02-05T04:49:37.403 INFO:teuthology.orchestra.run.smithi091.stdout:================================================================================ 2024-02-05T04:49:37.403 INFO:teuthology.orchestra.run.smithi091.stdout:Install 6 Packages 2024-02-05T04:49:37.403 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:49:37.404 INFO:teuthology.orchestra.run.smithi091.stdout:Total download size: 46 M 2024-02-05T04:49:37.404 INFO:teuthology.orchestra.run.smithi091.stdout:Installed size: 196 M 2024-02-05T04:49:37.404 INFO:teuthology.orchestra.run.smithi091.stdout:Downloading Packages: 2024-02-05T04:49:37.552 INFO:teuthology.orchestra.run.smithi091.stdout:(1/6): socat-1.7.4.1-5.el9.x86_64.rpm 2.0 MB/s | 300 kB 00:00 2024-02-05T04:49:37.585 INFO:teuthology.orchestra.run.smithi091.stdout:(2/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-02-05T04:49:37.628 INFO:teuthology.orchestra.run.smithi091.stdout:(3/6): jq-1.6-16.el9.x86_64.rpm 4.4 MB/s | 186 kB 00:00 2024-02-05T04:49:37.669 INFO:teuthology.orchestra.run.smithi091.stdout:(4/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 5.1 MB/s | 217 kB 00:00 2024-02-05T04:49:37.673 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-02-05T04:49:37.673 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-02-05T04:49:37.994 INFO:teuthology.orchestra.run.smithi091.stdout:(5/6): libcephsqlite-19.0.0-658.gd4ed20cf.el9.x 277 kB/s | 163 kB 00:00 2024-02-05T04:49:38.238 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-02-05T04:49:38.319 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 1/6 2024-02-05T04:49:38.355 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : jq-1.6-16.el9.x86_64 2/6 2024-02-05T04:49:38.424 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/6 2024-02-05T04:49:38.464 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/6 2024-02-05T04:49:38.484 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libcephsqlite-2:19.0.0-658.gd4ed20cf.el9.x86_64 5/6 2024-02-05T04:49:39.692 INFO:teuthology.orchestra.run.smithi150.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-02-05T04:49:42.261 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-02-05T04:49:42.274 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-02-05T04:49:42.275 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2024-02-05T04:49:42.275 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-02-05T04:49:42.275 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2024-02-05T04:49:42.275 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-radosgw x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 8.6 M 2024-02-05T04:49:42.275 INFO:teuthology.orchestra.run.smithi150.stdout:Upgrading: 2024-02-05T04:49:42.275 INFO:teuthology.orchestra.run.smithi150.stdout: librados2 x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 3.4 M 2024-02-05T04:49:42.275 INFO:teuthology.orchestra.run.smithi150.stdout: librbd1 x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 3.1 M 2024-02-05T04:49:42.275 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2024-02-05T04:49:42.275 INFO:teuthology.orchestra.run.smithi150.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-02-05T04:49:42.275 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-base x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 5.1 M 2024-02-05T04:49:42.275 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-common x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 19 M 2024-02-05T04:49:42.275 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-selinux x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 25 k 2024-02-05T04:49:42.275 INFO:teuthology.orchestra.run.smithi150.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-02-05T04:49:42.276 INFO:teuthology.orchestra.run.smithi150.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-02-05T04:49:42.276 INFO:teuthology.orchestra.run.smithi150.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-02-05T04:49:42.276 INFO:teuthology.orchestra.run.smithi150.stdout: libcephfs2 x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 742 k 2024-02-05T04:49:42.276 INFO:teuthology.orchestra.run.smithi150.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-02-05T04:49:42.276 INFO:teuthology.orchestra.run.smithi150.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-02-05T04:49:42.276 INFO:teuthology.orchestra.run.smithi150.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-02-05T04:49:42.276 INFO:teuthology.orchestra.run.smithi150.stdout: libradosstriper1 x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 499 k 2024-02-05T04:49:42.276 INFO:teuthology.orchestra.run.smithi150.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-02-05T04:49:42.276 INFO:teuthology.orchestra.run.smithi150.stdout: librgw2 x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 4.8 M 2024-02-05T04:49:42.276 INFO:teuthology.orchestra.run.smithi150.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-02-05T04:49:42.276 INFO:teuthology.orchestra.run.smithi150.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-02-05T04:49:42.276 INFO:teuthology.orchestra.run.smithi150.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-02-05T04:49:42.276 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ceph-argparse x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 45 k 2024-02-05T04:49:42.277 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ceph-common x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 130 k 2024-02-05T04:49:42.277 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cephfs x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 162 k 2024-02-05T04:49:42.277 INFO:teuthology.orchestra.run.smithi150.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-02-05T04:49:42.277 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rados x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 320 k 2024-02-05T04:49:42.277 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rbd x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 298 k 2024-02-05T04:49:42.277 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rgw x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 100 k 2024-02-05T04:49:42.277 INFO:teuthology.orchestra.run.smithi150.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-02-05T04:49:42.277 INFO:teuthology.orchestra.run.smithi150.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-02-05T04:49:42.277 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:49:42.277 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-02-05T04:49:42.277 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-02-05T04:49:42.277 INFO:teuthology.orchestra.run.smithi150.stdout:Install 27 Packages 2024-02-05T04:49:42.278 INFO:teuthology.orchestra.run.smithi150.stdout:Upgrade 2 Packages 2024-02-05T04:49:42.278 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:49:42.278 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 55 M 2024-02-05T04:49:42.278 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-02-05T04:49:42.731 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: libcephsqlite-2:19.0.0-658.gd4ed20cf.el9.x86_64 5/6 2024-02-05T04:49:42.811 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-test-2:19.0.0-658.gd4ed20cf.el9.x86_64 6/6 2024-02-05T04:49:45.054 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-test-2:19.0.0-658.gd4ed20cf.el9.x86_64 6/6 2024-02-05T04:49:45.054 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-test-2:19.0.0-658.gd4ed20cf.el9.x86_64 1/6 2024-02-05T04:49:45.054 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libcephsqlite-2:19.0.0-658.gd4ed20cf.el9.x86_64 2/6 2024-02-05T04:49:45.054 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/6 2024-02-05T04:49:45.054 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/6 2024-02-05T04:49:45.054 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : jq-1.6-16.el9.x86_64 5/6 2024-02-05T04:49:45.376 INFO:teuthology.orchestra.run.smithi062.stdout:(6/6): ceph-test-19.0.0-658.gd4ed20cf.el9.x86_6 5.3 MB/s | 45 MB 00:08 2024-02-05T04:49:45.379 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:49:45.380 INFO:teuthology.orchestra.run.smithi062.stdout:Total 5.4 MB/s | 46 MB 00:08 2024-02-05T04:49:45.380 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2024-02-05T04:49:45.402 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2024-02-05T04:49:45.403 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2024-02-05T04:49:45.439 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 6/6 2024-02-05T04:49:45.439 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:49:45.439 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-02-05T04:49:45.439 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-test-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:49:45.439 INFO:teuthology.orchestra.run.smithi196.stdout: jq-1.6-16.el9.x86_64 2024-02-05T04:49:45.439 INFO:teuthology.orchestra.run.smithi196.stdout: libcephsqlite-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:49:45.440 INFO:teuthology.orchestra.run.smithi196.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-02-05T04:49:45.440 INFO:teuthology.orchestra.run.smithi196.stdout: socat-1.7.4.1-5.el9.x86_64 2024-02-05T04:49:45.440 INFO:teuthology.orchestra.run.smithi196.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-02-05T04:49:45.440 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:49:45.440 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-02-05T04:49:45.703 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph 2024-02-05T04:49:45.784 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2024-02-05T04:49:45.784 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2024-02-05T04:49:45.960 INFO:teuthology.orchestra.run.smithi150.stdout:(1/29): ceph-base-19.0.0-658.gd4ed20cf.el9.x86_ 1.4 MB/s | 5.1 MB 00:03 2024-02-05T04:49:46.027 INFO:teuthology.orchestra.run.smithi150.stdout:(2/29): ceph-selinux-19.0.0-658.gd4ed20cf.el9.x 379 kB/s | 25 kB 00:00 2024-02-05T04:49:46.035 INFO:teuthology.orchestra.run.smithi091.stdout:(6/6): ceph-test-19.0.0-658.gd4ed20cf.el9.x86_6 5.2 MB/s | 45 MB 00:08 2024-02-05T04:49:46.038 INFO:teuthology.orchestra.run.smithi091.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:49:46.039 INFO:teuthology.orchestra.run.smithi091.stdout:Total 5.3 MB/s | 46 MB 00:08 2024-02-05T04:49:46.039 INFO:teuthology.orchestra.run.smithi091.stdout:Running transaction check 2024-02-05T04:49:46.059 INFO:teuthology.orchestra.run.smithi091.stdout:Transaction check succeeded. 2024-02-05T04:49:46.059 INFO:teuthology.orchestra.run.smithi091.stdout:Running transaction test 2024-02-05T04:49:46.297 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:00:47 ago on Mon 05 Feb 2024 04:48:59 AM UTC. 2024-02-05T04:49:46.348 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2024-02-05T04:49:46.352 INFO:teuthology.orchestra.run.smithi150.stdout:(3/29): libcephfs2-19.0.0-658.gd4ed20cf.el9.x86 2.2 MB/s | 742 kB 00:00 2024-02-05T04:49:46.415 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 1/6 2024-02-05T04:49:46.433 INFO:teuthology.orchestra.run.smithi091.stdout:Transaction test succeeded. 2024-02-05T04:49:46.433 INFO:teuthology.orchestra.run.smithi091.stdout:Running transaction 2024-02-05T04:49:46.457 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : jq-1.6-16.el9.x86_64 2/6 2024-02-05T04:49:46.471 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-02-05T04:49:46.473 INFO:teuthology.orchestra.run.smithi196.stdout:======================================================================================== 2024-02-05T04:49:46.473 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-02-05T04:49:46.474 INFO:teuthology.orchestra.run.smithi196.stdout:======================================================================================== 2024-02-05T04:49:46.474 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-02-05T04:49:46.474 INFO:teuthology.orchestra.run.smithi196.stdout: ceph x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 6.5 k 2024-02-05T04:49:46.474 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-02-05T04:49:46.474 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mds x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 2.1 M 2024-02-05T04:49:46.474 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 1.5 M 2024-02-05T04:49:46.474 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-modules-core noarch 2:19.0.0-658.gd4ed20cf.el9 ceph-noarch 245 k 2024-02-05T04:49:46.475 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mon x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 4.7 M 2024-02-05T04:49:46.475 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-osd x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 17 M 2024-02-05T04:49:46.475 INFO:teuthology.orchestra.run.smithi196.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-02-05T04:49:46.475 INFO:teuthology.orchestra.run.smithi196.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-02-05T04:49:46.475 INFO:teuthology.orchestra.run.smithi196.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-02-05T04:49:46.475 INFO:teuthology.orchestra.run.smithi196.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-02-05T04:49:46.475 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-02-05T04:49:46.475 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-02-05T04:49:46.475 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-02-05T04:49:46.475 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-02-05T04:49:46.475 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-02-05T04:49:46.475 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-02-05T04:49:46.476 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-02-05T04:49:46.476 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-02-05T04:49:46.476 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-02-05T04:49:46.476 INFO:teuthology.orchestra.run.smithi196.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-02-05T04:49:46.476 INFO:teuthology.orchestra.run.smithi196.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-02-05T04:49:46.476 INFO:teuthology.orchestra.run.smithi196.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-02-05T04:49:46.476 INFO:teuthology.orchestra.run.smithi196.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-02-05T04:49:46.476 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-02-05T04:49:46.476 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-02-05T04:49:46.476 INFO:teuthology.orchestra.run.smithi196.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-02-05T04:49:46.477 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-02-05T04:49:46.477 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-02-05T04:49:46.477 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-02-05T04:49:46.477 INFO:teuthology.orchestra.run.smithi196.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-02-05T04:49:46.477 INFO:teuthology.orchestra.run.smithi196.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-02-05T04:49:46.477 INFO:teuthology.orchestra.run.smithi196.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-02-05T04:49:46.477 INFO:teuthology.orchestra.run.smithi196.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-02-05T04:49:46.477 INFO:teuthology.orchestra.run.smithi196.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-02-05T04:49:46.477 INFO:teuthology.orchestra.run.smithi196.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-02-05T04:49:46.477 INFO:teuthology.orchestra.run.smithi196.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-02-05T04:49:46.477 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:49:46.477 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-02-05T04:49:46.478 INFO:teuthology.orchestra.run.smithi196.stdout:======================================================================================== 2024-02-05T04:49:46.478 INFO:teuthology.orchestra.run.smithi196.stdout:Install 36 Packages 2024-02-05T04:49:46.478 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:49:46.478 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 30 M 2024-02-05T04:49:46.478 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 106 M 2024-02-05T04:49:46.479 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-02-05T04:49:46.521 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/6 2024-02-05T04:49:46.561 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/6 2024-02-05T04:49:46.586 INFO:teuthology.orchestra.run.smithi150.stdout:(4/29): libradosstriper1-19.0.0-658.gd4ed20cf.e 2.1 MB/s | 499 kB 00:00 2024-02-05T04:49:46.588 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libcephsqlite-2:19.0.0-658.gd4ed20cf.el9.x86_64 5/6 2024-02-05T04:49:46.993 INFO:teuthology.orchestra.run.smithi091.stdout: Preparing : 1/1 2024-02-05T04:49:47.061 INFO:teuthology.orchestra.run.smithi196.stdout:(1/36): ceph-19.0.0-658.gd4ed20cf.el9.x86_64.rp 26 kB/s | 6.5 kB 00:00 2024-02-05T04:49:47.061 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 1/6 2024-02-05T04:49:47.092 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : jq-1.6-16.el9.x86_64 2/6 2024-02-05T04:49:47.136 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/6 2024-02-05T04:49:47.177 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/6 2024-02-05T04:49:47.204 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : libcephsqlite-2:19.0.0-658.gd4ed20cf.el9.x86_64 5/6 2024-02-05T04:49:47.486 INFO:teuthology.orchestra.run.smithi150.stdout:(5/29): ceph-radosgw-19.0.0-658.gd4ed20cf.el9.x 1.7 MB/s | 8.6 MB 00:05 2024-02-05T04:49:47.570 INFO:teuthology.orchestra.run.smithi150.stdout:(6/29): python3-ceph-argparse-19.0.0-658.gd4ed2 541 kB/s | 45 kB 00:00 2024-02-05T04:49:47.594 INFO:teuthology.orchestra.run.smithi196.stdout:(2/36): ceph-mgr-19.0.0-658.gd4ed20cf.el9.x86_6 1.9 MB/s | 1.5 MB 00:00 2024-02-05T04:49:47.712 INFO:teuthology.orchestra.run.smithi150.stdout:(7/29): python3-ceph-common-19.0.0-658.gd4ed20c 918 kB/s | 130 kB 00:00 2024-02-05T04:49:47.879 INFO:teuthology.orchestra.run.smithi150.stdout:(8/29): python3-cephfs-19.0.0-658.gd4ed20cf.el9 972 kB/s | 162 kB 00:00 2024-02-05T04:49:47.986 INFO:teuthology.orchestra.run.smithi196.stdout:(3/36): ceph-mds-19.0.0-658.gd4ed20cf.el9.x86_6 1.8 MB/s | 2.1 MB 00:01 2024-02-05T04:49:48.112 INFO:teuthology.orchestra.run.smithi150.stdout:(9/29): python3-rados-19.0.0-658.gd4ed20cf.el9. 1.3 MB/s | 320 kB 00:00 2024-02-05T04:49:48.120 INFO:teuthology.orchestra.run.smithi196.stdout:(4/36): ceph-mgr-modules-core-19.0.0-658.gd4ed2 1.8 MB/s | 245 kB 00:00 2024-02-05T04:49:48.228 INFO:teuthology.orchestra.run.smithi196.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.7 MB/s | 187 kB 00:00 2024-02-05T04:49:48.270 INFO:teuthology.orchestra.run.smithi196.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-02-05T04:49:48.304 INFO:teuthology.orchestra.run.smithi196.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 936 kB/s | 32 kB 00:00 2024-02-05T04:49:48.329 INFO:teuthology.orchestra.run.smithi150.stdout:(10/29): python3-rbd-19.0.0-658.gd4ed20cf.el9.x 1.3 MB/s | 298 kB 00:00 2024-02-05T04:49:48.347 INFO:teuthology.orchestra.run.smithi196.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-02-05T04:49:48.380 INFO:teuthology.orchestra.run.smithi196.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-02-05T04:49:48.421 INFO:teuthology.orchestra.run.smithi150.stdout:(11/29): python3-rgw-19.0.0-658.gd4ed20cf.el9.x 1.1 MB/s | 100 kB 00:00 2024-02-05T04:49:48.441 INFO:teuthology.orchestra.run.smithi196.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.9 MB/s | 241 kB 00:00 2024-02-05T04:49:48.516 INFO:teuthology.orchestra.run.smithi196.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-02-05T04:49:48.546 INFO:teuthology.orchestra.run.smithi150.stdout:(12/29): boost-program-options-1.75.0-8.el9.x86 847 kB/s | 106 kB 00:00 2024-02-05T04:49:48.550 INFO:teuthology.orchestra.run.smithi196.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-02-05T04:49:48.592 INFO:teuthology.orchestra.run.smithi196.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 2.9 MB/s | 124 kB 00:00 2024-02-05T04:49:48.597 INFO:teuthology.orchestra.run.smithi150.stdout:(13/29): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-02-05T04:49:48.626 INFO:teuthology.orchestra.run.smithi196.stdout:(14/36): python3-requests-2.25.1-8.el9.noarch.r 3.3 MB/s | 113 kB 00:00 2024-02-05T04:49:48.630 INFO:teuthology.orchestra.run.smithi150.stdout:(14/29): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-02-05T04:49:48.668 INFO:teuthology.orchestra.run.smithi196.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-02-05T04:49:48.689 INFO:teuthology.orchestra.run.smithi150.stdout:(15/29): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-02-05T04:49:48.718 INFO:teuthology.orchestra.run.smithi196.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 448 kB/s | 22 kB 00:00 2024-02-05T04:49:48.731 INFO:teuthology.orchestra.run.smithi150.stdout:(16/29): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-02-05T04:49:48.773 INFO:teuthology.orchestra.run.smithi150.stdout:(17/29): python3-prettytable-0.7.2-27.el9.noarc 992 kB/s | 41 kB 00:00 2024-02-05T04:49:48.835 INFO:teuthology.orchestra.run.smithi196.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 1.3 MB/s | 151 kB 00:00 2024-02-05T04:49:48.840 INFO:teuthology.orchestra.run.smithi150.stdout:(18/29): librgw2-19.0.0-658.gd4ed20cf.el9.x86_6 2.1 MB/s | 4.8 MB 00:02 2024-02-05T04:49:48.893 INFO:teuthology.orchestra.run.smithi196.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 748 kB/s | 43 kB 00:00 2024-02-05T04:49:48.915 INFO:teuthology.orchestra.run.smithi150.stdout:(19/29): gperftools-libs-2.9.1-2.el9.x86_64.rpm 2.1 MB/s | 309 kB 00:00 2024-02-05T04:49:48.944 INFO:teuthology.orchestra.run.smithi196.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 3.4 MB/s | 172 kB 00:00 2024-02-05T04:49:48.949 INFO:teuthology.orchestra.run.smithi150.stdout:(20/29): libarrow-doc-9.0.0-10.el9.noarch.rpm 784 kB/s | 26 kB 00:00 2024-02-05T04:49:48.982 INFO:teuthology.orchestra.run.smithi150.stdout:(21/29): liboath-2.6.7-2.el9.x86_64.rpm 1.4 MB/s | 49 kB 00:00 2024-02-05T04:49:49.002 INFO:teuthology.orchestra.run.smithi196.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 6.0 MB/s | 358 kB 00:00 2024-02-05T04:49:49.016 INFO:teuthology.orchestra.run.smithi150.stdout:(22/29): libunwind-1.6.2-1.el9.x86_64.rpm 2.0 MB/s | 67 kB 00:00 2024-02-05T04:49:49.052 INFO:teuthology.orchestra.run.smithi196.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 214 kB/s | 11 kB 00:00 2024-02-05T04:49:49.094 INFO:teuthology.orchestra.run.smithi196.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 424 kB/s | 18 kB 00:00 2024-02-05T04:49:49.141 INFO:teuthology.orchestra.run.smithi150.stdout:(23/29): parquet-libs-9.0.0-10.el9.x86_64.rpm 6.6 MB/s | 838 kB 00:00 2024-02-05T04:49:49.161 INFO:teuthology.orchestra.run.smithi196.stdout:(23/36): ceph-mon-19.0.0-658.gd4ed20cf.el9.x86_ 2.2 MB/s | 4.7 MB 00:02 2024-02-05T04:49:49.187 INFO:teuthology.orchestra.run.smithi196.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 252 kB/s | 23 kB 00:00 2024-02-05T04:49:49.216 INFO:teuthology.orchestra.run.smithi150.stdout:(24/29): libarrow-9.0.0-10.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-02-05T04:49:49.229 INFO:teuthology.orchestra.run.smithi196.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 472 kB/s | 20 kB 00:00 2024-02-05T04:49:49.242 INFO:teuthology.orchestra.run.smithi150.stdout:(25/29): re2-20211101-3.el9.x86_64.rpm 1.9 MB/s | 192 kB 00:00 2024-02-05T04:49:49.254 INFO:teuthology.orchestra.run.smithi196.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 211 kB/s | 19 kB 00:00 2024-02-05T04:49:49.279 INFO:teuthology.orchestra.run.smithi196.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 918 kB/s | 46 kB 00:00 2024-02-05T04:49:49.321 INFO:teuthology.orchestra.run.smithi196.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 6.4 MB/s | 272 kB 00:00 2024-02-05T04:49:49.325 INFO:teuthology.orchestra.run.smithi150.stdout:(26/29): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-02-05T04:49:49.363 INFO:teuthology.orchestra.run.smithi196.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 394 kB/s | 16 kB 00:00 2024-02-05T04:49:49.388 INFO:teuthology.orchestra.run.smithi196.stdout:(30/36): python3-more-itertools-8.12.0-2.el9.no 587 kB/s | 79 kB 00:00 2024-02-05T04:49:49.414 INFO:teuthology.orchestra.run.smithi196.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-02-05T04:49:49.431 INFO:teuthology.orchestra.run.smithi196.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 852 kB/s | 36 kB 00:00 2024-02-05T04:49:49.456 INFO:teuthology.orchestra.run.smithi196.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 5.4 MB/s | 230 kB 00:00 2024-02-05T04:49:49.490 INFO:teuthology.orchestra.run.smithi196.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 600 kB/s | 20 kB 00:00 2024-02-05T04:49:49.540 INFO:teuthology.orchestra.run.smithi196.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2024-02-05T04:49:49.982 INFO:teuthology.orchestra.run.smithi196.stdout:(36/36): ceph-osd-19.0.0-658.gd4ed20cf.el9.x86_ 7.0 MB/s | 17 MB 00:02 2024-02-05T04:49:49.986 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:49:49.986 INFO:teuthology.orchestra.run.smithi196.stdout:Total 8.4 MB/s | 30 MB 00:03 2024-02-05T04:49:50.001 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-02-05T04:49:50.044 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-02-05T04:49:50.044 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-02-05T04:49:50.461 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-02-05T04:49:50.461 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-02-05T04:49:50.792 INFO:teuthology.orchestra.run.smithi150.stdout:(27/29): librados2-19.0.0-658.gd4ed20cf.el9.x86 2.2 MB/s | 3.4 MB 00:01 2024-02-05T04:49:50.884 INFO:teuthology.orchestra.run.smithi150.stdout:(28/29): librbd1-19.0.0-658.gd4ed20cf.el9.x86_6 2.0 MB/s | 3.1 MB 00:01 2024-02-05T04:49:51.077 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: libcephsqlite-2:19.0.0-658.gd4ed20cf.el9.x86_64 5/6 2024-02-05T04:49:51.165 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-test-2:19.0.0-658.gd4ed20cf.el9.x86_64 6/6 2024-02-05T04:49:51.490 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: libcephsqlite-2:19.0.0-658.gd4ed20cf.el9.x86_64 5/6 2024-02-05T04:49:51.561 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : ceph-test-2:19.0.0-658.gd4ed20cf.el9.x86_64 6/6 2024-02-05T04:49:51.768 INFO:teuthology.orchestra.run.smithi150.stdout:(29/29): ceph-common-19.0.0-658.gd4ed20cf.el9.x 2.1 MB/s | 19 MB 00:09 2024-02-05T04:49:51.771 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:49:51.771 INFO:teuthology.orchestra.run.smithi150.stdout:Total 5.8 MB/s | 55 MB 00:09 2024-02-05T04:49:51.771 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-02-05T04:49:51.804 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-02-05T04:49:51.804 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-02-05T04:49:51.811 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-02-05T04:49:51.869 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-02-05T04:49:51.904 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-02-05T04:49:51.937 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-02-05T04:49:52.007 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-02-05T04:49:52.053 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-02-05T04:49:52.085 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-02-05T04:49:52.177 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-02-05T04:49:52.256 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-02-05T04:49:52.285 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-02-05T04:49:52.286 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-02-05T04:49:52.297 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-02-05T04:49:52.369 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-02-05T04:49:52.422 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-02-05T04:49:52.461 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-02-05T04:49:52.510 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-02-05T04:49:52.565 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-02-05T04:49:52.752 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-02-05T04:49:52.798 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-02-05T04:49:52.829 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-02-05T04:49:52.872 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-02-05T04:49:52.902 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-02-05T04:49:52.978 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-02-05T04:49:53.012 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-02-05T04:49:53.046 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-02-05T04:49:53.086 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-02-05T04:49:53.121 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-02-05T04:49:53.164 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-02-05T04:49:53.253 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-02-05T04:49:53.295 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-02-05T04:49:53.376 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-02-05T04:49:53.394 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-test-2:19.0.0-658.gd4ed20cf.el9.x86_64 6/6 2024-02-05T04:49:53.394 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-test-2:19.0.0-658.gd4ed20cf.el9.x86_64 1/6 2024-02-05T04:49:53.394 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libcephsqlite-2:19.0.0-658.gd4ed20cf.el9.x86_64 2/6 2024-02-05T04:49:53.394 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/6 2024-02-05T04:49:53.394 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/6 2024-02-05T04:49:53.394 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : jq-1.6-16.el9.x86_64 5/6 2024-02-05T04:49:53.528 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-02-05T04:49:53.654 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-02-05T04:49:53.668 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-02-05T04:49:53.748 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 6/6 2024-02-05T04:49:53.748 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:49:53.748 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2024-02-05T04:49:53.748 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-test-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:49:53.748 INFO:teuthology.orchestra.run.smithi062.stdout: jq-1.6-16.el9.x86_64 2024-02-05T04:49:53.749 INFO:teuthology.orchestra.run.smithi062.stdout: libcephsqlite-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:49:53.749 INFO:teuthology.orchestra.run.smithi062.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-02-05T04:49:53.749 INFO:teuthology.orchestra.run.smithi062.stdout: socat-1.7.4.1-5.el9.x86_64 2024-02-05T04:49:53.749 INFO:teuthology.orchestra.run.smithi062.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-02-05T04:49:53.749 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:49:53.749 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-02-05T04:49:53.760 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/31 2024-02-05T04:49:53.806 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/31 2024-02-05T04:49:53.814 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-658.gd4ed20cf.el9.n 31/36 2024-02-05T04:49:53.842 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-2:19.0.0-658.gd4ed20cf.el9.x86_64 32/36 2024-02-05T04:49:53.856 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/31 2024-02-05T04:49:53.867 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mgr-2:19.0.0-658.gd4ed20cf.el9.x86_64 32/36 2024-02-05T04:49:53.867 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T04:49:53.867 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-02-05T04:49:53.867 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-05T04:49:53.867 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-05T04:49:53.867 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:49:53.937 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: ceph-test-2:19.0.0-658.gd4ed20cf.el9.x86_64 6/6 2024-02-05T04:49:53.938 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : ceph-test-2:19.0.0-658.gd4ed20cf.el9.x86_64 1/6 2024-02-05T04:49:53.938 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : libcephsqlite-2:19.0.0-658.gd4ed20cf.el9.x86_64 2/6 2024-02-05T04:49:53.938 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/6 2024-02-05T04:49:53.938 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/6 2024-02-05T04:49:53.938 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : jq-1.6-16.el9.x86_64 5/6 2024-02-05T04:49:54.039 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install ceph 2024-02-05T04:49:54.175 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/31 2024-02-05T04:49:54.222 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : librados2-2:19.0.0-658.gd4ed20cf.el9.x86_64 5/31 2024-02-05T04:49:54.309 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 6/6 2024-02-05T04:49:54.310 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:49:54.310 INFO:teuthology.orchestra.run.smithi091.stdout:Installed: 2024-02-05T04:49:54.310 INFO:teuthology.orchestra.run.smithi091.stdout: ceph-test-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:49:54.310 INFO:teuthology.orchestra.run.smithi091.stdout: jq-1.6-16.el9.x86_64 2024-02-05T04:49:54.310 INFO:teuthology.orchestra.run.smithi091.stdout: libcephsqlite-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:49:54.310 INFO:teuthology.orchestra.run.smithi091.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-02-05T04:49:54.310 INFO:teuthology.orchestra.run.smithi091.stdout: socat-1.7.4.1-5.el9.x86_64 2024-02-05T04:49:54.310 INFO:teuthology.orchestra.run.smithi091.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-02-05T04:49:54.310 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:49:54.310 INFO:teuthology.orchestra.run.smithi091.stdout:Complete! 2024-02-05T04:49:54.313 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librados2-2:19.0.0-658.gd4ed20cf.el9.x86_64 5/31 2024-02-05T04:49:54.400 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-rados-2:19.0.0-658.gd4ed20cf.el9.x86_64 6/31 2024-02-05T04:49:54.431 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libcephfs2-2:19.0.0-658.gd4ed20cf.el9.x86_64 7/31 2024-02-05T04:49:54.477 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: libcephfs2-2:19.0.0-658.gd4ed20cf.el9.x86_64 7/31 2024-02-05T04:49:54.506 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/31 2024-02-05T04:49:54.542 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-ceph-argparse-2:19.0.0-658.gd4ed20cf.el9.x 9/31 2024-02-05T04:49:54.587 DEBUG:teuthology.orchestra.run.smithi091:> sudo yum -y install ceph 2024-02-05T04:49:54.609 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cephfs-2:19.0.0-658.gd4ed20cf.el9.x86_64 10/31 2024-02-05T04:49:54.611 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:00:47 ago on Mon 05 Feb 2024 04:49:07 AM UTC. 2024-02-05T04:49:54.633 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libradosstriper1-2:19.0.0-658.gd4ed20cf.el9.x86_64 11/31 2024-02-05T04:49:54.682 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: libradosstriper1-2:19.0.0-658.gd4ed20cf.el9.x86_64 11/31 2024-02-05T04:49:54.776 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-02-05T04:49:54.778 INFO:teuthology.orchestra.run.smithi062.stdout:======================================================================================== 2024-02-05T04:49:54.778 INFO:teuthology.orchestra.run.smithi062.stdout: Package Arch Version Repository Size 2024-02-05T04:49:54.779 INFO:teuthology.orchestra.run.smithi062.stdout:======================================================================================== 2024-02-05T04:49:54.779 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2024-02-05T04:49:54.779 INFO:teuthology.orchestra.run.smithi062.stdout: ceph x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 6.5 k 2024-02-05T04:49:54.779 INFO:teuthology.orchestra.run.smithi062.stdout:Installing dependencies: 2024-02-05T04:49:54.779 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mds x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 2.1 M 2024-02-05T04:49:54.779 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 1.5 M 2024-02-05T04:49:54.779 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-modules-core noarch 2:19.0.0-658.gd4ed20cf.el9 ceph-noarch 245 k 2024-02-05T04:49:54.779 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mon x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 4.7 M 2024-02-05T04:49:54.779 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-osd x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 17 M 2024-02-05T04:49:54.779 INFO:teuthology.orchestra.run.smithi062.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-02-05T04:49:54.779 INFO:teuthology.orchestra.run.smithi062.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-02-05T04:49:54.779 INFO:teuthology.orchestra.run.smithi062.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-02-05T04:49:54.779 INFO:teuthology.orchestra.run.smithi062.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-02-05T04:49:54.779 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-02-05T04:49:54.780 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-02-05T04:49:54.780 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-02-05T04:49:54.780 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-02-05T04:49:54.780 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-02-05T04:49:54.780 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-02-05T04:49:54.780 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-02-05T04:49:54.780 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-02-05T04:49:54.780 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-02-05T04:49:54.780 INFO:teuthology.orchestra.run.smithi062.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-02-05T04:49:54.780 INFO:teuthology.orchestra.run.smithi062.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-02-05T04:49:54.780 INFO:teuthology.orchestra.run.smithi062.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-02-05T04:49:54.780 INFO:teuthology.orchestra.run.smithi062.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-02-05T04:49:54.781 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-02-05T04:49:54.781 INFO:teuthology.orchestra.run.smithi062.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-02-05T04:49:54.781 INFO:teuthology.orchestra.run.smithi062.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-02-05T04:49:54.781 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-02-05T04:49:54.781 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-02-05T04:49:54.781 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-02-05T04:49:54.781 INFO:teuthology.orchestra.run.smithi062.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-02-05T04:49:54.781 INFO:teuthology.orchestra.run.smithi062.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-02-05T04:49:54.781 INFO:teuthology.orchestra.run.smithi062.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-02-05T04:49:54.781 INFO:teuthology.orchestra.run.smithi062.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-02-05T04:49:54.781 INFO:teuthology.orchestra.run.smithi062.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-02-05T04:49:54.781 INFO:teuthology.orchestra.run.smithi062.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-02-05T04:49:54.781 INFO:teuthology.orchestra.run.smithi062.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-02-05T04:49:54.781 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:49:54.782 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2024-02-05T04:49:54.782 INFO:teuthology.orchestra.run.smithi062.stdout:======================================================================================== 2024-02-05T04:49:54.782 INFO:teuthology.orchestra.run.smithi062.stdout:Install 36 Packages 2024-02-05T04:49:54.782 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:49:54.782 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 30 M 2024-02-05T04:49:54.783 INFO:teuthology.orchestra.run.smithi062.stdout:Installed size: 106 M 2024-02-05T04:49:54.783 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2024-02-05T04:49:54.789 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/31 2024-02-05T04:49:54.855 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/31 2024-02-05T04:49:54.910 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/31 2024-02-05T04:49:54.939 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/31 2024-02-05T04:49:54.968 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/31 2024-02-05T04:49:55.001 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 17/31 2024-02-05T04:49:55.156 INFO:teuthology.orchestra.run.smithi091.stdout:Last metadata expiration check: 0:00:46 ago on Mon 05 Feb 2024 04:49:09 AM UTC. 2024-02-05T04:49:55.292 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-osd-2:19.0.0-658.gd4ed20cf.el9.x86_64 33/36 2024-02-05T04:49:55.311 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 18/31 2024-02-05T04:49:55.314 INFO:teuthology.orchestra.run.smithi091.stdout:Dependencies resolved. 2024-02-05T04:49:55.316 INFO:teuthology.orchestra.run.smithi091.stdout:======================================================================================== 2024-02-05T04:49:55.316 INFO:teuthology.orchestra.run.smithi091.stdout: Package Arch Version Repository Size 2024-02-05T04:49:55.316 INFO:teuthology.orchestra.run.smithi091.stdout:======================================================================================== 2024-02-05T04:49:55.316 INFO:teuthology.orchestra.run.smithi091.stdout:Installing: 2024-02-05T04:49:55.316 INFO:teuthology.orchestra.run.smithi091.stdout: ceph x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 6.5 k 2024-02-05T04:49:55.316 INFO:teuthology.orchestra.run.smithi091.stdout:Installing dependencies: 2024-02-05T04:49:55.317 INFO:teuthology.orchestra.run.smithi091.stdout: ceph-mds x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 2.1 M 2024-02-05T04:49:55.317 INFO:teuthology.orchestra.run.smithi091.stdout: ceph-mgr x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 1.5 M 2024-02-05T04:49:55.317 INFO:teuthology.orchestra.run.smithi091.stdout: ceph-mgr-modules-core noarch 2:19.0.0-658.gd4ed20cf.el9 ceph-noarch 245 k 2024-02-05T04:49:55.317 INFO:teuthology.orchestra.run.smithi091.stdout: ceph-mon x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 4.7 M 2024-02-05T04:49:55.317 INFO:teuthology.orchestra.run.smithi091.stdout: ceph-osd x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 17 M 2024-02-05T04:49:55.317 INFO:teuthology.orchestra.run.smithi091.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-02-05T04:49:55.317 INFO:teuthology.orchestra.run.smithi091.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-02-05T04:49:55.317 INFO:teuthology.orchestra.run.smithi091.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-02-05T04:49:55.317 INFO:teuthology.orchestra.run.smithi091.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-02-05T04:49:55.317 INFO:teuthology.orchestra.run.smithi091.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-02-05T04:49:55.317 INFO:teuthology.orchestra.run.smithi091.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-02-05T04:49:55.317 INFO:teuthology.orchestra.run.smithi091.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-02-05T04:49:55.318 INFO:teuthology.orchestra.run.smithi091.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-02-05T04:49:55.318 INFO:teuthology.orchestra.run.smithi091.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-02-05T04:49:55.318 INFO:teuthology.orchestra.run.smithi091.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-02-05T04:49:55.318 INFO:teuthology.orchestra.run.smithi091.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-02-05T04:49:55.318 INFO:teuthology.orchestra.run.smithi091.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-02-05T04:49:55.318 INFO:teuthology.orchestra.run.smithi091.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-02-05T04:49:55.318 INFO:teuthology.orchestra.run.smithi091.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-02-05T04:49:55.318 INFO:teuthology.orchestra.run.smithi091.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-02-05T04:49:55.318 INFO:teuthology.orchestra.run.smithi091.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-02-05T04:49:55.318 INFO:teuthology.orchestra.run.smithi091.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-02-05T04:49:55.319 INFO:teuthology.orchestra.run.smithi091.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-02-05T04:49:55.319 INFO:teuthology.orchestra.run.smithi091.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-02-05T04:49:55.319 INFO:teuthology.orchestra.run.smithi091.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-02-05T04:49:55.319 INFO:teuthology.orchestra.run.smithi091.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-02-05T04:49:55.319 INFO:teuthology.orchestra.run.smithi091.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-02-05T04:49:55.319 INFO:teuthology.orchestra.run.smithi091.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-02-05T04:49:55.319 INFO:teuthology.orchestra.run.smithi091.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-02-05T04:49:55.319 INFO:teuthology.orchestra.run.smithi091.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-02-05T04:49:55.319 INFO:teuthology.orchestra.run.smithi091.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-02-05T04:49:55.319 INFO:teuthology.orchestra.run.smithi091.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-02-05T04:49:55.319 INFO:teuthology.orchestra.run.smithi091.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-02-05T04:49:55.319 INFO:teuthology.orchestra.run.smithi091.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-02-05T04:49:55.320 INFO:teuthology.orchestra.run.smithi091.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-02-05T04:49:55.320 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:49:55.320 INFO:teuthology.orchestra.run.smithi091.stdout:Transaction Summary 2024-02-05T04:49:55.320 INFO:teuthology.orchestra.run.smithi091.stdout:======================================================================================== 2024-02-05T04:49:55.320 INFO:teuthology.orchestra.run.smithi091.stdout:Install 36 Packages 2024-02-05T04:49:55.320 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:49:55.321 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-osd-2:19.0.0-658.gd4ed20cf.el9.x86_64 33/36 2024-02-05T04:49:55.321 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T04:49:55.321 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-02-05T04:49:55.321 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-05T04:49:55.321 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-05T04:49:55.321 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:49:55.322 INFO:teuthology.orchestra.run.smithi091.stdout:Total download size: 30 M 2024-02-05T04:49:55.322 INFO:teuthology.orchestra.run.smithi091.stdout:Installed size: 106 M 2024-02-05T04:49:55.322 INFO:teuthology.orchestra.run.smithi091.stdout:Downloading Packages: 2024-02-05T04:49:55.339 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : librbd1-2:19.0.0-658.gd4ed20cf.el9.x86_64 19/31 2024-02-05T04:49:55.389 INFO:teuthology.orchestra.run.smithi062.stdout:(1/36): ceph-19.0.0-658.gd4ed20cf.el9.x86_64.rp 37 kB/s | 6.5 kB 00:00 2024-02-05T04:49:55.429 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librbd1-2:19.0.0-658.gd4ed20cf.el9.x86_64 19/31 2024-02-05T04:49:55.460 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-rbd-2:19.0.0-658.gd4ed20cf.el9.x86_64 20/31 2024-02-05T04:49:55.504 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 21/31 2024-02-05T04:49:55.664 INFO:teuthology.orchestra.run.smithi062.stdout:(2/36): ceph-mgr-19.0.0-658.gd4ed20cf.el9.x86_6 3.3 MB/s | 1.5 MB 00:00 2024-02-05T04:49:55.756 INFO:teuthology.orchestra.run.smithi091.stdout:(1/36): ceph-19.0.0-658.gd4ed20cf.el9.x86_64.rp 27 kB/s | 6.5 kB 00:00 2024-02-05T04:49:55.759 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mon-2:19.0.0-658.gd4ed20cf.el9.x86_64 34/36 2024-02-05T04:49:55.783 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mon-2:19.0.0-658.gd4ed20cf.el9.x86_64 34/36 2024-02-05T04:49:55.783 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T04:49:55.783 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-02-05T04:49:55.783 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-05T04:49:55.783 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-05T04:49:55.783 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:49:55.933 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 22/31 2024-02-05T04:49:55.956 INFO:teuthology.orchestra.run.smithi062.stdout:(3/36): ceph-mon-19.0.0-658.gd4ed20cf.el9.x86_6 8.3 MB/s | 4.7 MB 00:00 2024-02-05T04:49:55.958 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : librgw2-2:19.0.0-658.gd4ed20cf.el9.x86_64 23/31 2024-02-05T04:49:56.002 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mds-2:19.0.0-658.gd4ed20cf.el9.x86_64 35/36 2024-02-05T04:49:56.006 INFO:teuthology.orchestra.run.smithi062.stdout:(4/36): ceph-mds-19.0.0-658.gd4ed20cf.el9.x86_6 2.7 MB/s | 2.1 MB 00:00 2024-02-05T04:49:56.012 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librgw2-2:19.0.0-658.gd4ed20cf.el9.x86_64 23/31 2024-02-05T04:49:56.025 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mds-2:19.0.0-658.gd4ed20cf.el9.x86_64 35/36 2024-02-05T04:49:56.025 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T04:49:56.025 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-02-05T04:49:56.025 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-05T04:49:56.025 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-05T04:49:56.026 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:49:56.032 INFO:teuthology.orchestra.run.smithi062.stdout:(5/36): ceph-mgr-modules-core-19.0.0-658.gd4ed2 3.2 MB/s | 245 kB 00:00 2024-02-05T04:49:56.080 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-rgw-2:19.0.0-658.gd4ed20cf.el9.x86_64 24/31 2024-02-05T04:49:56.091 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-2:19.0.0-658.gd4ed20cf.el9.x86_64 36/36 2024-02-05T04:49:56.119 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-ceph-common-2:19.0.0-658.gd4ed20cf.el9.x86 25/31 2024-02-05T04:49:56.165 INFO:teuthology.orchestra.run.smithi062.stdout:(6/36): lua-5.4.4-4.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-02-05T04:49:56.191 INFO:teuthology.orchestra.run.smithi062.stdout:(7/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.1 MB/s | 144 kB 00:00 2024-02-05T04:49:56.216 INFO:teuthology.orchestra.run.smithi062.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 630 kB/s | 32 kB 00:00 2024-02-05T04:49:56.241 INFO:teuthology.orchestra.run.smithi062.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 939 kB/s | 47 kB 00:00 2024-02-05T04:49:56.267 INFO:teuthology.orchestra.run.smithi062.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 767 kB/s | 39 kB 00:00 2024-02-05T04:49:56.292 INFO:teuthology.orchestra.run.smithi062.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-02-05T04:49:56.323 INFO:teuthology.orchestra.run.smithi091.stdout:(2/36): ceph-mds-19.0.0-658.gd4ed20cf.el9.x86_6 2.6 MB/s | 2.1 MB 00:00 2024-02-05T04:49:56.342 INFO:teuthology.orchestra.run.smithi062.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-02-05T04:49:56.367 INFO:teuthology.orchestra.run.smithi062.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.3 MB/s | 103 kB 00:00 2024-02-05T04:49:56.393 INFO:teuthology.orchestra.run.smithi062.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-02-05T04:49:56.421 INFO:teuthology.orchestra.run.smithi062.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 2.1 MB/s | 113 kB 00:00 2024-02-05T04:49:56.446 INFO:teuthology.orchestra.run.smithi062.stdout:(16/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.5 MB/s | 187 kB 00:00 2024-02-05T04:49:56.588 INFO:teuthology.orchestra.run.smithi062.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 1.0 MB/s | 151 kB 00:00 2024-02-05T04:49:56.630 INFO:teuthology.orchestra.run.smithi062.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.0 MB/s | 43 kB 00:00 2024-02-05T04:49:56.756 INFO:teuthology.orchestra.run.smithi091.stdout:(3/36): ceph-mgr-19.0.0-658.gd4ed20cf.el9.x86_6 1.2 MB/s | 1.5 MB 00:01 2024-02-05T04:49:56.814 INFO:teuthology.orchestra.run.smithi062.stdout:(19/36): ceph-osd-19.0.0-658.gd4ed20cf.el9.x86_ 15 MB/s | 17 MB 00:01 2024-02-05T04:49:56.864 INFO:teuthology.orchestra.run.smithi062.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 735 kB/s | 172 kB 00:00 2024-02-05T04:49:56.881 INFO:teuthology.orchestra.run.smithi091.stdout:(4/36): ceph-mgr-modules-core-19.0.0-658.gd4ed2 1.9 MB/s | 245 kB 00:00 2024-02-05T04:49:56.906 INFO:teuthology.orchestra.run.smithi062.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 255 kB/s | 11 kB 00:00 2024-02-05T04:49:56.956 INFO:teuthology.orchestra.run.smithi062.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 354 kB/s | 18 kB 00:00 2024-02-05T04:49:56.990 INFO:teuthology.orchestra.run.smithi091.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.7 MB/s | 187 kB 00:00 2024-02-05T04:49:57.006 INFO:teuthology.orchestra.run.smithi062.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 464 kB/s | 23 kB 00:00 2024-02-05T04:49:57.032 INFO:teuthology.orchestra.run.smithi091.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-02-05T04:49:57.049 INFO:teuthology.orchestra.run.smithi062.stdout:(24/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.7 MB/s | 358 kB 00:00 2024-02-05T04:49:57.065 INFO:teuthology.orchestra.run.smithi091.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 947 kB/s | 32 kB 00:00 2024-02-05T04:49:57.074 INFO:teuthology.orchestra.run.smithi062.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 289 kB/s | 19 kB 00:00 2024-02-05T04:49:57.099 INFO:teuthology.orchestra.run.smithi091.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-02-05T04:49:57.100 INFO:teuthology.orchestra.run.smithi062.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 389 kB/s | 20 kB 00:00 2024-02-05T04:49:57.124 INFO:teuthology.orchestra.run.smithi062.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 920 kB/s | 46 kB 00:00 2024-02-05T04:49:57.133 INFO:teuthology.orchestra.run.smithi091.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-02-05T04:49:57.167 INFO:teuthology.orchestra.run.smithi062.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 1.1 MB/s | 79 kB 00:00 2024-02-05T04:49:57.175 INFO:teuthology.orchestra.run.smithi091.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2024-02-05T04:49:57.192 INFO:teuthology.orchestra.run.smithi062.stdout:(29/36): lua-devel-5.4.4-4.el9.x86_64.rpm 29 kB/s | 22 kB 00:00 2024-02-05T04:49:57.217 INFO:teuthology.orchestra.run.smithi062.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 331 kB/s | 16 kB 00:00 2024-02-05T04:49:57.242 INFO:teuthology.orchestra.run.smithi091.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2024-02-05T04:49:57.243 INFO:teuthology.orchestra.run.smithi062.stdout:(31/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.3 MB/s | 272 kB 00:00 2024-02-05T04:49:57.268 INFO:teuthology.orchestra.run.smithi062.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 710 kB/s | 36 kB 00:00 2024-02-05T04:49:57.276 INFO:teuthology.orchestra.run.smithi091.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-02-05T04:49:57.309 INFO:teuthology.orchestra.run.smithi091.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.7 MB/s | 124 kB 00:00 2024-02-05T04:49:57.326 INFO:teuthology.orchestra.run.smithi062.stdout:(33/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 671 kB/s | 90 kB 00:00 2024-02-05T04:49:57.343 INFO:teuthology.orchestra.run.smithi091.stdout:(14/36): python3-requests-2.25.1-8.el9.noarch.r 3.3 MB/s | 113 kB 00:00 2024-02-05T04:49:57.351 INFO:teuthology.orchestra.run.smithi062.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-02-05T04:49:57.377 INFO:teuthology.orchestra.run.smithi062.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 399 kB/s | 20 kB 00:00 2024-02-05T04:49:57.384 INFO:teuthology.orchestra.run.smithi091.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-02-05T04:49:57.419 INFO:teuthology.orchestra.run.smithi062.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.8 MB/s | 427 kB 00:00 2024-02-05T04:49:57.423 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:49:57.423 INFO:teuthology.orchestra.run.smithi062.stdout:Total 11 MB/s | 30 MB 00:02 2024-02-05T04:49:57.445 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2024-02-05T04:49:57.489 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2024-02-05T04:49:57.489 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2024-02-05T04:49:57.752 INFO:teuthology.orchestra.run.smithi091.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 61 kB/s | 22 kB 00:00 2024-02-05T04:49:57.819 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-2:19.0.0-658.gd4ed20cf.el9.x86_64 36/36 2024-02-05T04:49:57.819 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-2:19.0.0-658.gd4ed20cf.el9.x86_64 1/36 2024-02-05T04:49:57.819 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mds-2:19.0.0-658.gd4ed20cf.el9.x86_64 2/36 2024-02-05T04:49:57.819 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-2:19.0.0-658.gd4ed20cf.el9.x86_64 3/36 2024-02-05T04:49:57.819 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mon-2:19.0.0-658.gd4ed20cf.el9.x86_64 4/36 2024-02-05T04:49:57.819 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-osd-2:19.0.0-658.gd4ed20cf.el9.x86_64 5/36 2024-02-05T04:49:57.819 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-658.gd4ed20cf.el9.n 6/36 2024-02-05T04:49:57.819 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-02-05T04:49:57.819 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-02-05T04:49:57.820 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-02-05T04:49:57.820 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-02-05T04:49:57.820 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-02-05T04:49:57.820 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-02-05T04:49:57.820 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-02-05T04:49:57.820 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-02-05T04:49:57.820 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-02-05T04:49:57.820 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 16/36 2024-02-05T04:49:57.820 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 17/36 2024-02-05T04:49:57.820 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-02-05T04:49:57.820 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-02-05T04:49:57.820 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-02-05T04:49:57.820 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-02-05T04:49:57.821 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-02-05T04:49:57.821 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-02-05T04:49:57.821 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-02-05T04:49:57.821 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-02-05T04:49:57.821 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-02-05T04:49:57.821 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-02-05T04:49:57.821 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-02-05T04:49:57.822 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-02-05T04:49:57.822 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-02-05T04:49:57.822 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-02-05T04:49:57.822 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-02-05T04:49:57.822 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-02-05T04:49:57.822 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-02-05T04:49:57.822 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-02-05T04:49:57.923 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2024-02-05T04:49:57.923 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2024-02-05T04:49:57.985 INFO:teuthology.orchestra.run.smithi091.stdout:(17/36): ceph-mon-19.0.0-658.gd4ed20cf.el9.x86_ 2.1 MB/s | 4.7 MB 00:02 2024-02-05T04:49:58.011 INFO:teuthology.orchestra.run.smithi091.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 584 kB/s | 151 kB 00:00 2024-02-05T04:49:58.211 INFO:teuthology.orchestra.run.smithi091.stdout:(19/36): ceph-osd-19.0.0-658.gd4ed20cf.el9.x86_ 8.9 MB/s | 17 MB 00:01 2024-02-05T04:49:58.236 INFO:teuthology.orchestra.run.smithi091.stdout:(20/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 173 kB/s | 43 kB 00:00 2024-02-05T04:49:58.260 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-common-2:19.0.0-658.gd4ed20cf.el9.x86_64 26/31 2024-02-05T04:49:58.261 INFO:teuthology.orchestra.run.smithi091.stdout:(21/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 685 kB/s | 172 kB 00:00 2024-02-05T04:49:58.279 INFO:teuthology.orchestra.run.smithi091.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 255 kB/s | 11 kB 00:00 2024-02-05T04:49:58.300 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-common-2:19.0.0-658.gd4ed20cf.el9.x86_64 26/31 2024-02-05T04:49:58.304 INFO:teuthology.orchestra.run.smithi091.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 422 kB/s | 18 kB 00:00 2024-02-05T04:49:58.326 INFO:teuthology.orchestra.run.smithi091.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 488 kB/s | 23 kB 00:00 2024-02-05T04:49:58.346 INFO:teuthology.orchestra.run.smithi091.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 461 kB/s | 19 kB 00:00 2024-02-05T04:49:58.363 INFO:teuthology.orchestra.run.smithi091.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 539 kB/s | 20 kB 00:00 2024-02-05T04:49:58.388 INFO:teuthology.orchestra.run.smithi091.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 1.1 MB/s | 46 kB 00:00 2024-02-05T04:49:58.422 INFO:teuthology.orchestra.run.smithi091.stdout:(28/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.7 MB/s | 358 kB 00:00 2024-02-05T04:49:58.447 INFO:teuthology.orchestra.run.smithi091.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 939 kB/s | 79 kB 00:00 2024-02-05T04:49:58.472 INFO:teuthology.orchestra.run.smithi091.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 327 kB/s | 16 kB 00:00 2024-02-05T04:49:58.506 INFO:teuthology.orchestra.run.smithi091.stdout:(31/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.3 MB/s | 272 kB 00:00 2024-02-05T04:49:58.531 INFO:teuthology.orchestra.run.smithi091.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.0 MB/s | 90 kB 00:00 2024-02-05T04:49:58.557 INFO:teuthology.orchestra.run.smithi091.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 426 kB/s | 36 kB 00:00 2024-02-05T04:49:58.607 INFO:teuthology.orchestra.run.smithi091.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-02-05T04:49:58.632 INFO:teuthology.orchestra.run.smithi091.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 265 kB/s | 20 kB 00:00 2024-02-05T04:49:58.715 INFO:teuthology.orchestra.run.smithi091.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.3 MB/s | 427 kB 00:00 2024-02-05T04:49:58.720 INFO:teuthology.orchestra.run.smithi091.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:49:58.720 INFO:teuthology.orchestra.run.smithi091.stdout:Total 8.7 MB/s | 30 MB 00:03 2024-02-05T04:49:58.737 INFO:teuthology.orchestra.run.smithi091.stdout:Running transaction check 2024-02-05T04:49:58.778 INFO:teuthology.orchestra.run.smithi091.stdout:Transaction check succeeded. 2024-02-05T04:49:58.778 INFO:teuthology.orchestra.run.smithi091.stdout:Running transaction test 2024-02-05T04:49:58.835 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-common-2:19.0.0-658.gd4ed20cf.el9.x86_64 26/31 2024-02-05T04:49:58.860 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-02-05T04:49:58.861 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:49:58.861 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-02-05T04:49:58.861 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:49:58.861 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mds-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:49:58.861 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:49:58.861 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-modules-core-2:19.0.0-658.gd4ed20cf.el9.noarch 2024-02-05T04:49:58.861 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mon-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:49:58.861 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-osd-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:49:58.861 INFO:teuthology.orchestra.run.smithi196.stdout: lua-5.4.4-4.el9.x86_64 2024-02-05T04:49:58.861 INFO:teuthology.orchestra.run.smithi196.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-02-05T04:49:58.861 INFO:teuthology.orchestra.run.smithi196.stdout: luarocks-3.9.2-1.el9.noarch 2024-02-05T04:49:58.861 INFO:teuthology.orchestra.run.smithi196.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-02-05T04:49:58.861 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-02-05T04:49:58.862 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-02-05T04:49:58.862 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-02-05T04:49:58.862 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-02-05T04:49:58.862 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-02-05T04:49:58.862 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-02-05T04:49:58.862 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-02-05T04:49:58.862 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-02-05T04:49:58.862 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-02-05T04:49:58.862 INFO:teuthology.orchestra.run.smithi196.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-02-05T04:49:58.862 INFO:teuthology.orchestra.run.smithi196.stdout: python3-mako-1.1.4-6.el9.noarch 2024-02-05T04:49:58.862 INFO:teuthology.orchestra.run.smithi196.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-02-05T04:49:58.862 INFO:teuthology.orchestra.run.smithi196.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-02-05T04:49:58.862 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-02-05T04:49:58.862 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ply-3.11-14.el9.noarch 2024-02-05T04:49:58.863 INFO:teuthology.orchestra.run.smithi196.stdout: python3-portend-3.1.0-2.el9.noarch 2024-02-05T04:49:58.863 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-02-05T04:49:58.863 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pycparser-2.20-6.el9.noarch 2024-02-05T04:49:58.863 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pytz-2021.1-5.el9.noarch 2024-02-05T04:49:58.863 INFO:teuthology.orchestra.run.smithi196.stdout: python3-requests-2.25.1-8.el9.noarch 2024-02-05T04:49:58.863 INFO:teuthology.orchestra.run.smithi196.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-02-05T04:49:58.863 INFO:teuthology.orchestra.run.smithi196.stdout: python3-toml-0.10.2-6.el9.noarch 2024-02-05T04:49:58.863 INFO:teuthology.orchestra.run.smithi196.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-02-05T04:49:58.863 INFO:teuthology.orchestra.run.smithi196.stdout: python3-webob-1.8.7-6.el9.noarch 2024-02-05T04:49:58.863 INFO:teuthology.orchestra.run.smithi196.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-02-05T04:49:58.863 INFO:teuthology.orchestra.run.smithi196.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-02-05T04:49:58.863 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:49:58.863 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-02-05T04:49:58.879 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-base-2:19.0.0-658.gd4ed20cf.el9.x86_64 27/31 2024-02-05T04:49:58.987 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-base-2:19.0.0-658.gd4ed20cf.el9.x86_64 27/31 2024-02-05T04:49:58.988 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-02-05T04:49:58.988 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-02-05T04:49:58.988 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:49:59.014 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-selinux-2:19.0.0-658.gd4ed20cf.el9.x86_64 28/31 2024-02-05T04:49:59.193 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-base 2024-02-05T04:49:59.195 INFO:teuthology.orchestra.run.smithi091.stdout:Transaction test succeeded. 2024-02-05T04:49:59.196 INFO:teuthology.orchestra.run.smithi091.stdout:Running transaction 2024-02-05T04:49:59.286 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2024-02-05T04:49:59.330 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-02-05T04:49:59.364 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-02-05T04:49:59.391 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-02-05T04:49:59.459 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-02-05T04:49:59.507 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-02-05T04:49:59.540 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-02-05T04:49:59.636 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-02-05T04:49:59.713 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-02-05T04:49:59.760 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-02-05T04:49:59.765 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:00 ago on Mon 05 Feb 2024 04:48:59 AM UTC. 2024-02-05T04:49:59.835 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-02-05T04:49:59.869 INFO:teuthology.orchestra.run.smithi196.stdout:Package ceph-base-2:19.0.0-658.gd4ed20cf.el9.x86_64 is already installed. 2024-02-05T04:49:59.894 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-02-05T04:49:59.921 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-02-05T04:49:59.922 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-02-05T04:49:59.922 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-02-05T04:49:59.932 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-02-05T04:49:59.984 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install cephadm 2024-02-05T04:49:59.990 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-02-05T04:50:00.046 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-02-05T04:50:00.242 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-02-05T04:50:00.303 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-02-05T04:50:00.341 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-02-05T04:50:00.385 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-02-05T04:50:00.414 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-02-05T04:50:00.494 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-02-05T04:50:00.525 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-02-05T04:50:00.532 INFO:teuthology.orchestra.run.smithi091.stdout: Preparing : 1/1 2024-02-05T04:50:00.549 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:01 ago on Mon 05 Feb 2024 04:48:59 AM UTC. 2024-02-05T04:50:00.558 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-02-05T04:50:00.599 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-02-05T04:50:00.602 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-02-05T04:50:00.642 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-02-05T04:50:00.643 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-02-05T04:50:00.676 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-02-05T04:50:00.679 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-02-05T04:50:00.706 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-02-05T04:50:00.707 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-02-05T04:50:00.707 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-02-05T04:50:00.707 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-02-05T04:50:00.707 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-02-05T04:50:00.707 INFO:teuthology.orchestra.run.smithi196.stdout: cephadm noarch 2:19.0.0-658.gd4ed20cf.el9 ceph-noarch 742 k 2024-02-05T04:50:00.707 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:50:00.707 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-02-05T04:50:00.707 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-02-05T04:50:00.708 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-02-05T04:50:00.708 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:50:00.708 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 742 k 2024-02-05T04:50:00.708 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 748 k 2024-02-05T04:50:00.708 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-02-05T04:50:00.750 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-02-05T04:50:00.772 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-02-05T04:50:00.795 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-02-05T04:50:00.816 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-02-05T04:50:00.827 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-02-05T04:50:00.891 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-02-05T04:50:00.920 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-02-05T04:50:00.997 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-02-05T04:50:01.034 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-02-05T04:50:01.038 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-02-05T04:50:01.110 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-02-05T04:50:01.163 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-02-05T04:50:01.165 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-02-05T04:50:01.202 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-02-05T04:50:01.218 INFO:teuthology.orchestra.run.smithi196.stdout:cephadm-19.0.0-658.gd4ed20cf.el9.noarch.rpm 1.4 MB/s | 742 kB 00:00 2024-02-05T04:50:01.218 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:50:01.219 INFO:teuthology.orchestra.run.smithi196.stdout:Total 1.4 MB/s | 742 kB 00:00 2024-02-05T04:50:01.219 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-02-05T04:50:01.224 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-02-05T04:50:01.224 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-02-05T04:50:01.230 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-02-05T04:50:01.230 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-02-05T04:50:01.260 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-02-05T04:50:01.300 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-02-05T04:50:01.315 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-02-05T04:50:01.332 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-658.gd4ed20cf.el9.n 31/36 2024-02-05T04:50:01.370 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-mgr-2:19.0.0-658.gd4ed20cf.el9.x86_64 32/36 2024-02-05T04:50:01.407 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-mgr-2:19.0.0-658.gd4ed20cf.el9.x86_64 32/36 2024-02-05T04:50:01.407 INFO:teuthology.orchestra.run.smithi062.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T04:50:01.407 INFO:teuthology.orchestra.run.smithi062.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-02-05T04:50:01.408 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-05T04:50:01.408 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-05T04:50:01.408 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:50:01.502 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-02-05T04:50:01.550 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-02-05T04:50:01.589 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-02-05T04:50:01.640 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-02-05T04:50:01.671 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-02-05T04:50:01.680 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: cephadm-2:19.0.0-658.gd4ed20cf.el9.noarch 1/1 2024-02-05T04:50:01.745 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-02-05T04:50:01.764 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : cephadm-2:19.0.0-658.gd4ed20cf.el9.noarch 1/1 2024-02-05T04:50:01.780 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-02-05T04:50:01.814 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-02-05T04:50:01.854 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-02-05T04:50:01.889 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-02-05T04:50:01.924 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-02-05T04:50:02.012 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-02-05T04:50:02.062 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-02-05T04:50:02.135 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-02-05T04:50:02.156 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: cephadm-2:19.0.0-658.gd4ed20cf.el9.noarch 1/1 2024-02-05T04:50:02.272 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-02-05T04:50:02.398 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-02-05T04:50:02.452 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : cephadm-2:19.0.0-658.gd4ed20cf.el9.noarch 1/1 2024-02-05T04:50:02.452 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:50:02.452 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-02-05T04:50:02.452 INFO:teuthology.orchestra.run.smithi196.stdout: cephadm-2:19.0.0-658.gd4ed20cf.el9.noarch 2024-02-05T04:50:02.452 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:50:02.452 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-02-05T04:50:02.554 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-658.gd4ed20cf.el9.n 31/36 2024-02-05T04:50:02.584 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : ceph-mgr-2:19.0.0-658.gd4ed20cf.el9.x86_64 32/36 2024-02-05T04:50:02.620 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: ceph-mgr-2:19.0.0-658.gd4ed20cf.el9.x86_64 32/36 2024-02-05T04:50:02.621 INFO:teuthology.orchestra.run.smithi091.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T04:50:02.621 INFO:teuthology.orchestra.run.smithi091.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-02-05T04:50:02.621 INFO:teuthology.orchestra.run.smithi091.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-05T04:50:02.621 INFO:teuthology.orchestra.run.smithi091.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-05T04:50:02.621 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:50:02.674 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-immutable-object-cache 2024-02-05T04:50:02.921 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-osd-2:19.0.0-658.gd4ed20cf.el9.x86_64 33/36 2024-02-05T04:50:02.947 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-osd-2:19.0.0-658.gd4ed20cf.el9.x86_64 33/36 2024-02-05T04:50:02.948 INFO:teuthology.orchestra.run.smithi062.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T04:50:02.948 INFO:teuthology.orchestra.run.smithi062.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-02-05T04:50:02.948 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-05T04:50:02.948 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-05T04:50:02.948 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:50:03.243 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:04 ago on Mon 05 Feb 2024 04:48:59 AM UTC. 2024-02-05T04:50:03.400 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-02-05T04:50:03.401 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-02-05T04:50:03.401 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repo Size 2024-02-05T04:50:03.401 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-02-05T04:50:03.401 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-02-05T04:50:03.401 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 145 k 2024-02-05T04:50:03.402 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:50:03.402 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-02-05T04:50:03.402 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-02-05T04:50:03.402 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-02-05T04:50:03.402 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:50:03.402 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 145 k 2024-02-05T04:50:03.402 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 436 k 2024-02-05T04:50:03.402 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-02-05T04:50:03.414 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-mon-2:19.0.0-658.gd4ed20cf.el9.x86_64 34/36 2024-02-05T04:50:03.438 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-mon-2:19.0.0-658.gd4ed20cf.el9.x86_64 34/36 2024-02-05T04:50:03.439 INFO:teuthology.orchestra.run.smithi062.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T04:50:03.439 INFO:teuthology.orchestra.run.smithi062.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-02-05T04:50:03.439 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-05T04:50:03.439 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-05T04:50:03.439 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:50:03.624 INFO:teuthology.orchestra.run.smithi196.stdout:ceph-immutable-object-cache-19.0.0-658.gd4ed20c 655 kB/s | 145 kB 00:00 2024-02-05T04:50:03.624 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:50:03.625 INFO:teuthology.orchestra.run.smithi196.stdout:Total 651 kB/s | 145 kB 00:00 2024-02-05T04:50:03.625 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-02-05T04:50:03.632 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-02-05T04:50:03.632 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-02-05T04:50:03.656 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-mds-2:19.0.0-658.gd4ed20cf.el9.x86_64 35/36 2024-02-05T04:50:03.681 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-mds-2:19.0.0-658.gd4ed20cf.el9.x86_64 35/36 2024-02-05T04:50:03.681 INFO:teuthology.orchestra.run.smithi062.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T04:50:03.681 INFO:teuthology.orchestra.run.smithi062.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-02-05T04:50:03.681 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-05T04:50:03.681 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-05T04:50:03.681 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:50:03.684 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-02-05T04:50:03.684 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-02-05T04:50:03.748 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-2:19.0.0-658.gd4ed20cf.el9.x86_64 36/36 2024-02-05T04:50:03.841 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-02-05T04:50:03.900 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-658.gd4ed20cf.e 1/1 2024-02-05T04:50:03.924 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-658.gd4ed20cf.e 1/1 2024-02-05T04:50:03.925 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T04:50:03.925 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-02-05T04:50:03.925 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:50:04.060 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : ceph-osd-2:19.0.0-658.gd4ed20cf.el9.x86_64 33/36 2024-02-05T04:50:04.087 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: ceph-osd-2:19.0.0-658.gd4ed20cf.el9.x86_64 33/36 2024-02-05T04:50:04.087 INFO:teuthology.orchestra.run.smithi091.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T04:50:04.088 INFO:teuthology.orchestra.run.smithi091.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-02-05T04:50:04.088 INFO:teuthology.orchestra.run.smithi091.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-05T04:50:04.088 INFO:teuthology.orchestra.run.smithi091.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-05T04:50:04.088 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:50:04.528 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : ceph-mon-2:19.0.0-658.gd4ed20cf.el9.x86_64 34/36 2024-02-05T04:50:04.551 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: ceph-mon-2:19.0.0-658.gd4ed20cf.el9.x86_64 34/36 2024-02-05T04:50:04.551 INFO:teuthology.orchestra.run.smithi091.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T04:50:04.551 INFO:teuthology.orchestra.run.smithi091.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-02-05T04:50:04.551 INFO:teuthology.orchestra.run.smithi091.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-05T04:50:04.552 INFO:teuthology.orchestra.run.smithi091.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-05T04:50:04.552 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:50:04.761 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : ceph-mds-2:19.0.0-658.gd4ed20cf.el9.x86_64 35/36 2024-02-05T04:50:04.785 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: ceph-mds-2:19.0.0-658.gd4ed20cf.el9.x86_64 35/36 2024-02-05T04:50:04.785 INFO:teuthology.orchestra.run.smithi091.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T04:50:04.785 INFO:teuthology.orchestra.run.smithi091.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-02-05T04:50:04.785 INFO:teuthology.orchestra.run.smithi091.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-05T04:50:04.786 INFO:teuthology.orchestra.run.smithi091.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-05T04:50:04.786 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:50:04.851 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : ceph-2:19.0.0-658.gd4ed20cf.el9.x86_64 36/36 2024-02-05T04:50:04.926 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-658.gd4ed20cf.e 1/1 2024-02-05T04:50:04.926 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:50:04.926 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-02-05T04:50:04.927 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-immutable-object-cache-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:50:04.927 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:50:04.927 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-02-05T04:50:05.165 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-mgr 2024-02-05T04:50:05.526 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-2:19.0.0-658.gd4ed20cf.el9.x86_64 36/36 2024-02-05T04:50:05.526 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-2:19.0.0-658.gd4ed20cf.el9.x86_64 1/36 2024-02-05T04:50:05.526 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-mds-2:19.0.0-658.gd4ed20cf.el9.x86_64 2/36 2024-02-05T04:50:05.526 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-mgr-2:19.0.0-658.gd4ed20cf.el9.x86_64 3/36 2024-02-05T04:50:05.526 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-mon-2:19.0.0-658.gd4ed20cf.el9.x86_64 4/36 2024-02-05T04:50:05.526 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-osd-2:19.0.0-658.gd4ed20cf.el9.x86_64 5/36 2024-02-05T04:50:05.526 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-658.gd4ed20cf.el9.n 6/36 2024-02-05T04:50:05.526 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-02-05T04:50:05.527 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-02-05T04:50:05.527 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-02-05T04:50:05.527 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-02-05T04:50:05.527 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-02-05T04:50:05.527 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-02-05T04:50:05.527 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-02-05T04:50:05.527 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-02-05T04:50:05.527 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-02-05T04:50:05.527 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 16/36 2024-02-05T04:50:05.527 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 17/36 2024-02-05T04:50:05.527 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-02-05T04:50:05.527 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-02-05T04:50:05.527 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-02-05T04:50:05.527 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-02-05T04:50:05.527 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-02-05T04:50:05.528 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-02-05T04:50:05.528 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-02-05T04:50:05.528 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-02-05T04:50:05.528 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-02-05T04:50:05.528 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-02-05T04:50:05.528 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-02-05T04:50:05.528 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-02-05T04:50:05.528 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-02-05T04:50:05.529 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-02-05T04:50:05.529 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-02-05T04:50:05.529 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-02-05T04:50:05.529 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-02-05T04:50:05.529 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-02-05T04:50:05.750 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:06 ago on Mon 05 Feb 2024 04:48:59 AM UTC. 2024-02-05T04:50:05.854 INFO:teuthology.orchestra.run.smithi196.stdout:Package ceph-mgr-2:19.0.0-658.gd4ed20cf.el9.x86_64 is already installed. 2024-02-05T04:50:05.905 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-02-05T04:50:05.906 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-02-05T04:50:05.906 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-02-05T04:50:05.970 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-mgr-dashboard 2024-02-05T04:50:06.534 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:07 ago on Mon 05 Feb 2024 04:48:59 AM UTC. 2024-02-05T04:50:06.651 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: ceph-2:19.0.0-658.gd4ed20cf.el9.x86_64 36/36 2024-02-05T04:50:06.652 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : ceph-2:19.0.0-658.gd4ed20cf.el9.x86_64 1/36 2024-02-05T04:50:06.652 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : ceph-mds-2:19.0.0-658.gd4ed20cf.el9.x86_64 2/36 2024-02-05T04:50:06.652 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : ceph-mgr-2:19.0.0-658.gd4ed20cf.el9.x86_64 3/36 2024-02-05T04:50:06.652 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : ceph-mon-2:19.0.0-658.gd4ed20cf.el9.x86_64 4/36 2024-02-05T04:50:06.652 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : ceph-osd-2:19.0.0-658.gd4ed20cf.el9.x86_64 5/36 2024-02-05T04:50:06.652 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-658.gd4ed20cf.el9.n 6/36 2024-02-05T04:50:06.652 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-02-05T04:50:06.652 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-02-05T04:50:06.652 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-02-05T04:50:06.652 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-02-05T04:50:06.652 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-02-05T04:50:06.653 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-02-05T04:50:06.653 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-02-05T04:50:06.653 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-02-05T04:50:06.653 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-02-05T04:50:06.653 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 16/36 2024-02-05T04:50:06.653 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 17/36 2024-02-05T04:50:06.653 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-02-05T04:50:06.653 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-02-05T04:50:06.653 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-02-05T04:50:06.653 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-02-05T04:50:06.653 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-02-05T04:50:06.653 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-02-05T04:50:06.653 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-02-05T04:50:06.653 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-02-05T04:50:06.654 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-02-05T04:50:06.654 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-02-05T04:50:06.654 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-02-05T04:50:06.654 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-02-05T04:50:06.654 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-02-05T04:50:06.654 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-02-05T04:50:06.654 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-02-05T04:50:06.654 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-02-05T04:50:06.654 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-02-05T04:50:06.654 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-02-05T04:50:06.689 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-02-05T04:50:06.690 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-02-05T04:50:06.690 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-02-05T04:50:06.690 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-02-05T04:50:06.690 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-02-05T04:50:06.690 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-dashboard noarch 2:19.0.0-658.gd4ed20cf.el9 ceph-noarch 1.7 M 2024-02-05T04:50:06.690 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-02-05T04:50:06.690 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-grafana-dashboards noarch 2:19.0.0-658.gd4ed20cf.el9 ceph-noarch 28 k 2024-02-05T04:50:06.690 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-prometheus-alerts noarch 2:19.0.0-658.gd4ed20cf.el9 ceph-noarch 15 k 2024-02-05T04:50:06.690 INFO:teuthology.orchestra.run.smithi196.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-02-05T04:50:06.690 INFO:teuthology.orchestra.run.smithi196.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-02-05T04:50:06.691 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:50:06.691 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-02-05T04:50:06.691 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-02-05T04:50:06.691 INFO:teuthology.orchestra.run.smithi196.stdout:Install 5 Packages 2024-02-05T04:50:06.691 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:50:06.691 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 2.0 M 2024-02-05T04:50:06.691 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 8.4 M 2024-02-05T04:50:06.692 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-02-05T04:50:06.692 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-02-05T04:50:06.692 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:50:06.692 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2024-02-05T04:50:06.692 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:50:06.692 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mds-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:50:06.692 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:50:06.692 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-modules-core-2:19.0.0-658.gd4ed20cf.el9.noarch 2024-02-05T04:50:06.692 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mon-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:50:06.693 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-osd-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:50:06.693 INFO:teuthology.orchestra.run.smithi062.stdout: lua-5.4.4-4.el9.x86_64 2024-02-05T04:50:06.693 INFO:teuthology.orchestra.run.smithi062.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-02-05T04:50:06.693 INFO:teuthology.orchestra.run.smithi062.stdout: luarocks-3.9.2-1.el9.noarch 2024-02-05T04:50:06.693 INFO:teuthology.orchestra.run.smithi062.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-02-05T04:50:06.693 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-02-05T04:50:06.693 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-02-05T04:50:06.693 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-02-05T04:50:06.693 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-02-05T04:50:06.693 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-02-05T04:50:06.693 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-02-05T04:50:06.693 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-02-05T04:50:06.693 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-02-05T04:50:06.694 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-02-05T04:50:06.694 INFO:teuthology.orchestra.run.smithi062.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-02-05T04:50:06.694 INFO:teuthology.orchestra.run.smithi062.stdout: python3-mako-1.1.4-6.el9.noarch 2024-02-05T04:50:06.694 INFO:teuthology.orchestra.run.smithi062.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-02-05T04:50:06.694 INFO:teuthology.orchestra.run.smithi062.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-02-05T04:50:06.694 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-02-05T04:50:06.694 INFO:teuthology.orchestra.run.smithi062.stdout: python3-ply-3.11-14.el9.noarch 2024-02-05T04:50:06.694 INFO:teuthology.orchestra.run.smithi062.stdout: python3-portend-3.1.0-2.el9.noarch 2024-02-05T04:50:06.694 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-02-05T04:50:06.694 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pycparser-2.20-6.el9.noarch 2024-02-05T04:50:06.694 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pytz-2021.1-5.el9.noarch 2024-02-05T04:50:06.694 INFO:teuthology.orchestra.run.smithi062.stdout: python3-requests-2.25.1-8.el9.noarch 2024-02-05T04:50:06.694 INFO:teuthology.orchestra.run.smithi062.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-02-05T04:50:06.694 INFO:teuthology.orchestra.run.smithi062.stdout: python3-toml-0.10.2-6.el9.noarch 2024-02-05T04:50:06.695 INFO:teuthology.orchestra.run.smithi062.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-02-05T04:50:06.695 INFO:teuthology.orchestra.run.smithi062.stdout: python3-webob-1.8.7-6.el9.noarch 2024-02-05T04:50:06.695 INFO:teuthology.orchestra.run.smithi062.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-02-05T04:50:06.695 INFO:teuthology.orchestra.run.smithi062.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-02-05T04:50:06.695 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:50:06.695 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-02-05T04:50:06.969 INFO:teuthology.orchestra.run.smithi196.stdout:(1/5): ceph-grafana-dashboards-19.0.0-658.gd4ed 136 kB/s | 28 kB 00:00 2024-02-05T04:50:07.003 INFO:teuthology.orchestra.run.smithi196.stdout:(2/5): ceph-prometheus-alerts-19.0.0-658.gd4ed2 63 kB/s | 15 kB 00:00 2024-02-05T04:50:07.043 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install ceph-base 2024-02-05T04:50:07.128 INFO:teuthology.orchestra.run.smithi196.stdout:(3/5): ceph-mgr-dashboard-19.0.0-658.gd4ed20cf. 4.8 MB/s | 1.7 MB 00:00 2024-02-05T04:50:07.620 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:01:00 ago on Mon 05 Feb 2024 04:49:07 AM UTC. 2024-02-05T04:50:07.653 INFO:teuthology.orchestra.run.smithi196.stdout:(4/5): python3-repoze-lru-0.7-16.el9.noarch.rpm 45 kB/s | 31 kB 00:00 2024-02-05T04:50:07.729 INFO:teuthology.orchestra.run.smithi062.stdout:Package ceph-base-2:19.0.0-658.gd4ed20cf.el9.x86_64 is already installed. 2024-02-05T04:50:07.784 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-02-05T04:50:07.785 INFO:teuthology.orchestra.run.smithi062.stdout:Nothing to do. 2024-02-05T04:50:07.785 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-02-05T04:50:07.794 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-02-05T04:50:07.795 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:50:07.795 INFO:teuthology.orchestra.run.smithi091.stdout:Installed: 2024-02-05T04:50:07.795 INFO:teuthology.orchestra.run.smithi091.stdout: ceph-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:50:07.795 INFO:teuthology.orchestra.run.smithi091.stdout: ceph-mds-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:50:07.795 INFO:teuthology.orchestra.run.smithi091.stdout: ceph-mgr-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:50:07.795 INFO:teuthology.orchestra.run.smithi091.stdout: ceph-mgr-modules-core-2:19.0.0-658.gd4ed20cf.el9.noarch 2024-02-05T04:50:07.795 INFO:teuthology.orchestra.run.smithi091.stdout: ceph-mon-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:50:07.795 INFO:teuthology.orchestra.run.smithi091.stdout: ceph-osd-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:50:07.795 INFO:teuthology.orchestra.run.smithi091.stdout: lua-5.4.4-4.el9.x86_64 2024-02-05T04:50:07.795 INFO:teuthology.orchestra.run.smithi091.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-02-05T04:50:07.795 INFO:teuthology.orchestra.run.smithi091.stdout: luarocks-3.9.2-1.el9.noarch 2024-02-05T04:50:07.795 INFO:teuthology.orchestra.run.smithi091.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-02-05T04:50:07.795 INFO:teuthology.orchestra.run.smithi091.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-02-05T04:50:07.795 INFO:teuthology.orchestra.run.smithi091.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-02-05T04:50:07.796 INFO:teuthology.orchestra.run.smithi091.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-02-05T04:50:07.796 INFO:teuthology.orchestra.run.smithi091.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-02-05T04:50:07.796 INFO:teuthology.orchestra.run.smithi091.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-02-05T04:50:07.796 INFO:teuthology.orchestra.run.smithi091.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-02-05T04:50:07.796 INFO:teuthology.orchestra.run.smithi091.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-02-05T04:50:07.796 INFO:teuthology.orchestra.run.smithi091.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-02-05T04:50:07.796 INFO:teuthology.orchestra.run.smithi091.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-02-05T04:50:07.796 INFO:teuthology.orchestra.run.smithi091.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-02-05T04:50:07.796 INFO:teuthology.orchestra.run.smithi091.stdout: python3-mako-1.1.4-6.el9.noarch 2024-02-05T04:50:07.796 INFO:teuthology.orchestra.run.smithi091.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-02-05T04:50:07.796 INFO:teuthology.orchestra.run.smithi091.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-02-05T04:50:07.796 INFO:teuthology.orchestra.run.smithi091.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-02-05T04:50:07.796 INFO:teuthology.orchestra.run.smithi091.stdout: python3-ply-3.11-14.el9.noarch 2024-02-05T04:50:07.797 INFO:teuthology.orchestra.run.smithi091.stdout: python3-portend-3.1.0-2.el9.noarch 2024-02-05T04:50:07.797 INFO:teuthology.orchestra.run.smithi091.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-02-05T04:50:07.797 INFO:teuthology.orchestra.run.smithi091.stdout: python3-pycparser-2.20-6.el9.noarch 2024-02-05T04:50:07.797 INFO:teuthology.orchestra.run.smithi091.stdout: python3-pytz-2021.1-5.el9.noarch 2024-02-05T04:50:07.797 INFO:teuthology.orchestra.run.smithi091.stdout: python3-requests-2.25.1-8.el9.noarch 2024-02-05T04:50:07.797 INFO:teuthology.orchestra.run.smithi091.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-02-05T04:50:07.797 INFO:teuthology.orchestra.run.smithi091.stdout: python3-toml-0.10.2-6.el9.noarch 2024-02-05T04:50:07.797 INFO:teuthology.orchestra.run.smithi091.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-02-05T04:50:07.797 INFO:teuthology.orchestra.run.smithi091.stdout: python3-webob-1.8.7-6.el9.noarch 2024-02-05T04:50:07.797 INFO:teuthology.orchestra.run.smithi091.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-02-05T04:50:07.797 INFO:teuthology.orchestra.run.smithi091.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-02-05T04:50:07.797 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:50:07.797 INFO:teuthology.orchestra.run.smithi091.stdout:Complete! 2024-02-05T04:50:07.847 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install cephadm 2024-02-05T04:50:08.143 DEBUG:teuthology.orchestra.run.smithi091:> sudo yum -y install ceph-base 2024-02-05T04:50:08.170 INFO:teuthology.orchestra.run.smithi196.stdout:(5/5): python3-routes-2.5.1-5.el9.noarch.rpm 162 kB/s | 188 kB 00:01 2024-02-05T04:50:08.174 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:50:08.174 INFO:teuthology.orchestra.run.smithi196.stdout:Total 1.3 MB/s | 2.0 MB 00:01 2024-02-05T04:50:08.175 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-02-05T04:50:08.183 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-02-05T04:50:08.183 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-02-05T04:50:08.250 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-02-05T04:50:08.250 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-02-05T04:50:08.498 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-02-05T04:50:08.528 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:01:01 ago on Mon 05 Feb 2024 04:49:07 AM UTC. 2024-02-05T04:50:08.544 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-selinux-2:19.0.0-658.gd4ed20cf.el9.x86_64 28/31 2024-02-05T04:50:08.544 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /sys 2024-02-05T04:50:08.544 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /proc 2024-02-05T04:50:08.544 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /mnt 2024-02-05T04:50:08.544 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /var/tmp 2024-02-05T04:50:08.544 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /home 2024-02-05T04:50:08.544 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /root 2024-02-05T04:50:08.544 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /tmp 2024-02-05T04:50:08.544 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:50:08.575 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/5 2024-02-05T04:50:08.608 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/5 2024-02-05T04:50:08.638 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-658.gd4ed20cf.el9.no 3/5 2024-02-05T04:50:08.689 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-02-05T04:50:08.690 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-02-05T04:50:08.690 INFO:teuthology.orchestra.run.smithi062.stdout: Package Arch Version Repository Size 2024-02-05T04:50:08.690 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-02-05T04:50:08.690 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2024-02-05T04:50:08.690 INFO:teuthology.orchestra.run.smithi062.stdout: cephadm noarch 2:19.0.0-658.gd4ed20cf.el9 ceph-noarch 742 k 2024-02-05T04:50:08.690 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:50:08.690 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2024-02-05T04:50:08.690 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-02-05T04:50:08.690 INFO:teuthology.orchestra.run.smithi062.stdout:Install 1 Package 2024-02-05T04:50:08.690 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:50:08.690 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 742 k 2024-02-05T04:50:08.690 INFO:teuthology.orchestra.run.smithi062.stdout:Installed size: 748 k 2024-02-05T04:50:08.690 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2024-02-05T04:50:08.720 INFO:teuthology.orchestra.run.smithi091.stdout:Last metadata expiration check: 0:00:59 ago on Mon 05 Feb 2024 04:49:09 AM UTC. 2024-02-05T04:50:08.823 INFO:teuthology.orchestra.run.smithi091.stdout:Package ceph-base-2:19.0.0-658.gd4ed20cf.el9.x86_64 is already installed. 2024-02-05T04:50:08.872 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-658.gd4ed20cf.el9.n 4/5 2024-02-05T04:50:08.875 INFO:teuthology.orchestra.run.smithi091.stdout:Dependencies resolved. 2024-02-05T04:50:08.876 INFO:teuthology.orchestra.run.smithi091.stdout:Nothing to do. 2024-02-05T04:50:08.876 INFO:teuthology.orchestra.run.smithi091.stdout:Complete! 2024-02-05T04:50:08.906 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-658.gd4ed20cf.el9.noarch 5/5 2024-02-05T04:50:08.940 DEBUG:teuthology.orchestra.run.smithi091:> sudo yum -y install cephadm 2024-02-05T04:50:09.400 INFO:teuthology.orchestra.run.smithi062.stdout:cephadm-19.0.0-658.gd4ed20cf.el9.noarch.rpm 1.0 MB/s | 742 kB 00:00 2024-02-05T04:50:09.401 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:50:09.401 INFO:teuthology.orchestra.run.smithi062.stdout:Total 1.0 MB/s | 742 kB 00:00 2024-02-05T04:50:09.401 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2024-02-05T04:50:09.407 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2024-02-05T04:50:09.407 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2024-02-05T04:50:09.413 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-radosgw-2:19.0.0-658.gd4ed20cf.el9.x86_64 29/31 2024-02-05T04:50:09.413 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2024-02-05T04:50:09.413 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2024-02-05T04:50:09.437 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-658.gd4ed20cf.el9.x86_64 29/31 2024-02-05T04:50:09.437 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T04:50:09.437 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-02-05T04:50:09.437 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-05T04:50:09.437 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-05T04:50:09.437 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:50:09.439 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 30/31 2024-02-05T04:50:09.466 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-658.gd4ed20cf.el9.noarch 5/5 2024-02-05T04:50:09.466 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-658.gd4ed20cf.el9.n 1/5 2024-02-05T04:50:09.466 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-658.gd4ed20cf.el9.noarch 2/5 2024-02-05T04:50:09.466 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-658.gd4ed20cf.el9.no 3/5 2024-02-05T04:50:09.466 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 4/5 2024-02-05T04:50:09.493 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2024-02-05T04:50:09.510 INFO:teuthology.orchestra.run.smithi091.stdout:Last metadata expiration check: 0:01:00 ago on Mon 05 Feb 2024 04:49:09 AM UTC. 2024-02-05T04:50:09.515 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 30/31 2024-02-05T04:50:09.516 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 31/31 2024-02-05T04:50:09.665 INFO:teuthology.orchestra.run.smithi091.stdout:Dependencies resolved. 2024-02-05T04:50:09.666 INFO:teuthology.orchestra.run.smithi091.stdout:================================================================================ 2024-02-05T04:50:09.666 INFO:teuthology.orchestra.run.smithi091.stdout: Package Arch Version Repository Size 2024-02-05T04:50:09.666 INFO:teuthology.orchestra.run.smithi091.stdout:================================================================================ 2024-02-05T04:50:09.666 INFO:teuthology.orchestra.run.smithi091.stdout:Installing: 2024-02-05T04:50:09.666 INFO:teuthology.orchestra.run.smithi091.stdout: cephadm noarch 2:19.0.0-658.gd4ed20cf.el9 ceph-noarch 742 k 2024-02-05T04:50:09.666 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:50:09.666 INFO:teuthology.orchestra.run.smithi091.stdout:Transaction Summary 2024-02-05T04:50:09.666 INFO:teuthology.orchestra.run.smithi091.stdout:================================================================================ 2024-02-05T04:50:09.666 INFO:teuthology.orchestra.run.smithi091.stdout:Install 1 Package 2024-02-05T04:50:09.666 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:50:09.667 INFO:teuthology.orchestra.run.smithi091.stdout:Total download size: 742 k 2024-02-05T04:50:09.667 INFO:teuthology.orchestra.run.smithi091.stdout:Installed size: 748 k 2024-02-05T04:50:09.667 INFO:teuthology.orchestra.run.smithi091.stdout:Downloading Packages: 2024-02-05T04:50:09.758 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 5/5 2024-02-05T04:50:09.759 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:50:09.759 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-02-05T04:50:09.759 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-grafana-dashboards-2:19.0.0-658.gd4ed20cf.el9.noarch 2024-02-05T04:50:09.759 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-dashboard-2:19.0.0-658.gd4ed20cf.el9.noarch 2024-02-05T04:50:09.759 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-prometheus-alerts-2:19.0.0-658.gd4ed20cf.el9.noarch 2024-02-05T04:50:09.759 INFO:teuthology.orchestra.run.smithi196.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-02-05T04:50:09.759 INFO:teuthology.orchestra.run.smithi196.stdout: python3-routes-2.5.1-5.el9.noarch 2024-02-05T04:50:09.759 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:50:09.759 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-02-05T04:50:09.965 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: cephadm-2:19.0.0-658.gd4ed20cf.el9.noarch 1/1 2024-02-05T04:50:09.985 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-05T04:50:10.049 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : cephadm-2:19.0.0-658.gd4ed20cf.el9.noarch 1/1 2024-02-05T04:50:10.334 INFO:teuthology.orchestra.run.smithi091.stdout:cephadm-19.0.0-658.gd4ed20cf.el9.noarch.rpm 1.1 MB/s | 742 kB 00:00 2024-02-05T04:50:10.335 INFO:teuthology.orchestra.run.smithi091.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:50:10.335 INFO:teuthology.orchestra.run.smithi091.stdout:Total 1.1 MB/s | 742 kB 00:00 2024-02-05T04:50:10.335 INFO:teuthology.orchestra.run.smithi091.stdout:Running transaction check 2024-02-05T04:50:10.340 INFO:teuthology.orchestra.run.smithi091.stdout:Transaction check succeeded. 2024-02-05T04:50:10.340 INFO:teuthology.orchestra.run.smithi091.stdout:Running transaction test 2024-02-05T04:50:10.346 INFO:teuthology.orchestra.run.smithi091.stdout:Transaction test succeeded. 2024-02-05T04:50:10.346 INFO:teuthology.orchestra.run.smithi091.stdout:Running transaction 2024-02-05T04:50:10.406 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: cephadm-2:19.0.0-658.gd4ed20cf.el9.noarch 1/1 2024-02-05T04:50:10.418 INFO:teuthology.orchestra.run.smithi091.stdout: Preparing : 1/1 2024-02-05T04:50:10.545 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:11 ago on Mon 05 Feb 2024 04:48:59 AM UTC. 2024-02-05T04:50:10.673 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 31/31 2024-02-05T04:50:10.673 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-base-2:19.0.0-658.gd4ed20cf.el9.x86_64 1/31 2024-02-05T04:50:10.673 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-common-2:19.0.0-658.gd4ed20cf.el9.x86_64 2/31 2024-02-05T04:50:10.673 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-radosgw-2:19.0.0-658.gd4ed20cf.el9.x86_64 3/31 2024-02-05T04:50:10.673 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-selinux-2:19.0.0-658.gd4ed20cf.el9.x86_64 4/31 2024-02-05T04:50:10.673 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libcephfs2-2:19.0.0-658.gd4ed20cf.el9.x86_64 5/31 2024-02-05T04:50:10.673 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libradosstriper1-2:19.0.0-658.gd4ed20cf.el9.x86_64 6/31 2024-02-05T04:50:10.673 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librgw2-2:19.0.0-658.gd4ed20cf.el9.x86_64 7/31 2024-02-05T04:50:10.673 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-ceph-argparse-2:19.0.0-658.gd4ed20cf.el9.x 8/31 2024-02-05T04:50:10.674 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-ceph-common-2:19.0.0-658.gd4ed20cf.el9.x86 9/31 2024-02-05T04:50:10.674 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cephfs-2:19.0.0-658.gd4ed20cf.el9.x86_64 10/31 2024-02-05T04:50:10.674 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-rados-2:19.0.0-658.gd4ed20cf.el9.x86_64 11/31 2024-02-05T04:50:10.674 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-rbd-2:19.0.0-658.gd4ed20cf.el9.x86_64 12/31 2024-02-05T04:50:10.674 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-rgw-2:19.0.0-658.gd4ed20cf.el9.x86_64 13/31 2024-02-05T04:50:10.674 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/31 2024-02-05T04:50:10.674 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/31 2024-02-05T04:50:10.674 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/31 2024-02-05T04:50:10.674 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/31 2024-02-05T04:50:10.674 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/31 2024-02-05T04:50:10.674 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/31 2024-02-05T04:50:10.674 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 20/31 2024-02-05T04:50:10.674 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 21/31 2024-02-05T04:50:10.675 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 22/31 2024-02-05T04:50:10.675 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 23/31 2024-02-05T04:50:10.675 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 24/31 2024-02-05T04:50:10.675 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 25/31 2024-02-05T04:50:10.675 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : re2-1:20211101-3.el9.x86_64 26/31 2024-02-05T04:50:10.675 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 27/31 2024-02-05T04:50:10.675 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librados2-2:19.0.0-658.gd4ed20cf.el9.x86_64 28/31 2024-02-05T04:50:10.675 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 29/31 2024-02-05T04:50:10.675 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librbd1-2:19.0.0-658.gd4ed20cf.el9.x86_64 30/31 2024-02-05T04:50:10.702 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-02-05T04:50:10.703 INFO:teuthology.orchestra.run.smithi196.stdout:=========================================================================================== 2024-02-05T04:50:10.703 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-02-05T04:50:10.703 INFO:teuthology.orchestra.run.smithi196.stdout:=========================================================================================== 2024-02-05T04:50:10.704 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-02-05T04:50:10.704 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-658.gd4ed20cf.el9 ceph-noarch 7.4 M 2024-02-05T04:50:10.704 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-02-05T04:50:10.704 INFO:teuthology.orchestra.run.smithi196.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-02-05T04:50:10.704 INFO:teuthology.orchestra.run.smithi196.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-02-05T04:50:10.704 INFO:teuthology.orchestra.run.smithi196.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-02-05T04:50:10.704 INFO:teuthology.orchestra.run.smithi196.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-02-05T04:50:10.704 INFO:teuthology.orchestra.run.smithi196.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-02-05T04:50:10.704 INFO:teuthology.orchestra.run.smithi196.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-02-05T04:50:10.704 INFO:teuthology.orchestra.run.smithi196.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-02-05T04:50:10.704 INFO:teuthology.orchestra.run.smithi196.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-02-05T04:50:10.704 INFO:teuthology.orchestra.run.smithi196.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-02-05T04:50:10.705 INFO:teuthology.orchestra.run.smithi196.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-02-05T04:50:10.705 INFO:teuthology.orchestra.run.smithi196.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-02-05T04:50:10.705 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:50:10.705 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-02-05T04:50:10.705 INFO:teuthology.orchestra.run.smithi196.stdout:=========================================================================================== 2024-02-05T04:50:10.705 INFO:teuthology.orchestra.run.smithi196.stdout:Install 12 Packages 2024-02-05T04:50:10.705 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:50:10.705 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 37 M 2024-02-05T04:50:10.706 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 209 M 2024-02-05T04:50:10.706 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-02-05T04:50:10.771 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: cephadm-2:19.0.0-658.gd4ed20cf.el9.noarch 1/1 2024-02-05T04:50:10.823 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : cephadm-2:19.0.0-658.gd4ed20cf.el9.noarch 1/1 2024-02-05T04:50:10.823 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:50:10.823 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2024-02-05T04:50:10.824 INFO:teuthology.orchestra.run.smithi062.stdout: cephadm-2:19.0.0-658.gd4ed20cf.el9.noarch 2024-02-05T04:50:10.824 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:50:10.824 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-02-05T04:50:10.863 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : cephadm-2:19.0.0-658.gd4ed20cf.el9.noarch 1/1 2024-02-05T04:50:10.903 INFO:teuthology.orchestra.run.smithi196.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 238 kB/s | 32 kB 00:00 2024-02-05T04:50:10.937 INFO:teuthology.orchestra.run.smithi196.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 537 kB/s | 17 kB 00:00 2024-02-05T04:50:10.979 INFO:teuthology.orchestra.run.smithi196.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 855 kB/s | 34 kB 00:00 2024-02-05T04:50:11.008 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install ceph-immutable-object-cache 2024-02-05T04:50:11.029 INFO:teuthology.orchestra.run.smithi196.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2024-02-05T04:50:11.205 INFO:teuthology.orchestra.run.smithi196.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 21 MB/s | 4.7 MB 00:00 2024-02-05T04:50:11.266 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: cephadm-2:19.0.0-658.gd4ed20cf.el9.noarch 1/1 2024-02-05T04:50:11.271 INFO:teuthology.orchestra.run.smithi196.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 20 MB/s | 4.8 MB 00:00 2024-02-05T04:50:11.296 INFO:teuthology.orchestra.run.smithi196.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-02-05T04:50:11.439 INFO:teuthology.orchestra.run.smithi196.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 5.6 MB/s | 802 kB 00:00 2024-02-05T04:50:11.583 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:01:04 ago on Mon 05 Feb 2024 04:49:07 AM UTC. 2024-02-05T04:50:11.583 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : cephadm-2:19.0.0-658.gd4ed20cf.el9.noarch 1/1 2024-02-05T04:50:11.583 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:50:11.583 INFO:teuthology.orchestra.run.smithi091.stdout:Installed: 2024-02-05T04:50:11.583 INFO:teuthology.orchestra.run.smithi091.stdout: cephadm-2:19.0.0-658.gd4ed20cf.el9.noarch 2024-02-05T04:50:11.584 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:50:11.584 INFO:teuthology.orchestra.run.smithi091.stdout:Complete! 2024-02-05T04:50:11.597 INFO:teuthology.orchestra.run.smithi196.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 48 MB/s | 16 MB 00:00 2024-02-05T04:50:11.623 INFO:teuthology.orchestra.run.smithi196.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 194 kB 00:00 2024-02-05T04:50:11.723 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 31/31 2024-02-05T04:50:11.723 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:50:11.723 INFO:teuthology.orchestra.run.smithi150.stdout:Upgraded: 2024-02-05T04:50:11.723 INFO:teuthology.orchestra.run.smithi150.stdout: librados2-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:50:11.723 INFO:teuthology.orchestra.run.smithi150.stdout: librbd1-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:50:11.723 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2024-02-05T04:50:11.724 INFO:teuthology.orchestra.run.smithi150.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-02-05T04:50:11.724 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-base-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:50:11.724 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-common-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:50:11.724 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-radosgw-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:50:11.724 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-selinux-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:50:11.724 INFO:teuthology.orchestra.run.smithi150.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-02-05T04:50:11.724 INFO:teuthology.orchestra.run.smithi150.stdout: libarrow-9.0.0-10.el9.x86_64 2024-02-05T04:50:11.724 INFO:teuthology.orchestra.run.smithi150.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-02-05T04:50:11.724 INFO:teuthology.orchestra.run.smithi150.stdout: libcephfs2-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:50:11.724 INFO:teuthology.orchestra.run.smithi150.stdout: liboath-2.6.7-2.el9.x86_64 2024-02-05T04:50:11.724 INFO:teuthology.orchestra.run.smithi150.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-02-05T04:50:11.724 INFO:teuthology.orchestra.run.smithi150.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-02-05T04:50:11.724 INFO:teuthology.orchestra.run.smithi150.stdout: libradosstriper1-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:50:11.724 INFO:teuthology.orchestra.run.smithi150.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-02-05T04:50:11.725 INFO:teuthology.orchestra.run.smithi150.stdout: librgw2-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:50:11.725 INFO:teuthology.orchestra.run.smithi150.stdout: libunwind-1.6.2-1.el9.x86_64 2024-02-05T04:50:11.725 INFO:teuthology.orchestra.run.smithi150.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-02-05T04:50:11.725 INFO:teuthology.orchestra.run.smithi150.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-02-05T04:50:11.725 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ceph-argparse-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:50:11.725 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ceph-common-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:50:11.725 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cephfs-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:50:11.725 INFO:teuthology.orchestra.run.smithi150.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-02-05T04:50:11.725 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rados-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:50:11.725 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rbd-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:50:11.725 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rgw-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:50:11.725 INFO:teuthology.orchestra.run.smithi150.stdout: re2-1:20211101-3.el9.x86_64 2024-02-05T04:50:11.725 INFO:teuthology.orchestra.run.smithi150.stdout: thrift-0.15.0-2.el9.x86_64 2024-02-05T04:50:11.725 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:50:11.725 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-02-05T04:50:11.746 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-02-05T04:50:11.747 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-02-05T04:50:11.747 INFO:teuthology.orchestra.run.smithi062.stdout: Package Arch Version Repo Size 2024-02-05T04:50:11.747 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-02-05T04:50:11.747 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2024-02-05T04:50:11.747 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 145 k 2024-02-05T04:50:11.747 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:50:11.747 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2024-02-05T04:50:11.747 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-02-05T04:50:11.747 INFO:teuthology.orchestra.run.smithi062.stdout:Install 1 Package 2024-02-05T04:50:11.747 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:50:11.747 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 145 k 2024-02-05T04:50:11.747 INFO:teuthology.orchestra.run.smithi062.stdout:Installed size: 436 k 2024-02-05T04:50:11.748 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2024-02-05T04:50:11.806 DEBUG:teuthology.orchestra.run.smithi091:> sudo yum -y install ceph-immutable-object-cache 2024-02-05T04:50:12.054 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-test 2024-02-05T04:50:12.089 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-immutable-object-cache-19.0.0-658.gd4ed20c 425 kB/s | 145 kB 00:00 2024-02-05T04:50:12.090 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:50:12.090 INFO:teuthology.orchestra.run.smithi062.stdout:Total 423 kB/s | 145 kB 00:00 2024-02-05T04:50:12.090 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2024-02-05T04:50:12.099 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2024-02-05T04:50:12.099 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2024-02-05T04:50:12.153 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2024-02-05T04:50:12.154 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2024-02-05T04:50:12.173 INFO:teuthology.orchestra.run.smithi196.stdout:(11/12): ceph-mgr-diskprediction-local-19.0.0-6 5.3 MB/s | 7.4 MB 00:01 2024-02-05T04:50:12.385 INFO:teuthology.orchestra.run.smithi091.stdout:Last metadata expiration check: 0:01:03 ago on Mon 05 Feb 2024 04:49:09 AM UTC. 2024-02-05T04:50:12.431 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2024-02-05T04:50:12.490 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-658.gd4ed20cf.e 1/1 2024-02-05T04:50:12.514 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-658.gd4ed20cf.e 1/1 2024-02-05T04:50:12.514 INFO:teuthology.orchestra.run.smithi062.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T04:50:12.514 INFO:teuthology.orchestra.run.smithi062.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-02-05T04:50:12.515 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:50:12.541 INFO:teuthology.orchestra.run.smithi091.stdout:Dependencies resolved. 2024-02-05T04:50:12.542 INFO:teuthology.orchestra.run.smithi091.stdout:================================================================================ 2024-02-05T04:50:12.542 INFO:teuthology.orchestra.run.smithi091.stdout: Package Arch Version Repo Size 2024-02-05T04:50:12.542 INFO:teuthology.orchestra.run.smithi091.stdout:================================================================================ 2024-02-05T04:50:12.542 INFO:teuthology.orchestra.run.smithi091.stdout:Installing: 2024-02-05T04:50:12.542 INFO:teuthology.orchestra.run.smithi091.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 145 k 2024-02-05T04:50:12.542 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:50:12.542 INFO:teuthology.orchestra.run.smithi091.stdout:Transaction Summary 2024-02-05T04:50:12.543 INFO:teuthology.orchestra.run.smithi091.stdout:================================================================================ 2024-02-05T04:50:12.543 INFO:teuthology.orchestra.run.smithi091.stdout:Install 1 Package 2024-02-05T04:50:12.543 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:50:12.543 INFO:teuthology.orchestra.run.smithi091.stdout:Total download size: 145 k 2024-02-05T04:50:12.543 INFO:teuthology.orchestra.run.smithi091.stdout:Installed size: 436 k 2024-02-05T04:50:12.543 INFO:teuthology.orchestra.run.smithi091.stdout:Downloading Packages: 2024-02-05T04:50:12.619 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:00:33 ago on Mon 05 Feb 2024 04:49:39 AM UTC. 2024-02-05T04:50:12.779 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-02-05T04:50:12.780 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-02-05T04:50:12.780 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2024-02-05T04:50:12.780 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-02-05T04:50:12.780 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2024-02-05T04:50:12.780 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-test x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 45 M 2024-02-05T04:50:12.780 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2024-02-05T04:50:12.780 INFO:teuthology.orchestra.run.smithi150.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-02-05T04:50:12.780 INFO:teuthology.orchestra.run.smithi150.stdout: libcephsqlite x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 163 k 2024-02-05T04:50:12.780 INFO:teuthology.orchestra.run.smithi150.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-02-05T04:50:12.781 INFO:teuthology.orchestra.run.smithi150.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-02-05T04:50:12.781 INFO:teuthology.orchestra.run.smithi150.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-02-05T04:50:12.781 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:50:12.781 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-02-05T04:50:12.781 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-02-05T04:50:12.781 INFO:teuthology.orchestra.run.smithi150.stdout:Install 6 Packages 2024-02-05T04:50:12.781 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:50:12.781 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 46 M 2024-02-05T04:50:12.782 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 196 M 2024-02-05T04:50:12.782 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-02-05T04:50:12.906 INFO:teuthology.orchestra.run.smithi091.stdout:ceph-immutable-object-cache-19.0.0-658.gd4ed20c 399 kB/s | 145 kB 00:00 2024-02-05T04:50:12.907 INFO:teuthology.orchestra.run.smithi091.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:50:12.907 INFO:teuthology.orchestra.run.smithi091.stdout:Total 398 kB/s | 145 kB 00:00 2024-02-05T04:50:12.907 INFO:teuthology.orchestra.run.smithi091.stdout:Running transaction check 2024-02-05T04:50:12.915 INFO:teuthology.orchestra.run.smithi091.stdout:Transaction check succeeded. 2024-02-05T04:50:12.915 INFO:teuthology.orchestra.run.smithi091.stdout:Running transaction test 2024-02-05T04:50:12.939 INFO:teuthology.orchestra.run.smithi196.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 183 kB/s | 245 kB 00:01 2024-02-05T04:50:12.942 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:50:12.943 INFO:teuthology.orchestra.run.smithi196.stdout:Total 17 MB/s | 37 MB 00:02 2024-02-05T04:50:12.954 INFO:teuthology.orchestra.run.smithi150.stdout:(1/6): socat-1.7.4.1-5.el9.x86_64.rpm 1.7 MB/s | 300 kB 00:00 2024-02-05T04:50:12.957 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-02-05T04:50:12.966 INFO:teuthology.orchestra.run.smithi091.stdout:Transaction test succeeded. 2024-02-05T04:50:12.966 INFO:teuthology.orchestra.run.smithi091.stdout:Running transaction 2024-02-05T04:50:12.984 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-02-05T04:50:12.984 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-02-05T04:50:12.988 INFO:teuthology.orchestra.run.smithi150.stdout:(2/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.9 MB/s | 63 kB 00:00 2024-02-05T04:50:13.030 INFO:teuthology.orchestra.run.smithi150.stdout:(3/6): jq-1.6-16.el9.x86_64.rpm 4.3 MB/s | 186 kB 00:00 2024-02-05T04:50:13.072 INFO:teuthology.orchestra.run.smithi150.stdout:(4/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 5.1 MB/s | 217 kB 00:00 2024-02-05T04:50:13.112 INFO:teuthology.orchestra.run.smithi091.stdout: Preparing : 1/1 2024-02-05T04:50:13.174 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-658.gd4ed20cf.e 1/1 2024-02-05T04:50:13.198 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-658.gd4ed20cf.e 1/1 2024-02-05T04:50:13.198 INFO:teuthology.orchestra.run.smithi091.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T04:50:13.199 INFO:teuthology.orchestra.run.smithi091.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-02-05T04:50:13.199 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:50:13.205 INFO:teuthology.orchestra.run.smithi150.stdout:(5/6): libcephsqlite-19.0.0-658.gd4ed20cf.el9.x 386 kB/s | 163 kB 00:00 2024-02-05T04:50:13.365 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-02-05T04:50:13.365 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-02-05T04:50:13.565 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-658.gd4ed20cf.e 1/1 2024-02-05T04:50:13.565 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:50:13.565 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2024-02-05T04:50:13.565 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-immutable-object-cache-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:50:13.565 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:50:13.565 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-02-05T04:50:13.849 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install ceph-mgr 2024-02-05T04:50:14.067 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-02-05T04:50:14.148 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-02-05T04:50:14.177 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-02-05T04:50:14.180 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-658.gd4ed20cf.e 1/1 2024-02-05T04:50:14.180 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:50:14.180 INFO:teuthology.orchestra.run.smithi091.stdout:Installed: 2024-02-05T04:50:14.180 INFO:teuthology.orchestra.run.smithi091.stdout: ceph-immutable-object-cache-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:50:14.180 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:50:14.180 INFO:teuthology.orchestra.run.smithi091.stdout:Complete! 2024-02-05T04:50:14.272 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-02-05T04:50:14.314 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-02-05T04:50:14.433 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:01:07 ago on Mon 05 Feb 2024 04:49:07 AM UTC. 2024-02-05T04:50:14.451 DEBUG:teuthology.orchestra.run.smithi091:> sudo yum -y install ceph-mgr 2024-02-05T04:50:14.499 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-02-05T04:50:14.529 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-02-05T04:50:14.542 INFO:teuthology.orchestra.run.smithi062.stdout:Package ceph-mgr-2:19.0.0-658.gd4ed20cf.el9.x86_64 is already installed. 2024-02-05T04:50:14.597 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-02-05T04:50:14.598 INFO:teuthology.orchestra.run.smithi062.stdout:Nothing to do. 2024-02-05T04:50:14.598 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-02-05T04:50:14.606 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-02-05T04:50:14.658 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install ceph-mgr-dashboard 2024-02-05T04:50:15.031 INFO:teuthology.orchestra.run.smithi091.stdout:Last metadata expiration check: 0:01:06 ago on Mon 05 Feb 2024 04:49:09 AM UTC. 2024-02-05T04:50:15.135 INFO:teuthology.orchestra.run.smithi091.stdout:Package ceph-mgr-2:19.0.0-658.gd4ed20cf.el9.x86_64 is already installed. 2024-02-05T04:50:15.186 INFO:teuthology.orchestra.run.smithi091.stdout:Dependencies resolved. 2024-02-05T04:50:15.187 INFO:teuthology.orchestra.run.smithi091.stdout:Nothing to do. 2024-02-05T04:50:15.187 INFO:teuthology.orchestra.run.smithi091.stdout:Complete! 2024-02-05T04:50:15.231 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:01:08 ago on Mon 05 Feb 2024 04:49:07 AM UTC. 2024-02-05T04:50:15.252 DEBUG:teuthology.orchestra.run.smithi091:> sudo yum -y install ceph-mgr-dashboard 2024-02-05T04:50:15.283 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-02-05T04:50:15.393 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-02-05T04:50:15.395 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-02-05T04:50:15.395 INFO:teuthology.orchestra.run.smithi062.stdout: Package Arch Version Repository Size 2024-02-05T04:50:15.395 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-02-05T04:50:15.395 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2024-02-05T04:50:15.395 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-dashboard noarch 2:19.0.0-658.gd4ed20cf.el9 ceph-noarch 1.7 M 2024-02-05T04:50:15.395 INFO:teuthology.orchestra.run.smithi062.stdout:Installing dependencies: 2024-02-05T04:50:15.395 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-grafana-dashboards noarch 2:19.0.0-658.gd4ed20cf.el9 ceph-noarch 28 k 2024-02-05T04:50:15.395 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-prometheus-alerts noarch 2:19.0.0-658.gd4ed20cf.el9 ceph-noarch 15 k 2024-02-05T04:50:15.395 INFO:teuthology.orchestra.run.smithi062.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-02-05T04:50:15.395 INFO:teuthology.orchestra.run.smithi062.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-02-05T04:50:15.395 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:50:15.395 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2024-02-05T04:50:15.395 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-02-05T04:50:15.396 INFO:teuthology.orchestra.run.smithi062.stdout:Install 5 Packages 2024-02-05T04:50:15.396 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:50:15.396 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 2.0 M 2024-02-05T04:50:15.396 INFO:teuthology.orchestra.run.smithi062.stdout:Installed size: 8.4 M 2024-02-05T04:50:15.396 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2024-02-05T04:50:15.408 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-02-05T04:50:15.681 INFO:teuthology.orchestra.run.smithi150.stdout:(6/6): ceph-test-19.0.0-658.gd4ed20cf.el9.x86_6 15 MB/s | 45 MB 00:02 2024-02-05T04:50:15.684 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:50:15.684 INFO:teuthology.orchestra.run.smithi150.stdout:Total 16 MB/s | 46 MB 00:02 2024-02-05T04:50:15.684 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-02-05T04:50:15.704 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-02-05T04:50:15.705 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-02-05T04:50:15.752 INFO:teuthology.orchestra.run.smithi062.stdout:(1/5): ceph-grafana-dashboards-19.0.0-658.gd4ed 145 kB/s | 28 kB 00:00 2024-02-05T04:50:15.818 INFO:teuthology.orchestra.run.smithi091.stdout:Last metadata expiration check: 0:01:06 ago on Mon 05 Feb 2024 04:49:09 AM UTC. 2024-02-05T04:50:15.861 INFO:teuthology.orchestra.run.smithi062.stdout:(2/5): ceph-prometheus-alerts-19.0.0-658.gd4ed2 50 kB/s | 15 kB 00:00 2024-02-05T04:50:15.878 INFO:teuthology.orchestra.run.smithi062.stdout:(3/5): python3-repoze-lru-0.7-16.el9.noarch.rpm 246 kB/s | 31 kB 00:00 2024-02-05T04:50:15.974 INFO:teuthology.orchestra.run.smithi091.stdout:Dependencies resolved. 2024-02-05T04:50:15.974 INFO:teuthology.orchestra.run.smithi091.stdout:================================================================================ 2024-02-05T04:50:15.975 INFO:teuthology.orchestra.run.smithi091.stdout: Package Arch Version Repository Size 2024-02-05T04:50:15.975 INFO:teuthology.orchestra.run.smithi091.stdout:================================================================================ 2024-02-05T04:50:15.975 INFO:teuthology.orchestra.run.smithi091.stdout:Installing: 2024-02-05T04:50:15.975 INFO:teuthology.orchestra.run.smithi091.stdout: ceph-mgr-dashboard noarch 2:19.0.0-658.gd4ed20cf.el9 ceph-noarch 1.7 M 2024-02-05T04:50:15.975 INFO:teuthology.orchestra.run.smithi091.stdout:Installing dependencies: 2024-02-05T04:50:15.975 INFO:teuthology.orchestra.run.smithi091.stdout: ceph-grafana-dashboards noarch 2:19.0.0-658.gd4ed20cf.el9 ceph-noarch 28 k 2024-02-05T04:50:15.975 INFO:teuthology.orchestra.run.smithi091.stdout: ceph-prometheus-alerts noarch 2:19.0.0-658.gd4ed20cf.el9 ceph-noarch 15 k 2024-02-05T04:50:15.975 INFO:teuthology.orchestra.run.smithi091.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-02-05T04:50:15.975 INFO:teuthology.orchestra.run.smithi091.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-02-05T04:50:15.975 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:50:15.975 INFO:teuthology.orchestra.run.smithi091.stdout:Transaction Summary 2024-02-05T04:50:15.975 INFO:teuthology.orchestra.run.smithi091.stdout:================================================================================ 2024-02-05T04:50:15.975 INFO:teuthology.orchestra.run.smithi091.stdout:Install 5 Packages 2024-02-05T04:50:15.975 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:50:15.976 INFO:teuthology.orchestra.run.smithi091.stdout:Total download size: 2.0 M 2024-02-05T04:50:15.976 INFO:teuthology.orchestra.run.smithi091.stdout:Installed size: 8.4 M 2024-02-05T04:50:15.976 INFO:teuthology.orchestra.run.smithi091.stdout:Downloading Packages: 2024-02-05T04:50:16.020 INFO:teuthology.orchestra.run.smithi062.stdout:(4/5): python3-routes-2.5.1-5.el9.noarch.rpm 1.2 MB/s | 188 kB 00:00 2024-02-05T04:50:16.071 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-02-05T04:50:16.071 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-02-05T04:50:16.227 INFO:teuthology.orchestra.run.smithi091.stdout:(1/5): ceph-grafana-dashboards-19.0.0-658.gd4ed 135 kB/s | 28 kB 00:00 2024-02-05T04:50:16.352 INFO:teuthology.orchestra.run.smithi091.stdout:(2/5): ceph-prometheus-alerts-19.0.0-658.gd4ed2 45 kB/s | 15 kB 00:00 2024-02-05T04:50:16.553 INFO:teuthology.orchestra.run.smithi062.stdout:(5/5): ceph-mgr-dashboard-19.0.0-658.gd4ed20cf. 1.7 MB/s | 1.7 MB 00:00 2024-02-05T04:50:16.557 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:50:16.557 INFO:teuthology.orchestra.run.smithi062.stdout:Total 1.7 MB/s | 2.0 MB 00:01 2024-02-05T04:50:16.557 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2024-02-05T04:50:16.565 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2024-02-05T04:50:16.565 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2024-02-05T04:50:16.611 INFO:teuthology.orchestra.run.smithi091.stdout:(3/5): python3-repoze-lru-0.7-16.el9.noarch.rpm 80 kB/s | 31 kB 00:00 2024-02-05T04:50:16.615 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-02-05T04:50:16.624 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-02-05T04:50:16.626 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2024-02-05T04:50:16.627 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2024-02-05T04:50:16.690 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 1/6 2024-02-05T04:50:16.722 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : jq-1.6-16.el9.x86_64 2/6 2024-02-05T04:50:16.765 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/6 2024-02-05T04:50:16.769 INFO:teuthology.orchestra.run.smithi091.stdout:(4/5): ceph-mgr-dashboard-19.0.0-658.gd4ed20cf. 2.3 MB/s | 1.7 MB 00:00 2024-02-05T04:50:16.805 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/6 2024-02-05T04:50:16.825 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libcephsqlite-2:19.0.0-658.gd4ed20cf.el9.x86_64 5/6 2024-02-05T04:50:16.875 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2024-02-05T04:50:16.911 INFO:teuthology.orchestra.run.smithi091.stdout:(5/5): python3-routes-2.5.1-5.el9.noarch.rpm 337 kB/s | 188 kB 00:00 2024-02-05T04:50:16.914 INFO:teuthology.orchestra.run.smithi091.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:50:16.915 INFO:teuthology.orchestra.run.smithi091.stdout:Total 2.1 MB/s | 2.0 MB 00:00 2024-02-05T04:50:16.915 INFO:teuthology.orchestra.run.smithi091.stdout:Running transaction check 2024-02-05T04:50:16.922 INFO:teuthology.orchestra.run.smithi091.stdout:Transaction check succeeded. 2024-02-05T04:50:16.922 INFO:teuthology.orchestra.run.smithi091.stdout:Running transaction test 2024-02-05T04:50:16.969 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/5 2024-02-05T04:50:16.990 INFO:teuthology.orchestra.run.smithi091.stdout:Transaction test succeeded. 2024-02-05T04:50:16.990 INFO:teuthology.orchestra.run.smithi091.stdout:Running transaction 2024-02-05T04:50:16.999 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/5 2024-02-05T04:50:17.030 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-658.gd4ed20cf.el9.no 3/5 2024-02-05T04:50:17.237 INFO:teuthology.orchestra.run.smithi091.stdout: Preparing : 1/1 2024-02-05T04:50:17.277 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-658.gd4ed20cf.el9.n 4/5 2024-02-05T04:50:17.315 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-658.gd4ed20cf.el9.noarch 5/5 2024-02-05T04:50:17.333 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/5 2024-02-05T04:50:17.382 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/5 2024-02-05T04:50:17.412 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-658.gd4ed20cf.el9.no 3/5 2024-02-05T04:50:17.639 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-658.gd4ed20cf.el9.n 4/5 2024-02-05T04:50:17.663 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-02-05T04:50:17.672 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-658.gd4ed20cf.el9.noarch 5/5 2024-02-05T04:50:17.688 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-658.gd4ed20 12/12 2024-02-05T04:50:17.855 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-658.gd4ed20cf.el9.noarch 5/5 2024-02-05T04:50:17.855 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-658.gd4ed20cf.el9.n 1/5 2024-02-05T04:50:17.855 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-658.gd4ed20cf.el9.noarch 2/5 2024-02-05T04:50:17.855 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-658.gd4ed20cf.el9.no 3/5 2024-02-05T04:50:17.855 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 4/5 2024-02-05T04:50:18.171 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 5/5 2024-02-05T04:50:18.171 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:50:18.171 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2024-02-05T04:50:18.171 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-grafana-dashboards-2:19.0.0-658.gd4ed20cf.el9.noarch 2024-02-05T04:50:18.171 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-dashboard-2:19.0.0-658.gd4ed20cf.el9.noarch 2024-02-05T04:50:18.171 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-prometheus-alerts-2:19.0.0-658.gd4ed20cf.el9.noarch 2024-02-05T04:50:18.171 INFO:teuthology.orchestra.run.smithi062.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-02-05T04:50:18.172 INFO:teuthology.orchestra.run.smithi062.stdout: python3-routes-2.5.1-5.el9.noarch 2024-02-05T04:50:18.172 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:50:18.172 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-02-05T04:50:18.243 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-658.gd4ed20cf.el9.noarch 5/5 2024-02-05T04:50:18.244 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-658.gd4ed20cf.el9.n 1/5 2024-02-05T04:50:18.244 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-658.gd4ed20cf.el9.noarch 2/5 2024-02-05T04:50:18.244 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-658.gd4ed20cf.el9.no 3/5 2024-02-05T04:50:18.244 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 4/5 2024-02-05T04:50:18.397 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-05T04:50:18.568 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 5/5 2024-02-05T04:50:18.568 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:50:18.568 INFO:teuthology.orchestra.run.smithi091.stdout:Installed: 2024-02-05T04:50:18.568 INFO:teuthology.orchestra.run.smithi091.stdout: ceph-grafana-dashboards-2:19.0.0-658.gd4ed20cf.el9.noarch 2024-02-05T04:50:18.569 INFO:teuthology.orchestra.run.smithi091.stdout: ceph-mgr-dashboard-2:19.0.0-658.gd4ed20cf.el9.noarch 2024-02-05T04:50:18.569 INFO:teuthology.orchestra.run.smithi091.stdout: ceph-prometheus-alerts-2:19.0.0-658.gd4ed20cf.el9.noarch 2024-02-05T04:50:18.569 INFO:teuthology.orchestra.run.smithi091.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-02-05T04:50:18.569 INFO:teuthology.orchestra.run.smithi091.stdout: python3-routes-2.5.1-5.el9.noarch 2024-02-05T04:50:18.569 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:50:18.569 INFO:teuthology.orchestra.run.smithi091.stdout:Complete! 2024-02-05T04:50:18.792 DEBUG:teuthology.orchestra.run.smithi091:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-05T04:50:18.967 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:01:11 ago on Mon 05 Feb 2024 04:49:07 AM UTC. 2024-02-05T04:50:19.131 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-02-05T04:50:19.132 INFO:teuthology.orchestra.run.smithi062.stdout:=========================================================================================== 2024-02-05T04:50:19.132 INFO:teuthology.orchestra.run.smithi062.stdout: Package Arch Version Repository Size 2024-02-05T04:50:19.132 INFO:teuthology.orchestra.run.smithi062.stdout:=========================================================================================== 2024-02-05T04:50:19.133 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2024-02-05T04:50:19.133 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-658.gd4ed20cf.el9 ceph-noarch 7.4 M 2024-02-05T04:50:19.133 INFO:teuthology.orchestra.run.smithi062.stdout:Installing dependencies: 2024-02-05T04:50:19.133 INFO:teuthology.orchestra.run.smithi062.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-02-05T04:50:19.133 INFO:teuthology.orchestra.run.smithi062.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-02-05T04:50:19.133 INFO:teuthology.orchestra.run.smithi062.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-02-05T04:50:19.133 INFO:teuthology.orchestra.run.smithi062.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-02-05T04:50:19.133 INFO:teuthology.orchestra.run.smithi062.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-02-05T04:50:19.133 INFO:teuthology.orchestra.run.smithi062.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-02-05T04:50:19.133 INFO:teuthology.orchestra.run.smithi062.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-02-05T04:50:19.133 INFO:teuthology.orchestra.run.smithi062.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-02-05T04:50:19.133 INFO:teuthology.orchestra.run.smithi062.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-02-05T04:50:19.133 INFO:teuthology.orchestra.run.smithi062.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-02-05T04:50:19.134 INFO:teuthology.orchestra.run.smithi062.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-02-05T04:50:19.134 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:50:19.134 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2024-02-05T04:50:19.134 INFO:teuthology.orchestra.run.smithi062.stdout:=========================================================================================== 2024-02-05T04:50:19.134 INFO:teuthology.orchestra.run.smithi062.stdout:Install 12 Packages 2024-02-05T04:50:19.134 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:50:19.134 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 37 M 2024-02-05T04:50:19.135 INFO:teuthology.orchestra.run.smithi062.stdout:Installed size: 209 M 2024-02-05T04:50:19.135 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2024-02-05T04:50:19.356 INFO:teuthology.orchestra.run.smithi091.stdout:Last metadata expiration check: 0:01:10 ago on Mon 05 Feb 2024 04:49:09 AM UTC. 2024-02-05T04:50:19.393 INFO:teuthology.orchestra.run.smithi062.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 228 kB/s | 32 kB 00:00 2024-02-05T04:50:19.427 INFO:teuthology.orchestra.run.smithi062.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 524 kB/s | 17 kB 00:00 2024-02-05T04:50:19.510 INFO:teuthology.orchestra.run.smithi062.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2024-02-05T04:50:19.517 INFO:teuthology.orchestra.run.smithi091.stdout:Dependencies resolved. 2024-02-05T04:50:19.518 INFO:teuthology.orchestra.run.smithi091.stdout:=========================================================================================== 2024-02-05T04:50:19.518 INFO:teuthology.orchestra.run.smithi091.stdout: Package Arch Version Repository Size 2024-02-05T04:50:19.518 INFO:teuthology.orchestra.run.smithi091.stdout:=========================================================================================== 2024-02-05T04:50:19.518 INFO:teuthology.orchestra.run.smithi091.stdout:Installing: 2024-02-05T04:50:19.518 INFO:teuthology.orchestra.run.smithi091.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-658.gd4ed20cf.el9 ceph-noarch 7.4 M 2024-02-05T04:50:19.519 INFO:teuthology.orchestra.run.smithi091.stdout:Installing dependencies: 2024-02-05T04:50:19.519 INFO:teuthology.orchestra.run.smithi091.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-02-05T04:50:19.519 INFO:teuthology.orchestra.run.smithi091.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-02-05T04:50:19.519 INFO:teuthology.orchestra.run.smithi091.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-02-05T04:50:19.519 INFO:teuthology.orchestra.run.smithi091.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-02-05T04:50:19.519 INFO:teuthology.orchestra.run.smithi091.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-02-05T04:50:19.520 INFO:teuthology.orchestra.run.smithi091.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-02-05T04:50:19.520 INFO:teuthology.orchestra.run.smithi091.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-02-05T04:50:19.520 INFO:teuthology.orchestra.run.smithi091.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-02-05T04:50:19.521 INFO:teuthology.orchestra.run.smithi091.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-02-05T04:50:19.521 INFO:teuthology.orchestra.run.smithi091.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-02-05T04:50:19.521 INFO:teuthology.orchestra.run.smithi091.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-02-05T04:50:19.521 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:50:19.521 INFO:teuthology.orchestra.run.smithi091.stdout:Transaction Summary 2024-02-05T04:50:19.522 INFO:teuthology.orchestra.run.smithi091.stdout:=========================================================================================== 2024-02-05T04:50:19.522 INFO:teuthology.orchestra.run.smithi091.stdout:Install 12 Packages 2024-02-05T04:50:19.522 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:50:19.523 INFO:teuthology.orchestra.run.smithi091.stdout:Total download size: 37 M 2024-02-05T04:50:19.523 INFO:teuthology.orchestra.run.smithi091.stdout:Installed size: 209 M 2024-02-05T04:50:19.523 INFO:teuthology.orchestra.run.smithi091.stdout:Downloading Packages: 2024-02-05T04:50:19.536 INFO:teuthology.orchestra.run.smithi062.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 324 kB/s | 34 kB 00:00 2024-02-05T04:50:19.734 INFO:teuthology.orchestra.run.smithi091.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 185 kB/s | 32 kB 00:00 2024-02-05T04:50:19.752 INFO:teuthology.orchestra.run.smithi062.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 20 MB/s | 4.7 MB 00:00 2024-02-05T04:50:19.776 INFO:teuthology.orchestra.run.smithi091.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 412 kB/s | 17 kB 00:00 2024-02-05T04:50:19.820 INFO:teuthology.orchestra.run.smithi062.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 17 MB/s | 4.8 MB 00:00 2024-02-05T04:50:19.845 INFO:teuthology.orchestra.run.smithi062.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-02-05T04:50:19.851 INFO:teuthology.orchestra.run.smithi091.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 10 MB/s | 3.0 MB 00:00 2024-02-05T04:50:19.868 INFO:teuthology.orchestra.run.smithi091.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 380 kB/s | 34 kB 00:00 2024-02-05T04:50:20.162 INFO:teuthology.orchestra.run.smithi062.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.5 MB/s | 802 kB 00:00 2024-02-05T04:50:20.224 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-658.gd4ed20 12/12 2024-02-05T04:50:20.224 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-658.gd4ed20 1/12 2024-02-05T04:50:20.225 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-02-05T04:50:20.225 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-02-05T04:50:20.225 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-02-05T04:50:20.225 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-02-05T04:50:20.225 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-02-05T04:50:20.225 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-02-05T04:50:20.225 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-02-05T04:50:20.225 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-02-05T04:50:20.225 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-02-05T04:50:20.225 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-02-05T04:50:20.321 INFO:teuthology.orchestra.run.smithi062.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 31 MB/s | 16 MB 00:00 2024-02-05T04:50:20.346 INFO:teuthology.orchestra.run.smithi062.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 194 kB 00:00 2024-02-05T04:50:20.521 INFO:teuthology.orchestra.run.smithi062.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.2 MB/s | 245 kB 00:00 2024-02-05T04:50:20.627 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-02-05T04:50:20.627 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:50:20.627 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-02-05T04:50:20.627 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-diskprediction-local-2:19.0.0-658.gd4ed20cf.el9.noarch 2024-02-05T04:50:20.627 INFO:teuthology.orchestra.run.smithi196.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-02-05T04:50:20.627 INFO:teuthology.orchestra.run.smithi196.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-02-05T04:50:20.627 INFO:teuthology.orchestra.run.smithi196.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-02-05T04:50:20.627 INFO:teuthology.orchestra.run.smithi196.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-02-05T04:50:20.628 INFO:teuthology.orchestra.run.smithi196.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-02-05T04:50:20.628 INFO:teuthology.orchestra.run.smithi196.stdout: openblas-0.3.21-2.el9.x86_64 2024-02-05T04:50:20.628 INFO:teuthology.orchestra.run.smithi196.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-02-05T04:50:20.628 INFO:teuthology.orchestra.run.smithi196.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-02-05T04:50:20.628 INFO:teuthology.orchestra.run.smithi196.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-02-05T04:50:20.628 INFO:teuthology.orchestra.run.smithi196.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-02-05T04:50:20.628 INFO:teuthology.orchestra.run.smithi196.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-02-05T04:50:20.628 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:50:20.628 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-02-05T04:50:20.993 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-mgr-rook 2024-02-05T04:50:21.068 INFO:teuthology.orchestra.run.smithi091.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 3.9 MB/s | 4.7 MB 00:01 2024-02-05T04:50:21.099 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: libcephsqlite-2:19.0.0-658.gd4ed20cf.el9.x86_64 5/6 2024-02-05T04:50:21.168 INFO:teuthology.orchestra.run.smithi091.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 3.7 MB/s | 4.8 MB 00:01 2024-02-05T04:50:21.182 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-test-2:19.0.0-658.gd4ed20cf.el9.x86_64 6/6 2024-02-05T04:50:21.194 INFO:teuthology.orchestra.run.smithi091.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.7 MB/s | 217 kB 00:00 2024-02-05T04:50:21.411 INFO:teuthology.orchestra.run.smithi091.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 3.6 MB/s | 802 kB 00:00 2024-02-05T04:50:21.561 INFO:teuthology.orchestra.run.smithi091.stdout:(9/12): ceph-mgr-diskprediction-local-19.0.0-65 3.7 MB/s | 7.4 MB 00:01 2024-02-05T04:50:21.562 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:22 ago on Mon 05 Feb 2024 04:48:59 AM UTC. 2024-02-05T04:50:21.721 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-02-05T04:50:21.722 INFO:teuthology.orchestra.run.smithi196.stdout:======================================================================================= 2024-02-05T04:50:21.722 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-02-05T04:50:21.722 INFO:teuthology.orchestra.run.smithi196.stdout:======================================================================================= 2024-02-05T04:50:21.722 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-02-05T04:50:21.722 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-rook noarch 2:19.0.0-658.gd4ed20cf.el9 ceph-noarch 50 k 2024-02-05T04:50:21.722 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-02-05T04:50:21.723 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-02-05T04:50:21.723 INFO:teuthology.orchestra.run.smithi196.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-02-05T04:50:21.723 INFO:teuthology.orchestra.run.smithi196.stdout: python3-google-auth noarch 1:2.27.0-1.el9 epel 203 k 2024-02-05T04:50:21.723 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-02-05T04:50:21.723 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-02-05T04:50:21.723 INFO:teuthology.orchestra.run.smithi196.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-02-05T04:50:21.723 INFO:teuthology.orchestra.run.smithi196.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-02-05T04:50:21.723 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-02-05T04:50:21.723 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-02-05T04:50:21.723 INFO:teuthology.orchestra.run.smithi196.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-02-05T04:50:21.723 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-02-05T04:50:21.723 INFO:teuthology.orchestra.run.smithi196.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-02-05T04:50:21.724 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:50:21.724 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-02-05T04:50:21.724 INFO:teuthology.orchestra.run.smithi196.stdout:======================================================================================= 2024-02-05T04:50:21.724 INFO:teuthology.orchestra.run.smithi196.stdout:Install 13 Packages 2024-02-05T04:50:21.724 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:50:21.724 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 2.0 M 2024-02-05T04:50:21.725 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 26 M 2024-02-05T04:50:21.725 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-02-05T04:50:21.728 INFO:teuthology.orchestra.run.smithi091.stdout:(10/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 28 MB/s | 16 MB 00:00 2024-02-05T04:50:21.778 INFO:teuthology.orchestra.run.smithi091.stdout:(11/12): libquadmath-11.4.1-3.el9.x86_64.rpm 532 kB/s | 194 kB 00:00 2024-02-05T04:50:21.906 INFO:teuthology.orchestra.run.smithi196.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 187 kB/s | 24 kB 00:00 2024-02-05T04:50:21.930 INFO:teuthology.orchestra.run.smithi196.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 116 kB/s | 18 kB 00:00 2024-02-05T04:50:21.989 INFO:teuthology.orchestra.run.smithi196.stdout:(3/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.3 MB/s | 132 kB 00:00 2024-02-05T04:50:22.014 INFO:teuthology.orchestra.run.smithi196.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.1 MB/s | 174 kB 00:00 2024-02-05T04:50:22.039 INFO:teuthology.orchestra.run.smithi196.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-02-05T04:50:22.065 INFO:teuthology.orchestra.run.smithi196.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 858 kB/s | 43 kB 00:00 2024-02-05T04:50:22.148 INFO:teuthology.orchestra.run.smithi196.stdout:(7/13): ceph-mgr-rook-19.0.0-658.gd4ed20cf.el9. 135 kB/s | 50 kB 00:00 2024-02-05T04:50:22.171 INFO:teuthology.orchestra.run.smithi062.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-6 2.5 MB/s | 7.4 MB 00:02 2024-02-05T04:50:22.174 INFO:teuthology.orchestra.run.smithi196.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 240 kB/s | 32 kB 00:00 2024-02-05T04:50:22.174 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:50:22.175 INFO:teuthology.orchestra.run.smithi062.stdout:Total 12 MB/s | 37 MB 00:03 2024-02-05T04:50:22.189 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2024-02-05T04:50:22.199 INFO:teuthology.orchestra.run.smithi196.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 112 kB/s | 15 kB 00:00 2024-02-05T04:50:22.216 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2024-02-05T04:50:22.216 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2024-02-05T04:50:22.232 INFO:teuthology.orchestra.run.smithi196.stdout:(10/13): python3-google-auth-2.27.0-1.el9.noarc 2.4 MB/s | 203 kB 00:00 2024-02-05T04:50:22.263 INFO:teuthology.orchestra.run.smithi091.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 351 kB/s | 245 kB 00:00 2024-02-05T04:50:22.265 INFO:teuthology.orchestra.run.smithi091.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:50:22.266 INFO:teuthology.orchestra.run.smithi091.stdout:Total 14 MB/s | 37 MB 00:02 2024-02-05T04:50:22.266 INFO:teuthology.orchestra.run.smithi196.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 888 kB/s | 59 kB 00:00 2024-02-05T04:50:22.280 INFO:teuthology.orchestra.run.smithi091.stdout:Running transaction check 2024-02-05T04:50:22.300 INFO:teuthology.orchestra.run.smithi196.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 8.1 MB/s | 1.0 MB 00:00 2024-02-05T04:50:22.306 INFO:teuthology.orchestra.run.smithi091.stdout:Transaction check succeeded. 2024-02-05T04:50:22.306 INFO:teuthology.orchestra.run.smithi091.stdout:Running transaction test 2024-02-05T04:50:22.325 INFO:teuthology.orchestra.run.smithi196.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 979 kB/s | 90 kB 00:00 2024-02-05T04:50:22.328 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:50:22.328 INFO:teuthology.orchestra.run.smithi196.stdout:Total 3.4 MB/s | 2.0 MB 00:00 2024-02-05T04:50:22.329 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-02-05T04:50:22.342 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-02-05T04:50:22.342 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-02-05T04:50:22.439 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-02-05T04:50:22.439 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-02-05T04:50:22.627 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2024-02-05T04:50:22.627 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2024-02-05T04:50:22.690 INFO:teuthology.orchestra.run.smithi091.stdout:Transaction test succeeded. 2024-02-05T04:50:22.691 INFO:teuthology.orchestra.run.smithi091.stdout:Running transaction 2024-02-05T04:50:22.955 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-02-05T04:50:23.122 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-02-05T04:50:23.184 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-02-05T04:50:23.240 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-02-05T04:50:23.288 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-02-05T04:50:23.342 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2024-02-05T04:50:23.343 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-02-05T04:50:23.387 INFO:teuthology.orchestra.run.smithi091.stdout: Preparing : 1/1 2024-02-05T04:50:23.410 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-test-2:19.0.0-658.gd4ed20cf.el9.x86_64 6/6 2024-02-05T04:50:23.410 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-test-2:19.0.0-658.gd4ed20cf.el9.x86_64 1/6 2024-02-05T04:50:23.410 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libcephsqlite-2:19.0.0-658.gd4ed20cf.el9.x86_64 2/6 2024-02-05T04:50:23.410 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/6 2024-02-05T04:50:23.410 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/6 2024-02-05T04:50:23.410 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : jq-1.6-16.el9.x86_64 5/6 2024-02-05T04:50:23.437 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-02-05T04:50:23.439 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-02-05T04:50:23.467 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-02-05T04:50:23.469 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-02-05T04:50:23.499 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-02-05T04:50:23.547 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-google-auth-1:2.27.0-1.el9.noarch 7/13 2024-02-05T04:50:23.564 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-02-05T04:50:23.584 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-02-05T04:50:23.594 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-02-05T04:50:23.604 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-02-05T04:50:23.619 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-02-05T04:50:23.785 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 6/6 2024-02-05T04:50:23.785 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:50:23.785 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2024-02-05T04:50:23.786 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-test-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:50:23.786 INFO:teuthology.orchestra.run.smithi150.stdout: jq-1.6-16.el9.x86_64 2024-02-05T04:50:23.786 INFO:teuthology.orchestra.run.smithi150.stdout: libcephsqlite-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:50:23.786 INFO:teuthology.orchestra.run.smithi150.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-02-05T04:50:23.786 INFO:teuthology.orchestra.run.smithi150.stdout: socat-1.7.4.1-5.el9.x86_64 2024-02-05T04:50:23.786 INFO:teuthology.orchestra.run.smithi150.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-02-05T04:50:23.786 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:50:23.786 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-02-05T04:50:23.793 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-02-05T04:50:23.807 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-02-05T04:50:23.828 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-02-05T04:50:23.851 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-02-05T04:50:23.906 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-02-05T04:50:23.927 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-02-05T04:50:24.056 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph 2024-02-05T04:50:24.312 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-02-05T04:50:24.359 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-02-05T04:50:24.393 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-02-05T04:50:24.440 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-02-05T04:50:24.471 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-rook-2:19.0.0-658.gd4ed20cf.el9.noarch 13/13 2024-02-05T04:50:24.601 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-02-05T04:50:24.618 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-02-05T04:50:24.632 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:00:45 ago on Mon 05 Feb 2024 04:49:39 AM UTC. 2024-02-05T04:50:24.705 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-02-05T04:50:24.735 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-02-05T04:50:24.798 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-02-05T04:50:24.800 INFO:teuthology.orchestra.run.smithi150.stdout:======================================================================================== 2024-02-05T04:50:24.801 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2024-02-05T04:50:24.801 INFO:teuthology.orchestra.run.smithi150.stdout:======================================================================================== 2024-02-05T04:50:24.801 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2024-02-05T04:50:24.801 INFO:teuthology.orchestra.run.smithi150.stdout: ceph x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 6.5 k 2024-02-05T04:50:24.801 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2024-02-05T04:50:24.801 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mds x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 2.1 M 2024-02-05T04:50:24.801 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 1.5 M 2024-02-05T04:50:24.801 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-modules-core noarch 2:19.0.0-658.gd4ed20cf.el9 ceph-noarch 245 k 2024-02-05T04:50:24.801 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mon x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 4.7 M 2024-02-05T04:50:24.801 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-osd x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 17 M 2024-02-05T04:50:24.801 INFO:teuthology.orchestra.run.smithi150.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-02-05T04:50:24.801 INFO:teuthology.orchestra.run.smithi150.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-02-05T04:50:24.801 INFO:teuthology.orchestra.run.smithi150.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-02-05T04:50:24.802 INFO:teuthology.orchestra.run.smithi150.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-02-05T04:50:24.802 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-02-05T04:50:24.802 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-02-05T04:50:24.802 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-02-05T04:50:24.802 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-02-05T04:50:24.802 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-02-05T04:50:24.802 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-02-05T04:50:24.802 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-02-05T04:50:24.802 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-02-05T04:50:24.802 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-02-05T04:50:24.802 INFO:teuthology.orchestra.run.smithi150.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-02-05T04:50:24.802 INFO:teuthology.orchestra.run.smithi150.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-02-05T04:50:24.803 INFO:teuthology.orchestra.run.smithi150.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-02-05T04:50:24.803 INFO:teuthology.orchestra.run.smithi150.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-02-05T04:50:24.803 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-02-05T04:50:24.803 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-02-05T04:50:24.803 INFO:teuthology.orchestra.run.smithi150.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-02-05T04:50:24.803 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-02-05T04:50:24.803 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-02-05T04:50:24.803 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-02-05T04:50:24.803 INFO:teuthology.orchestra.run.smithi150.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-02-05T04:50:24.803 INFO:teuthology.orchestra.run.smithi150.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-02-05T04:50:24.803 INFO:teuthology.orchestra.run.smithi150.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-02-05T04:50:24.803 INFO:teuthology.orchestra.run.smithi150.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-02-05T04:50:24.803 INFO:teuthology.orchestra.run.smithi150.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-02-05T04:50:24.804 INFO:teuthology.orchestra.run.smithi150.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-02-05T04:50:24.804 INFO:teuthology.orchestra.run.smithi150.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-02-05T04:50:24.804 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:50:24.804 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-02-05T04:50:24.804 INFO:teuthology.orchestra.run.smithi150.stdout:======================================================================================== 2024-02-05T04:50:24.804 INFO:teuthology.orchestra.run.smithi150.stdout:Install 36 Packages 2024-02-05T04:50:24.804 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:50:24.805 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 30 M 2024-02-05T04:50:24.805 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 106 M 2024-02-05T04:50:24.805 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-02-05T04:50:25.199 INFO:teuthology.orchestra.run.smithi150.stdout:(1/36): ceph-19.0.0-658.gd4ed20cf.el9.x86_64.rp 37 kB/s | 6.5 kB 00:00 2024-02-05T04:50:25.249 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-658.gd4ed20cf.el9.noarch 13/13 2024-02-05T04:50:25.249 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-rook-2:19.0.0-658.gd4ed20cf.el9.noarch 1/13 2024-02-05T04:50:25.249 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-02-05T04:50:25.249 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-02-05T04:50:25.249 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-02-05T04:50:25.249 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-02-05T04:50:25.249 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-02-05T04:50:25.250 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-02-05T04:50:25.250 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-02-05T04:50:25.250 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-02-05T04:50:25.250 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-google-auth-1:2.27.0-1.el9.noarch 10/13 2024-02-05T04:50:25.250 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-02-05T04:50:25.250 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-02-05T04:50:25.516 INFO:teuthology.orchestra.run.smithi150.stdout:(2/36): ceph-mgr-19.0.0-658.gd4ed20cf.el9.x86_6 3.0 MB/s | 1.5 MB 00:00 2024-02-05T04:50:25.631 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-02-05T04:50:25.631 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:50:25.631 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-02-05T04:50:25.631 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-rook-2:19.0.0-658.gd4ed20cf.el9.noarch 2024-02-05T04:50:25.631 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-02-05T04:50:25.631 INFO:teuthology.orchestra.run.smithi196.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-02-05T04:50:25.632 INFO:teuthology.orchestra.run.smithi196.stdout: python3-google-auth-1:2.27.0-1.el9.noarch 2024-02-05T04:50:25.632 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-02-05T04:50:25.632 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-02-05T04:50:25.632 INFO:teuthology.orchestra.run.smithi196.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-02-05T04:50:25.632 INFO:teuthology.orchestra.run.smithi196.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-02-05T04:50:25.632 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-02-05T04:50:25.632 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-02-05T04:50:25.632 INFO:teuthology.orchestra.run.smithi196.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-02-05T04:50:25.632 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rsa-4.9-2.el9.noarch 2024-02-05T04:50:25.632 INFO:teuthology.orchestra.run.smithi196.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-02-05T04:50:25.632 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:50:25.632 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-02-05T04:50:25.649 INFO:teuthology.orchestra.run.smithi150.stdout:(3/36): ceph-mds-19.0.0-658.gd4ed20cf.el9.x86_6 3.4 MB/s | 2.1 MB 00:00 2024-02-05T04:50:25.744 INFO:teuthology.orchestra.run.smithi150.stdout:(4/36): ceph-mgr-modules-core-19.0.0-658.gd4ed2 2.6 MB/s | 245 kB 00:00 2024-02-05T04:50:25.852 INFO:teuthology.orchestra.run.smithi150.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.7 MB/s | 187 kB 00:00 2024-02-05T04:50:25.889 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-mgr-cephadm 2024-02-05T04:50:25.894 INFO:teuthology.orchestra.run.smithi150.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-02-05T04:50:25.924 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-02-05T04:50:25.927 INFO:teuthology.orchestra.run.smithi150.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 955 kB/s | 32 kB 00:00 2024-02-05T04:50:25.994 INFO:teuthology.orchestra.run.smithi150.stdout:(8/36): ceph-mon-19.0.0-658.gd4ed20cf.el9.x86_6 5.9 MB/s | 4.7 MB 00:00 2024-02-05T04:50:26.020 INFO:teuthology.orchestra.run.smithi150.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 514 kB/s | 47 kB 00:00 2024-02-05T04:50:26.035 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-02-05T04:50:26.045 INFO:teuthology.orchestra.run.smithi150.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 770 kB/s | 39 kB 00:00 2024-02-05T04:50:26.071 INFO:teuthology.orchestra.run.smithi150.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-02-05T04:50:26.121 INFO:teuthology.orchestra.run.smithi150.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-02-05T04:50:26.138 INFO:teuthology.orchestra.run.smithi150.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.5 MB/s | 103 kB 00:00 2024-02-05T04:50:26.163 INFO:teuthology.orchestra.run.smithi150.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.9 MB/s | 124 kB 00:00 2024-02-05T04:50:26.188 INFO:teuthology.orchestra.run.smithi150.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 2.2 MB/s | 113 kB 00:00 2024-02-05T04:50:26.213 INFO:teuthology.orchestra.run.smithi150.stdout:(16/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-02-05T04:50:26.364 INFO:teuthology.orchestra.run.smithi150.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 127 kB/s | 22 kB 00:00 2024-02-05T04:50:26.455 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:27 ago on Mon 05 Feb 2024 04:48:59 AM UTC. 2024-02-05T04:50:26.611 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-02-05T04:50:26.612 INFO:teuthology.orchestra.run.smithi196.stdout:======================================================================================= 2024-02-05T04:50:26.612 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-02-05T04:50:26.612 INFO:teuthology.orchestra.run.smithi196.stdout:======================================================================================= 2024-02-05T04:50:26.612 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-02-05T04:50:26.612 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-cephadm noarch 2:19.0.0-658.gd4ed20cf.el9 ceph-noarch 133 k 2024-02-05T04:50:26.612 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-02-05T04:50:26.612 INFO:teuthology.orchestra.run.smithi196.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2024-02-05T04:50:26.612 INFO:teuthology.orchestra.run.smithi196.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-02-05T04:50:26.612 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2024-02-05T04:50:26.612 INFO:teuthology.orchestra.run.smithi196.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-02-05T04:50:26.612 INFO:teuthology.orchestra.run.smithi196.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-02-05T04:50:26.612 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:50:26.613 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-02-05T04:50:26.613 INFO:teuthology.orchestra.run.smithi196.stdout:======================================================================================= 2024-02-05T04:50:26.613 INFO:teuthology.orchestra.run.smithi196.stdout:Install 6 Packages 2024-02-05T04:50:26.613 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:50:26.613 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 6.8 M 2024-02-05T04:50:26.613 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 33 M 2024-02-05T04:50:26.614 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-02-05T04:50:26.697 INFO:teuthology.orchestra.run.smithi150.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 312 kB/s | 151 kB 00:00 2024-02-05T04:50:26.739 INFO:teuthology.orchestra.run.smithi150.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 116 kB/s | 43 kB 00:00 2024-02-05T04:50:26.915 INFO:teuthology.orchestra.run.smithi150.stdout:(20/36): ceph-osd-19.0.0-658.gd4ed20cf.el9.x86_ 12 MB/s | 17 MB 00:01 2024-02-05T04:50:26.940 INFO:teuthology.orchestra.run.smithi150.stdout:(21/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 709 kB/s | 172 kB 00:00 2024-02-05T04:50:26.968 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-02-05T04:50:26.992 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-658.gd4ed20 12/12 2024-02-05T04:50:27.099 INFO:teuthology.orchestra.run.smithi150.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 111 kB/s | 18 kB 00:00 2024-02-05T04:50:27.127 INFO:teuthology.orchestra.run.smithi196.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.4 MB/s | 228 kB 00:00 2024-02-05T04:50:27.132 INFO:teuthology.orchestra.run.smithi150.stdout:(23/36): python3-cherrypy-18.6.1-2.el9.noarch.r 913 kB/s | 358 kB 00:00 2024-02-05T04:50:27.141 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-02-05T04:50:27.157 INFO:teuthology.orchestra.run.smithi150.stdout:(24/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 44 kB/s | 11 kB 00:00 2024-02-05T04:50:27.168 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-658.gd4ed20 12/12 2024-02-05T04:50:27.194 INFO:teuthology.orchestra.run.smithi196.stdout:(2/6): ceph-mgr-cephadm-19.0.0-658.gd4ed20cf.el 583 kB/s | 133 kB 00:00 2024-02-05T04:50:27.250 INFO:teuthology.orchestra.run.smithi150.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 154 kB/s | 23 kB 00:00 2024-02-05T04:50:27.283 INFO:teuthology.orchestra.run.smithi150.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 129 kB/s | 19 kB 00:00 2024-02-05T04:50:27.308 INFO:teuthology.orchestra.run.smithi150.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 130 kB/s | 20 kB 00:00 2024-02-05T04:50:27.328 INFO:teuthology.orchestra.run.smithi196.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 16 MB/s | 5.8 MB 00:00 2024-02-05T04:50:27.392 INFO:teuthology.orchestra.run.smithi150.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 325 kB/s | 46 kB 00:00 2024-02-05T04:50:27.417 INFO:teuthology.orchestra.run.smithi150.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 589 kB/s | 79 kB 00:00 2024-02-05T04:50:27.518 INFO:teuthology.orchestra.run.smithi150.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 131 kB/s | 16 kB 00:00 2024-02-05T04:50:27.576 INFO:teuthology.orchestra.run.smithi150.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 567 kB/s | 90 kB 00:00 2024-02-05T04:50:27.595 INFO:teuthology.orchestra.run.smithi196.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 145 kB/s | 58 kB 00:00 2024-02-05T04:50:27.643 INFO:teuthology.orchestra.run.smithi150.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 286 kB/s | 36 kB 00:00 2024-02-05T04:50:27.670 INFO:teuthology.orchestra.run.smithi196.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 149 kB/s | 51 kB 00:00 2024-02-05T04:50:27.685 INFO:teuthology.orchestra.run.smithi150.stdout:(33/36): python3-pecan-1.4.2-3.el9.noarch.rpm 724 kB/s | 272 kB 00:00 2024-02-05T04:50:27.727 INFO:teuthology.orchestra.run.smithi150.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.5 MB/s | 230 kB 00:00 2024-02-05T04:50:27.728 INFO:teuthology.orchestra.run.smithi196.stdout:(6/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 913 kB/s | 548 kB 00:00 2024-02-05T04:50:27.730 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:50:27.731 INFO:teuthology.orchestra.run.smithi196.stdout:Total 6.1 MB/s | 6.8 MB 00:01 2024-02-05T04:50:27.731 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-02-05T04:50:27.740 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-02-05T04:50:27.740 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-02-05T04:50:27.819 INFO:teuthology.orchestra.run.smithi150.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 149 kB/s | 20 kB 00:00 2024-02-05T04:50:27.830 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-02-05T04:50:27.830 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-02-05T04:50:27.869 INFO:teuthology.orchestra.run.smithi150.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.9 MB/s | 427 kB 00:00 2024-02-05T04:50:27.873 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:50:27.873 INFO:teuthology.orchestra.run.smithi150.stdout:Total 9.6 MB/s | 30 MB 00:03 2024-02-05T04:50:27.892 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-02-05T04:50:27.935 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-02-05T04:50:27.935 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-02-05T04:50:28.134 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-02-05T04:50:28.300 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-02-05T04:50:28.334 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-02-05T04:50:28.334 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-02-05T04:50:28.355 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2024-02-05T04:50:28.862 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-02-05T04:50:28.956 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-02-05T04:50:29.026 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2024-02-05T04:50:29.051 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-658.gd4ed20cf.el9.noarch 6/6 2024-02-05T04:50:29.532 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-658.gd4ed20 12/12 2024-02-05T04:50:29.532 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-658.gd4ed20 1/12 2024-02-05T04:50:29.532 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-02-05T04:50:29.532 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-02-05T04:50:29.532 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-02-05T04:50:29.532 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-02-05T04:50:29.532 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-02-05T04:50:29.533 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-02-05T04:50:29.533 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-02-05T04:50:29.533 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-02-05T04:50:29.533 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-02-05T04:50:29.533 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-02-05T04:50:29.669 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-02-05T04:50:29.674 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-658.gd4ed20 12/12 2024-02-05T04:50:29.674 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-658.gd4ed20 1/12 2024-02-05T04:50:29.674 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-02-05T04:50:29.674 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-02-05T04:50:29.674 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-02-05T04:50:29.675 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-02-05T04:50:29.675 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-02-05T04:50:29.675 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-02-05T04:50:29.675 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-02-05T04:50:29.675 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-02-05T04:50:29.675 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-02-05T04:50:29.675 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-02-05T04:50:29.739 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-02-05T04:50:29.767 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-02-05T04:50:29.799 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-02-05T04:50:29.869 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-02-05T04:50:29.874 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-658.gd4ed20cf.el9.noarch 6/6 2024-02-05T04:50:29.875 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-658.gd4ed20cf.el9.noarch 1/6 2024-02-05T04:50:29.875 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-02-05T04:50:29.875 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2024-02-05T04:50:29.875 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2024-02-05T04:50:29.875 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-02-05T04:50:29.906 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-02-05T04:50:29.939 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-02-05T04:50:30.012 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-02-05T04:50:30.013 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:50:30.013 INFO:teuthology.orchestra.run.smithi091.stdout:Installed: 2024-02-05T04:50:30.013 INFO:teuthology.orchestra.run.smithi091.stdout: ceph-mgr-diskprediction-local-2:19.0.0-658.gd4ed20cf.el9.noarch 2024-02-05T04:50:30.013 INFO:teuthology.orchestra.run.smithi091.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-02-05T04:50:30.013 INFO:teuthology.orchestra.run.smithi091.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-02-05T04:50:30.013 INFO:teuthology.orchestra.run.smithi091.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-02-05T04:50:30.013 INFO:teuthology.orchestra.run.smithi091.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-02-05T04:50:30.013 INFO:teuthology.orchestra.run.smithi091.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-02-05T04:50:30.013 INFO:teuthology.orchestra.run.smithi091.stdout: openblas-0.3.21-2.el9.x86_64 2024-02-05T04:50:30.013 INFO:teuthology.orchestra.run.smithi091.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-02-05T04:50:30.013 INFO:teuthology.orchestra.run.smithi091.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-02-05T04:50:30.013 INFO:teuthology.orchestra.run.smithi091.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-02-05T04:50:30.014 INFO:teuthology.orchestra.run.smithi091.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-02-05T04:50:30.014 INFO:teuthology.orchestra.run.smithi091.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-02-05T04:50:30.014 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:50:30.014 INFO:teuthology.orchestra.run.smithi091.stdout:Complete! 2024-02-05T04:50:30.030 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-02-05T04:50:30.109 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-02-05T04:50:30.140 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-02-05T04:50:30.140 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:50:30.141 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2024-02-05T04:50:30.141 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-diskprediction-local-2:19.0.0-658.gd4ed20cf.el9.noarch 2024-02-05T04:50:30.141 INFO:teuthology.orchestra.run.smithi062.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-02-05T04:50:30.141 INFO:teuthology.orchestra.run.smithi062.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-02-05T04:50:30.141 INFO:teuthology.orchestra.run.smithi062.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-02-05T04:50:30.141 INFO:teuthology.orchestra.run.smithi062.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-02-05T04:50:30.141 INFO:teuthology.orchestra.run.smithi062.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-02-05T04:50:30.141 INFO:teuthology.orchestra.run.smithi062.stdout: openblas-0.3.21-2.el9.x86_64 2024-02-05T04:50:30.141 INFO:teuthology.orchestra.run.smithi062.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-02-05T04:50:30.141 INFO:teuthology.orchestra.run.smithi062.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-02-05T04:50:30.141 INFO:teuthology.orchestra.run.smithi062.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-02-05T04:50:30.142 INFO:teuthology.orchestra.run.smithi062.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-02-05T04:50:30.142 INFO:teuthology.orchestra.run.smithi062.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-02-05T04:50:30.142 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:50:30.142 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-02-05T04:50:30.150 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-02-05T04:50:30.178 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-02-05T04:50:30.178 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:50:30.178 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-02-05T04:50:30.178 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-cephadm-2:19.0.0-658.gd4ed20cf.el9.noarch 2024-02-05T04:50:30.179 INFO:teuthology.orchestra.run.smithi196.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2024-02-05T04:50:30.179 INFO:teuthology.orchestra.run.smithi196.stdout: python3-babel-2.9.1-2.el9.noarch 2024-02-05T04:50:30.179 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jinja2-2.11.3-4.el9.noarch 2024-02-05T04:50:30.179 INFO:teuthology.orchestra.run.smithi196.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-02-05T04:50:30.179 INFO:teuthology.orchestra.run.smithi196.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-02-05T04:50:30.179 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:50:30.179 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-02-05T04:50:30.221 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-02-05T04:50:30.267 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-02-05T04:50:30.306 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-02-05T04:50:30.354 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-02-05T04:50:30.370 DEBUG:teuthology.orchestra.run.smithi091:> sudo yum -y install ceph-mgr-rook 2024-02-05T04:50:30.410 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-02-05T04:50:30.418 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-fuse 2024-02-05T04:50:30.483 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install ceph-mgr-rook 2024-02-05T04:50:30.597 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-02-05T04:50:30.644 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-02-05T04:50:30.699 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-02-05T04:50:30.734 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-02-05T04:50:30.764 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-02-05T04:50:30.839 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-02-05T04:50:30.874 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-02-05T04:50:30.908 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-02-05T04:50:30.941 INFO:teuthology.orchestra.run.smithi091.stdout:Last metadata expiration check: 0:01:21 ago on Mon 05 Feb 2024 04:49:09 AM UTC. 2024-02-05T04:50:30.948 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-02-05T04:50:30.982 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:31 ago on Mon 05 Feb 2024 04:48:59 AM UTC. 2024-02-05T04:50:30.983 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-02-05T04:50:31.026 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-02-05T04:50:31.058 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:01:24 ago on Mon 05 Feb 2024 04:49:07 AM UTC. 2024-02-05T04:50:31.099 INFO:teuthology.orchestra.run.smithi091.stdout:Dependencies resolved. 2024-02-05T04:50:31.100 INFO:teuthology.orchestra.run.smithi091.stdout:======================================================================================= 2024-02-05T04:50:31.100 INFO:teuthology.orchestra.run.smithi091.stdout: Package Arch Version Repository Size 2024-02-05T04:50:31.100 INFO:teuthology.orchestra.run.smithi091.stdout:======================================================================================= 2024-02-05T04:50:31.101 INFO:teuthology.orchestra.run.smithi091.stdout:Installing: 2024-02-05T04:50:31.101 INFO:teuthology.orchestra.run.smithi091.stdout: ceph-mgr-rook noarch 2:19.0.0-658.gd4ed20cf.el9 ceph-noarch 50 k 2024-02-05T04:50:31.101 INFO:teuthology.orchestra.run.smithi091.stdout:Installing dependencies: 2024-02-05T04:50:31.101 INFO:teuthology.orchestra.run.smithi091.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-02-05T04:50:31.101 INFO:teuthology.orchestra.run.smithi091.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-02-05T04:50:31.101 INFO:teuthology.orchestra.run.smithi091.stdout: python3-google-auth noarch 1:2.27.0-1.el9 epel 203 k 2024-02-05T04:50:31.101 INFO:teuthology.orchestra.run.smithi091.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-02-05T04:50:31.101 INFO:teuthology.orchestra.run.smithi091.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-02-05T04:50:31.101 INFO:teuthology.orchestra.run.smithi091.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-02-05T04:50:31.101 INFO:teuthology.orchestra.run.smithi091.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-02-05T04:50:31.101 INFO:teuthology.orchestra.run.smithi091.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-02-05T04:50:31.101 INFO:teuthology.orchestra.run.smithi091.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-02-05T04:50:31.101 INFO:teuthology.orchestra.run.smithi091.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-02-05T04:50:31.101 INFO:teuthology.orchestra.run.smithi091.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-02-05T04:50:31.101 INFO:teuthology.orchestra.run.smithi091.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-02-05T04:50:31.102 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:50:31.102 INFO:teuthology.orchestra.run.smithi091.stdout:Transaction Summary 2024-02-05T04:50:31.102 INFO:teuthology.orchestra.run.smithi091.stdout:======================================================================================= 2024-02-05T04:50:31.102 INFO:teuthology.orchestra.run.smithi091.stdout:Install 13 Packages 2024-02-05T04:50:31.102 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:50:31.102 INFO:teuthology.orchestra.run.smithi091.stdout:Total download size: 2.0 M 2024-02-05T04:50:31.102 INFO:teuthology.orchestra.run.smithi091.stdout:Installed size: 26 M 2024-02-05T04:50:31.103 INFO:teuthology.orchestra.run.smithi091.stdout:Downloading Packages: 2024-02-05T04:50:31.115 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-02-05T04:50:31.140 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-02-05T04:50:31.141 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-02-05T04:50:31.141 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-02-05T04:50:31.141 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-02-05T04:50:31.141 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-02-05T04:50:31.141 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-fuse x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 887 k 2024-02-05T04:50:31.142 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-02-05T04:50:31.142 INFO:teuthology.orchestra.run.smithi196.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-02-05T04:50:31.142 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:50:31.142 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-02-05T04:50:31.142 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-02-05T04:50:31.142 INFO:teuthology.orchestra.run.smithi196.stdout:Install 2 Packages 2024-02-05T04:50:31.142 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:50:31.142 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 966 k 2024-02-05T04:50:31.142 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 2.8 M 2024-02-05T04:50:31.142 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-02-05T04:50:31.157 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-02-05T04:50:31.224 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-02-05T04:50:31.226 INFO:teuthology.orchestra.run.smithi062.stdout:======================================================================================= 2024-02-05T04:50:31.226 INFO:teuthology.orchestra.run.smithi062.stdout: Package Arch Version Repository Size 2024-02-05T04:50:31.226 INFO:teuthology.orchestra.run.smithi062.stdout:======================================================================================= 2024-02-05T04:50:31.226 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2024-02-05T04:50:31.226 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-rook noarch 2:19.0.0-658.gd4ed20cf.el9 ceph-noarch 50 k 2024-02-05T04:50:31.226 INFO:teuthology.orchestra.run.smithi062.stdout:Installing dependencies: 2024-02-05T04:50:31.226 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-02-05T04:50:31.226 INFO:teuthology.orchestra.run.smithi062.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-02-05T04:50:31.226 INFO:teuthology.orchestra.run.smithi062.stdout: python3-google-auth noarch 1:2.27.0-1.el9 epel 203 k 2024-02-05T04:50:31.226 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-02-05T04:50:31.226 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-02-05T04:50:31.227 INFO:teuthology.orchestra.run.smithi062.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-02-05T04:50:31.227 INFO:teuthology.orchestra.run.smithi062.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-02-05T04:50:31.227 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-02-05T04:50:31.227 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-02-05T04:50:31.227 INFO:teuthology.orchestra.run.smithi062.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-02-05T04:50:31.227 INFO:teuthology.orchestra.run.smithi062.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-02-05T04:50:31.227 INFO:teuthology.orchestra.run.smithi062.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-02-05T04:50:31.227 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:50:31.227 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2024-02-05T04:50:31.227 INFO:teuthology.orchestra.run.smithi062.stdout:======================================================================================= 2024-02-05T04:50:31.227 INFO:teuthology.orchestra.run.smithi062.stdout:Install 13 Packages 2024-02-05T04:50:31.227 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:50:31.228 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 2.0 M 2024-02-05T04:50:31.228 INFO:teuthology.orchestra.run.smithi062.stdout:Installed size: 26 M 2024-02-05T04:50:31.228 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2024-02-05T04:50:31.229 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-02-05T04:50:31.272 INFO:teuthology.orchestra.run.smithi196.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 606 kB/s | 79 kB 00:00 2024-02-05T04:50:31.292 INFO:teuthology.orchestra.run.smithi091.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 171 kB/s | 24 kB 00:00 2024-02-05T04:50:31.318 INFO:teuthology.orchestra.run.smithi091.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 108 kB/s | 18 kB 00:00 2024-02-05T04:50:31.365 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-02-05T04:50:31.401 INFO:teuthology.orchestra.run.smithi091.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.6 MB/s | 174 kB 00:00 2024-02-05T04:50:31.427 INFO:teuthology.orchestra.run.smithi091.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.2 MB/s | 132 kB 00:00 2024-02-05T04:50:31.447 INFO:teuthology.orchestra.run.smithi196.stdout:(2/2): ceph-fuse-19.0.0-658.gd4ed20cf.el9.x86_6 2.8 MB/s | 887 kB 00:00 2024-02-05T04:50:31.448 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:50:31.448 INFO:teuthology.orchestra.run.smithi196.stdout:Total 3.1 MB/s | 966 kB 00:00 2024-02-05T04:50:31.448 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-02-05T04:50:31.468 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-02-05T04:50:31.468 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-02-05T04:50:31.477 INFO:teuthology.orchestra.run.smithi091.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.7 MB/s | 211 kB 00:00 2024-02-05T04:50:31.484 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-02-05T04:50:31.501 INFO:teuthology.orchestra.run.smithi062.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 156 kB/s | 24 kB 00:00 2024-02-05T04:50:31.502 INFO:teuthology.orchestra.run.smithi091.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 570 kB/s | 43 kB 00:00 2024-02-05T04:50:31.518 INFO:teuthology.orchestra.run.smithi062.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 104 kB/s | 18 kB 00:00 2024-02-05T04:50:31.536 INFO:teuthology.orchestra.run.smithi091.stdout:(7/13): ceph-mgr-rook-19.0.0-658.gd4ed20cf.el9. 131 kB/s | 50 kB 00:00 2024-02-05T04:50:31.544 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-02-05T04:50:31.545 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-02-05T04:50:31.635 INFO:teuthology.orchestra.run.smithi062.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.3 MB/s | 174 kB 00:00 2024-02-05T04:50:31.650 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-658.gd4ed20cf.el9.n 31/36 2024-02-05T04:50:31.652 INFO:teuthology.orchestra.run.smithi062.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 989 kB/s | 132 kB 00:00 2024-02-05T04:50:31.661 INFO:teuthology.orchestra.run.smithi091.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 94 kB/s | 15 kB 00:00 2024-02-05T04:50:31.681 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-2:19.0.0-658.gd4ed20cf.el9.x86_64 32/36 2024-02-05T04:50:31.686 INFO:teuthology.orchestra.run.smithi091.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 154 kB/s | 32 kB 00:00 2024-02-05T04:50:31.702 INFO:teuthology.orchestra.run.smithi062.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.1 MB/s | 211 kB 00:00 2024-02-05T04:50:31.705 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-2:19.0.0-658.gd4ed20cf.el9.x86_64 32/36 2024-02-05T04:50:31.705 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T04:50:31.705 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-02-05T04:50:31.705 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-05T04:50:31.705 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-05T04:50:31.705 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:50:31.719 INFO:teuthology.orchestra.run.smithi062.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 638 kB/s | 43 kB 00:00 2024-02-05T04:50:31.742 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-02-05T04:50:31.745 INFO:teuthology.orchestra.run.smithi062.stdout:(7/13): ceph-mgr-rook-19.0.0-658.gd4ed20cf.el9. 126 kB/s | 50 kB 00:00 2024-02-05T04:50:31.803 INFO:teuthology.orchestra.run.smithi091.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 507 kB/s | 59 kB 00:00 2024-02-05T04:50:31.829 INFO:teuthology.orchestra.run.smithi091.stdout:(11/13): python3-google-auth-2.27.0-1.el9.noarc 693 kB/s | 203 kB 00:00 2024-02-05T04:50:31.845 INFO:teuthology.orchestra.run.smithi062.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 119 kB/s | 15 kB 00:00 2024-02-05T04:50:31.862 INFO:teuthology.orchestra.run.smithi062.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 202 kB/s | 32 kB 00:00 2024-02-05T04:50:31.881 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-02-05T04:50:31.887 INFO:teuthology.orchestra.run.smithi062.stdout:(10/13): python3-google-auth-2.27.0-1.el9.noarc 1.4 MB/s | 203 kB 00:00 2024-02-05T04:50:31.895 INFO:teuthology.orchestra.run.smithi091.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 977 kB/s | 90 kB 00:00 2024-02-05T04:50:31.929 INFO:teuthology.orchestra.run.smithi062.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 2.1 MB/s | 90 kB 00:00 2024-02-05T04:50:31.955 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-fuse-2:19.0.0-658.gd4ed20cf.el9.x86_64 2/2 2024-02-05T04:50:31.979 INFO:teuthology.orchestra.run.smithi062.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 504 kB/s | 59 kB 00:00 2024-02-05T04:50:32.088 INFO:teuthology.orchestra.run.smithi062.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 4.2 MB/s | 1.0 MB 00:00 2024-02-05T04:50:32.090 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:50:32.090 INFO:teuthology.orchestra.run.smithi062.stdout:Total 2.4 MB/s | 2.0 MB 00:00 2024-02-05T04:50:32.090 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2024-02-05T04:50:32.105 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2024-02-05T04:50:32.105 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2024-02-05T04:50:32.154 INFO:teuthology.orchestra.run.smithi091.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.1 MB/s | 1.0 MB 00:00 2024-02-05T04:50:32.156 INFO:teuthology.orchestra.run.smithi091.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:50:32.156 INFO:teuthology.orchestra.run.smithi091.stdout:Total 1.9 MB/s | 2.0 MB 00:01 2024-02-05T04:50:32.157 INFO:teuthology.orchestra.run.smithi091.stdout:Running transaction check 2024-02-05T04:50:32.170 INFO:teuthology.orchestra.run.smithi091.stdout:Transaction check succeeded. 2024-02-05T04:50:32.170 INFO:teuthology.orchestra.run.smithi091.stdout:Running transaction test 2024-02-05T04:50:32.214 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2024-02-05T04:50:32.215 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2024-02-05T04:50:32.272 INFO:teuthology.orchestra.run.smithi091.stdout:Transaction test succeeded. 2024-02-05T04:50:32.274 INFO:teuthology.orchestra.run.smithi091.stdout:Running transaction 2024-02-05T04:50:32.715 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-fuse-2:19.0.0-658.gd4ed20cf.el9.x86_64 2/2 2024-02-05T04:50:32.715 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-fuse-2:19.0.0-658.gd4ed20cf.el9.x86_64 1/2 2024-02-05T04:50:32.743 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2024-02-05T04:50:32.787 INFO:teuthology.orchestra.run.smithi091.stdout: Preparing : 1/1 2024-02-05T04:50:32.918 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-02-05T04:50:32.948 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-02-05T04:50:32.974 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-02-05T04:50:33.012 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-02-05T04:50:33.030 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-02-05T04:50:33.033 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-02-05T04:50:33.033 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:50:33.033 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-02-05T04:50:33.033 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-fuse-2:19.0.0-658.gd4ed20cf.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-02-05T04:50:33.033 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:50:33.033 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-02-05T04:50:33.065 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-02-05T04:50:33.067 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-02-05T04:50:33.108 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-02-05T04:50:33.115 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-02-05T04:50:33.123 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-osd-2:19.0.0-658.gd4ed20cf.el9.x86_64 33/36 2024-02-05T04:50:33.150 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-osd-2:19.0.0-658.gd4ed20cf.el9.x86_64 33/36 2024-02-05T04:50:33.150 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T04:50:33.150 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-02-05T04:50:33.150 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-05T04:50:33.150 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-05T04:50:33.150 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:50:33.158 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-02-05T04:50:33.204 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-02-05T04:50:33.248 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-02-05T04:50:33.300 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-volume 2024-02-05T04:50:33.313 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-google-auth-1:2.27.0-1.el9.noarch 7/13 2024-02-05T04:50:33.356 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-google-auth-1:2.27.0-1.el9.noarch 7/13 2024-02-05T04:50:33.368 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-02-05T04:50:33.415 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-02-05T04:50:33.590 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mon-2:19.0.0-658.gd4ed20cf.el9.x86_64 34/36 2024-02-05T04:50:33.614 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mon-2:19.0.0-658.gd4ed20cf.el9.x86_64 34/36 2024-02-05T04:50:33.614 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T04:50:33.615 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-02-05T04:50:33.615 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-05T04:50:33.615 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-05T04:50:33.615 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:50:33.833 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mds-2:19.0.0-658.gd4ed20cf.el9.x86_64 35/36 2024-02-05T04:50:33.857 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mds-2:19.0.0-658.gd4ed20cf.el9.x86_64 35/36 2024-02-05T04:50:33.857 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T04:50:33.857 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-02-05T04:50:33.857 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-05T04:50:33.857 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-05T04:50:33.857 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:50:33.881 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:34 ago on Mon 05 Feb 2024 04:48:59 AM UTC. 2024-02-05T04:50:33.922 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-2:19.0.0-658.gd4ed20cf.el9.x86_64 36/36 2024-02-05T04:50:34.043 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-02-05T04:50:34.043 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-02-05T04:50:34.044 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-02-05T04:50:34.044 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-02-05T04:50:34.044 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-02-05T04:50:34.044 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-volume noarch 2:19.0.0-658.gd4ed20cf.el9 ceph-noarch 262 k 2024-02-05T04:50:34.044 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:50:34.044 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-02-05T04:50:34.044 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-02-05T04:50:34.044 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-02-05T04:50:34.044 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:50:34.044 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 262 k 2024-02-05T04:50:34.044 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 1.2 M 2024-02-05T04:50:34.044 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-02-05T04:50:34.132 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-02-05T04:50:34.134 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-02-05T04:50:34.179 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-02-05T04:50:34.180 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-02-05T04:50:34.213 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-02-05T04:50:34.214 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-02-05T04:50:34.260 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-02-05T04:50:34.262 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-02-05T04:50:34.282 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : ceph-mgr-rook-2:19.0.0-658.gd4ed20cf.el9.noarch 13/13 2024-02-05T04:50:34.283 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-mgr-rook-2:19.0.0-658.gd4ed20cf.el9.noarch 13/13 2024-02-05T04:50:34.402 INFO:teuthology.orchestra.run.smithi196.stdout:ceph-volume-19.0.0-658.gd4ed20cf.el9.noarch.rpm 734 kB/s | 262 kB 00:00 2024-02-05T04:50:34.402 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:50:34.402 INFO:teuthology.orchestra.run.smithi196.stdout:Total 732 kB/s | 262 kB 00:00 2024-02-05T04:50:34.403 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-02-05T04:50:34.410 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-02-05T04:50:34.410 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-02-05T04:50:34.447 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-02-05T04:50:34.448 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-02-05T04:50:34.658 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-02-05T04:50:34.712 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-volume-2:19.0.0-658.gd4ed20cf.el9.noarch 1/1 2024-02-05T04:50:34.727 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-volume-2:19.0.0-658.gd4ed20cf.el9.noarch 1/1 2024-02-05T04:50:34.728 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T04:50:34.728 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-02-05T04:50:34.728 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:50:35.061 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-658.gd4ed20cf.el9.noarch 13/13 2024-02-05T04:50:35.061 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : ceph-mgr-rook-2:19.0.0-658.gd4ed20cf.el9.noarch 1/13 2024-02-05T04:50:35.062 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-02-05T04:50:35.062 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-02-05T04:50:35.062 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-02-05T04:50:35.062 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-02-05T04:50:35.062 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-02-05T04:50:35.062 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-02-05T04:50:35.062 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-02-05T04:50:35.062 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-02-05T04:50:35.062 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-google-auth-1:2.27.0-1.el9.noarch 10/13 2024-02-05T04:50:35.062 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-02-05T04:50:35.062 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-02-05T04:50:35.078 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-658.gd4ed20cf.el9.noarch 13/13 2024-02-05T04:50:35.078 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-mgr-rook-2:19.0.0-658.gd4ed20cf.el9.noarch 1/13 2024-02-05T04:50:35.078 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-02-05T04:50:35.078 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-02-05T04:50:35.078 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-02-05T04:50:35.078 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-02-05T04:50:35.078 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-02-05T04:50:35.078 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-02-05T04:50:35.078 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-02-05T04:50:35.078 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-02-05T04:50:35.078 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-google-auth-1:2.27.0-1.el9.noarch 10/13 2024-02-05T04:50:35.078 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-02-05T04:50:35.078 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-02-05T04:50:35.438 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-02-05T04:50:35.438 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:50:35.438 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2024-02-05T04:50:35.438 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-rook-2:19.0.0-658.gd4ed20cf.el9.noarch 2024-02-05T04:50:35.439 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-02-05T04:50:35.439 INFO:teuthology.orchestra.run.smithi062.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-02-05T04:50:35.439 INFO:teuthology.orchestra.run.smithi062.stdout: python3-google-auth-1:2.27.0-1.el9.noarch 2024-02-05T04:50:35.439 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-02-05T04:50:35.439 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-02-05T04:50:35.439 INFO:teuthology.orchestra.run.smithi062.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-02-05T04:50:35.439 INFO:teuthology.orchestra.run.smithi062.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-02-05T04:50:35.439 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-02-05T04:50:35.439 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-02-05T04:50:35.439 INFO:teuthology.orchestra.run.smithi062.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-02-05T04:50:35.439 INFO:teuthology.orchestra.run.smithi062.stdout: python3-rsa-4.9-2.el9.noarch 2024-02-05T04:50:35.439 INFO:teuthology.orchestra.run.smithi062.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-02-05T04:50:35.439 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:50:35.439 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-02-05T04:50:35.440 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-02-05T04:50:35.440 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:50:35.440 INFO:teuthology.orchestra.run.smithi091.stdout:Installed: 2024-02-05T04:50:35.440 INFO:teuthology.orchestra.run.smithi091.stdout: ceph-mgr-rook-2:19.0.0-658.gd4ed20cf.el9.noarch 2024-02-05T04:50:35.440 INFO:teuthology.orchestra.run.smithi091.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-02-05T04:50:35.440 INFO:teuthology.orchestra.run.smithi091.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-02-05T04:50:35.440 INFO:teuthology.orchestra.run.smithi091.stdout: python3-google-auth-1:2.27.0-1.el9.noarch 2024-02-05T04:50:35.441 INFO:teuthology.orchestra.run.smithi091.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-02-05T04:50:35.441 INFO:teuthology.orchestra.run.smithi091.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-02-05T04:50:35.441 INFO:teuthology.orchestra.run.smithi091.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-02-05T04:50:35.441 INFO:teuthology.orchestra.run.smithi091.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-02-05T04:50:35.441 INFO:teuthology.orchestra.run.smithi091.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-02-05T04:50:35.441 INFO:teuthology.orchestra.run.smithi091.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-02-05T04:50:35.441 INFO:teuthology.orchestra.run.smithi091.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-02-05T04:50:35.441 INFO:teuthology.orchestra.run.smithi091.stdout: python3-rsa-4.9-2.el9.noarch 2024-02-05T04:50:35.441 INFO:teuthology.orchestra.run.smithi091.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-02-05T04:50:35.441 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:50:35.441 INFO:teuthology.orchestra.run.smithi091.stdout:Complete! 2024-02-05T04:50:35.695 DEBUG:teuthology.orchestra.run.smithi091:> sudo yum -y install ceph-mgr-cephadm 2024-02-05T04:50:35.716 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install ceph-mgr-cephadm 2024-02-05T04:50:35.787 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-2:19.0.0-658.gd4ed20cf.el9.x86_64 36/36 2024-02-05T04:50:35.787 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-2:19.0.0-658.gd4ed20cf.el9.x86_64 1/36 2024-02-05T04:50:35.787 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mds-2:19.0.0-658.gd4ed20cf.el9.x86_64 2/36 2024-02-05T04:50:35.787 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-2:19.0.0-658.gd4ed20cf.el9.x86_64 3/36 2024-02-05T04:50:35.787 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mon-2:19.0.0-658.gd4ed20cf.el9.x86_64 4/36 2024-02-05T04:50:35.787 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-osd-2:19.0.0-658.gd4ed20cf.el9.x86_64 5/36 2024-02-05T04:50:35.787 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-658.gd4ed20cf.el9.n 6/36 2024-02-05T04:50:35.787 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-02-05T04:50:35.787 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-02-05T04:50:35.787 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-02-05T04:50:35.788 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-02-05T04:50:35.788 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-02-05T04:50:35.788 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-02-05T04:50:35.788 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-02-05T04:50:35.788 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-02-05T04:50:35.789 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-02-05T04:50:35.789 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 16/36 2024-02-05T04:50:35.789 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 17/36 2024-02-05T04:50:35.789 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-02-05T04:50:35.789 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-02-05T04:50:35.789 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-02-05T04:50:35.789 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-02-05T04:50:35.789 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-02-05T04:50:35.789 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-02-05T04:50:35.789 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-02-05T04:50:35.790 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-02-05T04:50:35.790 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-02-05T04:50:35.790 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-02-05T04:50:35.790 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-02-05T04:50:35.790 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-02-05T04:50:35.790 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-02-05T04:50:35.790 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-02-05T04:50:35.790 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-02-05T04:50:35.790 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-02-05T04:50:35.790 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-02-05T04:50:35.790 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-02-05T04:50:35.807 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-volume-2:19.0.0-658.gd4ed20cf.el9.noarch 1/1 2024-02-05T04:50:35.807 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:50:35.807 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-02-05T04:50:35.807 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-volume-2:19.0.0-658.gd4ed20cf.el9.noarch 2024-02-05T04:50:35.807 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:50:35.807 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-02-05T04:50:36.144 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install librados-devel 2024-02-05T04:50:36.265 INFO:teuthology.orchestra.run.smithi091.stdout:Last metadata expiration check: 0:01:27 ago on Mon 05 Feb 2024 04:49:09 AM UTC. 2024-02-05T04:50:36.297 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:01:29 ago on Mon 05 Feb 2024 04:49:07 AM UTC. 2024-02-05T04:50:36.421 INFO:teuthology.orchestra.run.smithi091.stdout:Dependencies resolved. 2024-02-05T04:50:36.421 INFO:teuthology.orchestra.run.smithi091.stdout:======================================================================================= 2024-02-05T04:50:36.422 INFO:teuthology.orchestra.run.smithi091.stdout: Package Arch Version Repository Size 2024-02-05T04:50:36.422 INFO:teuthology.orchestra.run.smithi091.stdout:======================================================================================= 2024-02-05T04:50:36.422 INFO:teuthology.orchestra.run.smithi091.stdout:Installing: 2024-02-05T04:50:36.422 INFO:teuthology.orchestra.run.smithi091.stdout: ceph-mgr-cephadm noarch 2:19.0.0-658.gd4ed20cf.el9 ceph-noarch 133 k 2024-02-05T04:50:36.422 INFO:teuthology.orchestra.run.smithi091.stdout:Installing dependencies: 2024-02-05T04:50:36.422 INFO:teuthology.orchestra.run.smithi091.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2024-02-05T04:50:36.422 INFO:teuthology.orchestra.run.smithi091.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-02-05T04:50:36.422 INFO:teuthology.orchestra.run.smithi091.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2024-02-05T04:50:36.422 INFO:teuthology.orchestra.run.smithi091.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-02-05T04:50:36.422 INFO:teuthology.orchestra.run.smithi091.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-02-05T04:50:36.422 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:50:36.423 INFO:teuthology.orchestra.run.smithi091.stdout:Transaction Summary 2024-02-05T04:50:36.423 INFO:teuthology.orchestra.run.smithi091.stdout:======================================================================================= 2024-02-05T04:50:36.423 INFO:teuthology.orchestra.run.smithi091.stdout:Install 6 Packages 2024-02-05T04:50:36.423 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:50:36.423 INFO:teuthology.orchestra.run.smithi091.stdout:Total download size: 6.8 M 2024-02-05T04:50:36.423 INFO:teuthology.orchestra.run.smithi091.stdout:Installed size: 33 M 2024-02-05T04:50:36.424 INFO:teuthology.orchestra.run.smithi091.stdout:Downloading Packages: 2024-02-05T04:50:36.461 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-02-05T04:50:36.461 INFO:teuthology.orchestra.run.smithi062.stdout:======================================================================================= 2024-02-05T04:50:36.461 INFO:teuthology.orchestra.run.smithi062.stdout: Package Arch Version Repository Size 2024-02-05T04:50:36.462 INFO:teuthology.orchestra.run.smithi062.stdout:======================================================================================= 2024-02-05T04:50:36.462 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2024-02-05T04:50:36.462 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-cephadm noarch 2:19.0.0-658.gd4ed20cf.el9 ceph-noarch 133 k 2024-02-05T04:50:36.462 INFO:teuthology.orchestra.run.smithi062.stdout:Installing dependencies: 2024-02-05T04:50:36.462 INFO:teuthology.orchestra.run.smithi062.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2024-02-05T04:50:36.462 INFO:teuthology.orchestra.run.smithi062.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-02-05T04:50:36.462 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2024-02-05T04:50:36.462 INFO:teuthology.orchestra.run.smithi062.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-02-05T04:50:36.462 INFO:teuthology.orchestra.run.smithi062.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-02-05T04:50:36.462 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:50:36.462 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2024-02-05T04:50:36.462 INFO:teuthology.orchestra.run.smithi062.stdout:======================================================================================= 2024-02-05T04:50:36.462 INFO:teuthology.orchestra.run.smithi062.stdout:Install 6 Packages 2024-02-05T04:50:36.462 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:50:36.463 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 6.8 M 2024-02-05T04:50:36.463 INFO:teuthology.orchestra.run.smithi062.stdout:Installed size: 33 M 2024-02-05T04:50:36.463 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2024-02-05T04:50:36.588 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-02-05T04:50:36.589 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:50:36.589 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2024-02-05T04:50:36.589 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:50:36.589 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mds-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:50:36.589 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:50:36.589 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-modules-core-2:19.0.0-658.gd4ed20cf.el9.noarch 2024-02-05T04:50:36.589 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mon-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:50:36.589 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-osd-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:50:36.589 INFO:teuthology.orchestra.run.smithi150.stdout: lua-5.4.4-4.el9.x86_64 2024-02-05T04:50:36.589 INFO:teuthology.orchestra.run.smithi150.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-02-05T04:50:36.589 INFO:teuthology.orchestra.run.smithi150.stdout: luarocks-3.9.2-1.el9.noarch 2024-02-05T04:50:36.589 INFO:teuthology.orchestra.run.smithi150.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-02-05T04:50:36.589 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-02-05T04:50:36.590 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-02-05T04:50:36.590 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-02-05T04:50:36.590 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-02-05T04:50:36.590 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-02-05T04:50:36.590 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-02-05T04:50:36.590 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-02-05T04:50:36.590 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-02-05T04:50:36.590 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-02-05T04:50:36.590 INFO:teuthology.orchestra.run.smithi150.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-02-05T04:50:36.590 INFO:teuthology.orchestra.run.smithi150.stdout: python3-mako-1.1.4-6.el9.noarch 2024-02-05T04:50:36.590 INFO:teuthology.orchestra.run.smithi150.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-02-05T04:50:36.590 INFO:teuthology.orchestra.run.smithi150.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-02-05T04:50:36.590 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-02-05T04:50:36.590 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ply-3.11-14.el9.noarch 2024-02-05T04:50:36.591 INFO:teuthology.orchestra.run.smithi150.stdout: python3-portend-3.1.0-2.el9.noarch 2024-02-05T04:50:36.591 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-02-05T04:50:36.591 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pycparser-2.20-6.el9.noarch 2024-02-05T04:50:36.591 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pytz-2021.1-5.el9.noarch 2024-02-05T04:50:36.591 INFO:teuthology.orchestra.run.smithi150.stdout: python3-requests-2.25.1-8.el9.noarch 2024-02-05T04:50:36.591 INFO:teuthology.orchestra.run.smithi150.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-02-05T04:50:36.591 INFO:teuthology.orchestra.run.smithi150.stdout: python3-toml-0.10.2-6.el9.noarch 2024-02-05T04:50:36.591 INFO:teuthology.orchestra.run.smithi150.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-02-05T04:50:36.591 INFO:teuthology.orchestra.run.smithi150.stdout: python3-webob-1.8.7-6.el9.noarch 2024-02-05T04:50:36.591 INFO:teuthology.orchestra.run.smithi150.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-02-05T04:50:36.591 INFO:teuthology.orchestra.run.smithi150.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-02-05T04:50:36.591 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:50:36.591 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-02-05T04:50:36.664 INFO:teuthology.orchestra.run.smithi091.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.3 MB/s | 228 kB 00:00 2024-02-05T04:50:36.698 INFO:teuthology.orchestra.run.smithi062.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.4 MB/s | 228 kB 00:00 2024-02-05T04:50:36.729 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:37 ago on Mon 05 Feb 2024 04:48:59 AM UTC. 2024-02-05T04:50:36.747 INFO:teuthology.orchestra.run.smithi091.stdout:(2/6): ceph-mgr-cephadm-19.0.0-658.gd4ed20cf.el 528 kB/s | 133 kB 00:00 2024-02-05T04:50:36.782 INFO:teuthology.orchestra.run.smithi062.stdout:(2/6): ceph-mgr-cephadm-19.0.0-658.gd4ed20cf.el 541 kB/s | 133 kB 00:00 2024-02-05T04:50:36.839 INFO:teuthology.orchestra.run.smithi091.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 17 MB/s | 5.8 MB 00:00 2024-02-05T04:50:36.843 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-base 2024-02-05T04:50:36.857 INFO:teuthology.orchestra.run.smithi062.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 18 MB/s | 5.8 MB 00:00 2024-02-05T04:50:36.887 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-02-05T04:50:36.887 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-02-05T04:50:36.887 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repo Size 2024-02-05T04:50:36.888 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-02-05T04:50:36.888 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-02-05T04:50:36.888 INFO:teuthology.orchestra.run.smithi196.stdout: librados-devel x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 127 k 2024-02-05T04:50:36.888 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:50:36.888 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-02-05T04:50:36.888 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-02-05T04:50:36.888 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-02-05T04:50:36.888 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:50:36.888 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 127 k 2024-02-05T04:50:36.888 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 456 k 2024-02-05T04:50:36.888 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-02-05T04:50:36.957 INFO:teuthology.orchestra.run.smithi062.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 334 kB/s | 58 kB 00:00 2024-02-05T04:50:36.983 INFO:teuthology.orchestra.run.smithi062.stdout:(5/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 1.9 MB/s | 548 kB 00:00 2024-02-05T04:50:37.008 INFO:teuthology.orchestra.run.smithi062.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 408 kB/s | 51 kB 00:00 2024-02-05T04:50:37.012 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:50:37.012 INFO:teuthology.orchestra.run.smithi062.stdout:Total 12 MB/s | 6.8 MB 00:00 2024-02-05T04:50:37.012 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2024-02-05T04:50:37.023 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2024-02-05T04:50:37.023 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2024-02-05T04:50:37.115 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2024-02-05T04:50:37.116 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2024-02-05T04:50:37.116 INFO:teuthology.orchestra.run.smithi196.stdout:librados-devel-19.0.0-658.gd4ed20cf.el9.x86_64. 558 kB/s | 127 kB 00:00 2024-02-05T04:50:37.116 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:50:37.116 INFO:teuthology.orchestra.run.smithi196.stdout:Total 555 kB/s | 127 kB 00:00 2024-02-05T04:50:37.117 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-02-05T04:50:37.122 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-02-05T04:50:37.122 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-02-05T04:50:37.177 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-02-05T04:50:37.177 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-02-05T04:50:37.206 INFO:teuthology.orchestra.run.smithi091.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 132 kB/s | 58 kB 00:00 2024-02-05T04:50:37.257 INFO:teuthology.orchestra.run.smithi091.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 122 kB/s | 51 kB 00:00 2024-02-05T04:50:37.323 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-02-05T04:50:37.357 INFO:teuthology.orchestra.run.smithi091.stdout:(6/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 793 kB/s | 548 kB 00:00 2024-02-05T04:50:37.360 INFO:teuthology.orchestra.run.smithi091.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:50:37.361 INFO:teuthology.orchestra.run.smithi091.stdout:Total 7.2 MB/s | 6.8 MB 00:00 2024-02-05T04:50:37.361 INFO:teuthology.orchestra.run.smithi091.stdout:Running transaction check 2024-02-05T04:50:37.370 INFO:teuthology.orchestra.run.smithi091.stdout:Transaction check succeeded. 2024-02-05T04:50:37.370 INFO:teuthology.orchestra.run.smithi091.stdout:Running transaction test 2024-02-05T04:50:37.417 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:00:58 ago on Mon 05 Feb 2024 04:49:39 AM UTC. 2024-02-05T04:50:37.418 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2024-02-05T04:50:37.439 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : librados-devel-2:19.0.0-658.gd4ed20cf.el9.x86_64 1/1 2024-02-05T04:50:37.459 INFO:teuthology.orchestra.run.smithi091.stdout:Transaction test succeeded. 2024-02-05T04:50:37.459 INFO:teuthology.orchestra.run.smithi091.stdout:Running transaction 2024-02-05T04:50:37.523 INFO:teuthology.orchestra.run.smithi150.stdout:Package ceph-base-2:19.0.0-658.gd4ed20cf.el9.x86_64 is already installed. 2024-02-05T04:50:37.574 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-02-05T04:50:37.575 INFO:teuthology.orchestra.run.smithi150.stdout:Nothing to do. 2024-02-05T04:50:37.575 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-02-05T04:50:37.616 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-02-05T04:50:37.641 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install cephadm 2024-02-05T04:50:37.669 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2024-02-05T04:50:37.761 INFO:teuthology.orchestra.run.smithi091.stdout: Preparing : 1/1 2024-02-05T04:50:37.897 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librados-devel-2:19.0.0-658.gd4ed20cf.el9.x86_64 1/1 2024-02-05T04:50:37.919 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-02-05T04:50:37.983 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2024-02-05T04:50:38.212 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librados-devel-2:19.0.0-658.gd4ed20cf.el9.x86_64 1/1 2024-02-05T04:50:38.212 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:50:38.212 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-02-05T04:50:38.212 INFO:teuthology.orchestra.run.smithi196.stdout: librados-devel-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:50:38.212 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:50:38.212 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-02-05T04:50:38.222 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-02-05T04:50:38.223 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:00:59 ago on Mon 05 Feb 2024 04:49:39 AM UTC. 2024-02-05T04:50:38.307 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-02-05T04:50:38.376 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2024-02-05T04:50:38.387 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-02-05T04:50:38.388 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-02-05T04:50:38.388 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2024-02-05T04:50:38.388 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-02-05T04:50:38.388 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2024-02-05T04:50:38.388 INFO:teuthology.orchestra.run.smithi150.stdout: cephadm noarch 2:19.0.0-658.gd4ed20cf.el9 ceph-noarch 742 k 2024-02-05T04:50:38.389 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:50:38.389 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-02-05T04:50:38.389 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-02-05T04:50:38.389 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2024-02-05T04:50:38.389 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:50:38.389 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 742 k 2024-02-05T04:50:38.389 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 748 k 2024-02-05T04:50:38.389 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-02-05T04:50:38.406 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-658.gd4ed20cf.el9.noarch 6/6 2024-02-05T04:50:38.438 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install libcephfs2 2024-02-05T04:50:38.513 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-02-05T04:50:38.603 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-02-05T04:50:38.663 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2024-02-05T04:50:38.687 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-658.gd4ed20cf.el9.noarch 6/6 2024-02-05T04:50:38.890 INFO:teuthology.orchestra.run.smithi150.stdout:cephadm-19.0.0-658.gd4ed20cf.el9.noarch.rpm 1.4 MB/s | 742 kB 00:00 2024-02-05T04:50:38.891 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:50:38.892 INFO:teuthology.orchestra.run.smithi150.stdout:Total 1.4 MB/s | 742 kB 00:00 2024-02-05T04:50:38.892 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-02-05T04:50:38.897 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-02-05T04:50:38.897 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-02-05T04:50:38.903 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-02-05T04:50:38.903 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-02-05T04:50:38.979 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-02-05T04:50:39.019 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:40 ago on Mon 05 Feb 2024 04:48:59 AM UTC. 2024-02-05T04:50:39.127 INFO:teuthology.orchestra.run.smithi196.stdout:Package libcephfs2-2:19.0.0-658.gd4ed20cf.el9.x86_64 is already installed. 2024-02-05T04:50:39.167 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-658.gd4ed20cf.el9.noarch 6/6 2024-02-05T04:50:39.167 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-658.gd4ed20cf.el9.noarch 1/6 2024-02-05T04:50:39.167 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-02-05T04:50:39.167 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2024-02-05T04:50:39.167 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2024-02-05T04:50:39.167 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-02-05T04:50:39.178 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-02-05T04:50:39.180 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-02-05T04:50:39.180 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-02-05T04:50:39.259 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install libcephfs-devel 2024-02-05T04:50:39.386 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: cephadm-2:19.0.0-658.gd4ed20cf.el9.noarch 1/1 2024-02-05T04:50:39.461 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-02-05T04:50:39.461 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:50:39.461 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2024-02-05T04:50:39.461 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-cephadm-2:19.0.0-658.gd4ed20cf.el9.noarch 2024-02-05T04:50:39.462 INFO:teuthology.orchestra.run.smithi062.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2024-02-05T04:50:39.462 INFO:teuthology.orchestra.run.smithi062.stdout: python3-babel-2.9.1-2.el9.noarch 2024-02-05T04:50:39.462 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jinja2-2.11.3-4.el9.noarch 2024-02-05T04:50:39.462 INFO:teuthology.orchestra.run.smithi062.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-02-05T04:50:39.462 INFO:teuthology.orchestra.run.smithi062.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-02-05T04:50:39.462 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:50:39.462 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-02-05T04:50:39.471 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : cephadm-2:19.0.0-658.gd4ed20cf.el9.noarch 1/1 2024-02-05T04:50:39.500 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-658.gd4ed20cf.el9.noarch 6/6 2024-02-05T04:50:39.500 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-658.gd4ed20cf.el9.noarch 1/6 2024-02-05T04:50:39.500 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-02-05T04:50:39.500 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2024-02-05T04:50:39.500 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2024-02-05T04:50:39.501 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-02-05T04:50:39.734 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install ceph-fuse 2024-02-05T04:50:39.812 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-02-05T04:50:39.812 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:50:39.813 INFO:teuthology.orchestra.run.smithi091.stdout:Installed: 2024-02-05T04:50:39.813 INFO:teuthology.orchestra.run.smithi091.stdout: ceph-mgr-cephadm-2:19.0.0-658.gd4ed20cf.el9.noarch 2024-02-05T04:50:39.813 INFO:teuthology.orchestra.run.smithi091.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2024-02-05T04:50:39.813 INFO:teuthology.orchestra.run.smithi091.stdout: python3-babel-2.9.1-2.el9.noarch 2024-02-05T04:50:39.813 INFO:teuthology.orchestra.run.smithi091.stdout: python3-jinja2-2.11.3-4.el9.noarch 2024-02-05T04:50:39.813 INFO:teuthology.orchestra.run.smithi091.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-02-05T04:50:39.813 INFO:teuthology.orchestra.run.smithi091.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-02-05T04:50:39.813 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:50:39.813 INFO:teuthology.orchestra.run.smithi091.stdout:Complete! 2024-02-05T04:50:39.826 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:40 ago on Mon 05 Feb 2024 04:48:59 AM UTC. 2024-02-05T04:50:39.916 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: cephadm-2:19.0.0-658.gd4ed20cf.el9.noarch 1/1 2024-02-05T04:50:39.981 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-02-05T04:50:39.981 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-02-05T04:50:39.981 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repo Size 2024-02-05T04:50:39.981 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-02-05T04:50:39.981 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-02-05T04:50:39.981 INFO:teuthology.orchestra.run.smithi196.stdout: libcephfs-devel x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 31 k 2024-02-05T04:50:39.982 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:50:39.982 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-02-05T04:50:39.982 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-02-05T04:50:39.982 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-02-05T04:50:39.982 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:50:39.982 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 31 k 2024-02-05T04:50:39.982 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 140 k 2024-02-05T04:50:39.983 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-02-05T04:50:40.082 DEBUG:teuthology.orchestra.run.smithi091:> sudo yum -y install ceph-fuse 2024-02-05T04:50:40.221 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : cephadm-2:19.0.0-658.gd4ed20cf.el9.noarch 1/1 2024-02-05T04:50:40.221 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:50:40.221 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2024-02-05T04:50:40.221 INFO:teuthology.orchestra.run.smithi150.stdout: cephadm-2:19.0.0-658.gd4ed20cf.el9.noarch 2024-02-05T04:50:40.221 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:50:40.221 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-02-05T04:50:40.229 INFO:teuthology.orchestra.run.smithi196.stdout:libcephfs-devel-19.0.0-658.gd4ed20cf.el9.x86_64 126 kB/s | 31 kB 00:00 2024-02-05T04:50:40.229 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:50:40.230 INFO:teuthology.orchestra.run.smithi196.stdout:Total 126 kB/s | 31 kB 00:00 2024-02-05T04:50:40.230 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-02-05T04:50:40.233 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-02-05T04:50:40.233 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-02-05T04:50:40.267 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-02-05T04:50:40.268 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-02-05T04:50:40.308 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:01:33 ago on Mon 05 Feb 2024 04:49:07 AM UTC. 2024-02-05T04:50:40.354 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-02-05T04:50:40.387 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-immutable-object-cache 2024-02-05T04:50:40.468 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libcephfs-devel-2:19.0.0-658.gd4ed20cf.el9.x86_64 1/1 2024-02-05T04:50:40.482 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-02-05T04:50:40.483 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-02-05T04:50:40.483 INFO:teuthology.orchestra.run.smithi062.stdout: Package Arch Version Repository Size 2024-02-05T04:50:40.483 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-02-05T04:50:40.483 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2024-02-05T04:50:40.483 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-fuse x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 887 k 2024-02-05T04:50:40.483 INFO:teuthology.orchestra.run.smithi062.stdout:Installing dependencies: 2024-02-05T04:50:40.483 INFO:teuthology.orchestra.run.smithi062.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-02-05T04:50:40.484 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:50:40.484 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2024-02-05T04:50:40.484 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-02-05T04:50:40.484 INFO:teuthology.orchestra.run.smithi062.stdout:Install 2 Packages 2024-02-05T04:50:40.484 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:50:40.484 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 966 k 2024-02-05T04:50:40.484 INFO:teuthology.orchestra.run.smithi062.stdout:Installed size: 2.8 M 2024-02-05T04:50:40.484 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2024-02-05T04:50:40.618 INFO:teuthology.orchestra.run.smithi062.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 588 kB/s | 79 kB 00:00 2024-02-05T04:50:40.643 INFO:teuthology.orchestra.run.smithi091.stdout:Last metadata expiration check: 0:01:31 ago on Mon 05 Feb 2024 04:49:09 AM UTC. 2024-02-05T04:50:40.803 INFO:teuthology.orchestra.run.smithi091.stdout:Dependencies resolved. 2024-02-05T04:50:40.805 INFO:teuthology.orchestra.run.smithi091.stdout:================================================================================ 2024-02-05T04:50:40.805 INFO:teuthology.orchestra.run.smithi091.stdout: Package Arch Version Repository Size 2024-02-05T04:50:40.805 INFO:teuthology.orchestra.run.smithi091.stdout:================================================================================ 2024-02-05T04:50:40.805 INFO:teuthology.orchestra.run.smithi091.stdout:Installing: 2024-02-05T04:50:40.805 INFO:teuthology.orchestra.run.smithi091.stdout: ceph-fuse x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 887 k 2024-02-05T04:50:40.805 INFO:teuthology.orchestra.run.smithi091.stdout:Installing dependencies: 2024-02-05T04:50:40.805 INFO:teuthology.orchestra.run.smithi091.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-02-05T04:50:40.805 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:50:40.805 INFO:teuthology.orchestra.run.smithi091.stdout:Transaction Summary 2024-02-05T04:50:40.805 INFO:teuthology.orchestra.run.smithi091.stdout:================================================================================ 2024-02-05T04:50:40.805 INFO:teuthology.orchestra.run.smithi091.stdout:Install 2 Packages 2024-02-05T04:50:40.805 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:50:40.805 INFO:teuthology.orchestra.run.smithi091.stdout:Total download size: 966 k 2024-02-05T04:50:40.805 INFO:teuthology.orchestra.run.smithi091.stdout:Installed size: 2.8 M 2024-02-05T04:50:40.806 INFO:teuthology.orchestra.run.smithi091.stdout:Downloading Packages: 2024-02-05T04:50:40.835 INFO:teuthology.orchestra.run.smithi062.stdout:(2/2): ceph-fuse-19.0.0-658.gd4ed20cf.el9.x86_6 2.5 MB/s | 887 kB 00:00 2024-02-05T04:50:40.836 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:50:40.836 INFO:teuthology.orchestra.run.smithi062.stdout:Total 2.7 MB/s | 966 kB 00:00 2024-02-05T04:50:40.836 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2024-02-05T04:50:40.857 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2024-02-05T04:50:40.858 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2024-02-05T04:50:40.881 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-658.gd4ed20cf.el9.x86_64 1/1 2024-02-05T04:50:40.937 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2024-02-05T04:50:40.937 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2024-02-05T04:50:40.941 INFO:teuthology.orchestra.run.smithi091.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 579 kB/s | 79 kB 00:00 2024-02-05T04:50:40.969 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:01 ago on Mon 05 Feb 2024 04:49:39 AM UTC. 2024-02-05T04:50:41.128 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-02-05T04:50:41.128 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-02-05T04:50:41.129 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repo Size 2024-02-05T04:50:41.129 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-02-05T04:50:41.129 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2024-02-05T04:50:41.129 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 145 k 2024-02-05T04:50:41.129 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:50:41.129 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-02-05T04:50:41.129 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-02-05T04:50:41.129 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2024-02-05T04:50:41.129 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:50:41.129 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 145 k 2024-02-05T04:50:41.129 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 436 k 2024-02-05T04:50:41.129 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-02-05T04:50:41.141 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2024-02-05T04:50:41.153 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libcephfs-devel-2:19.0.0-658.gd4ed20cf.el9.x86_64 1/1 2024-02-05T04:50:41.153 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:50:41.153 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-02-05T04:50:41.153 INFO:teuthology.orchestra.run.smithi196.stdout: libcephfs-devel-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:50:41.153 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:50:41.153 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-02-05T04:50:41.273 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-02-05T04:50:41.339 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install librados2 2024-02-05T04:50:41.345 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-fuse-2:19.0.0-658.gd4ed20cf.el9.x86_64 2/2 2024-02-05T04:50:41.363 INFO:teuthology.orchestra.run.smithi150.stdout:ceph-immutable-object-cache-19.0.0-658.gd4ed20c 622 kB/s | 145 kB 00:00 2024-02-05T04:50:41.363 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:50:41.364 INFO:teuthology.orchestra.run.smithi150.stdout:Total 618 kB/s | 145 kB 00:00 2024-02-05T04:50:41.364 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-02-05T04:50:41.371 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-02-05T04:50:41.372 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-02-05T04:50:41.424 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-02-05T04:50:41.424 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-02-05T04:50:41.569 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-02-05T04:50:41.600 INFO:teuthology.orchestra.run.smithi091.stdout:(2/2): ceph-fuse-19.0.0-658.gd4ed20cf.el9.x86_6 1.1 MB/s | 887 kB 00:00 2024-02-05T04:50:41.601 INFO:teuthology.orchestra.run.smithi091.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:50:41.601 INFO:teuthology.orchestra.run.smithi091.stdout:Total 1.2 MB/s | 966 kB 00:00 2024-02-05T04:50:41.601 INFO:teuthology.orchestra.run.smithi091.stdout:Running transaction check 2024-02-05T04:50:41.614 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-658.gd4ed20cf.e 1/1 2024-02-05T04:50:41.621 INFO:teuthology.orchestra.run.smithi091.stdout:Transaction check succeeded. 2024-02-05T04:50:41.621 INFO:teuthology.orchestra.run.smithi091.stdout:Running transaction test 2024-02-05T04:50:41.639 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-658.gd4ed20cf.e 1/1 2024-02-05T04:50:41.639 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T04:50:41.639 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-02-05T04:50:41.639 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:50:41.697 INFO:teuthology.orchestra.run.smithi091.stdout:Transaction test succeeded. 2024-02-05T04:50:41.697 INFO:teuthology.orchestra.run.smithi091.stdout:Running transaction 2024-02-05T04:50:41.899 INFO:teuthology.orchestra.run.smithi091.stdout: Preparing : 1/1 2024-02-05T04:50:41.903 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:42 ago on Mon 05 Feb 2024 04:48:59 AM UTC. 2024-02-05T04:50:42.010 INFO:teuthology.orchestra.run.smithi196.stdout:Package librados2-2:19.0.0-658.gd4ed20cf.el9.x86_64 is already installed. 2024-02-05T04:50:42.034 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-02-05T04:50:42.064 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-02-05T04:50:42.065 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-02-05T04:50:42.065 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-02-05T04:50:42.099 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : ceph-fuse-2:19.0.0-658.gd4ed20cf.el9.x86_64 2/2 2024-02-05T04:50:42.110 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-fuse-2:19.0.0-658.gd4ed20cf.el9.x86_64 2/2 2024-02-05T04:50:42.110 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-fuse-2:19.0.0-658.gd4ed20cf.el9.x86_64 1/2 2024-02-05T04:50:42.129 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install librbd1 2024-02-05T04:50:42.444 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-02-05T04:50:42.444 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:50:42.444 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2024-02-05T04:50:42.444 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-fuse-2:19.0.0-658.gd4ed20cf.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-02-05T04:50:42.444 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:50:42.444 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-02-05T04:50:42.671 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-658.gd4ed20cf.e 1/1 2024-02-05T04:50:42.671 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:50:42.671 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2024-02-05T04:50:42.671 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-immutable-object-cache-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:50:42.671 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:50:42.671 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-02-05T04:50:42.680 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install ceph-volume 2024-02-05T04:50:42.694 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:43 ago on Mon 05 Feb 2024 04:48:59 AM UTC. 2024-02-05T04:50:42.800 INFO:teuthology.orchestra.run.smithi196.stdout:Package librbd1-2:19.0.0-658.gd4ed20cf.el9.x86_64 is already installed. 2024-02-05T04:50:42.803 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: ceph-fuse-2:19.0.0-658.gd4ed20cf.el9.x86_64 2/2 2024-02-05T04:50:42.803 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : ceph-fuse-2:19.0.0-658.gd4ed20cf.el9.x86_64 1/2 2024-02-05T04:50:42.853 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-02-05T04:50:42.854 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-02-05T04:50:42.854 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-02-05T04:50:42.912 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-mgr 2024-02-05T04:50:42.915 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install python3-rados 2024-02-05T04:50:43.128 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-02-05T04:50:43.128 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:50:43.128 INFO:teuthology.orchestra.run.smithi091.stdout:Installed: 2024-02-05T04:50:43.128 INFO:teuthology.orchestra.run.smithi091.stdout: ceph-fuse-2:19.0.0-658.gd4ed20cf.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-02-05T04:50:43.128 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:50:43.128 INFO:teuthology.orchestra.run.smithi091.stdout:Complete! 2024-02-05T04:50:43.266 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:01:36 ago on Mon 05 Feb 2024 04:49:07 AM UTC. 2024-02-05T04:50:43.398 DEBUG:teuthology.orchestra.run.smithi091:> sudo yum -y install ceph-volume 2024-02-05T04:50:43.430 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-02-05T04:50:43.430 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-02-05T04:50:43.431 INFO:teuthology.orchestra.run.smithi062.stdout: Package Arch Version Repository Size 2024-02-05T04:50:43.431 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-02-05T04:50:43.431 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2024-02-05T04:50:43.431 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-volume noarch 2:19.0.0-658.gd4ed20cf.el9 ceph-noarch 262 k 2024-02-05T04:50:43.431 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:50:43.431 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2024-02-05T04:50:43.431 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-02-05T04:50:43.431 INFO:teuthology.orchestra.run.smithi062.stdout:Install 1 Package 2024-02-05T04:50:43.431 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:50:43.431 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 262 k 2024-02-05T04:50:43.431 INFO:teuthology.orchestra.run.smithi062.stdout:Installed size: 1.2 M 2024-02-05T04:50:43.431 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2024-02-05T04:50:43.489 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:44 ago on Mon 05 Feb 2024 04:48:59 AM UTC. 2024-02-05T04:50:43.500 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:04 ago on Mon 05 Feb 2024 04:49:39 AM UTC. 2024-02-05T04:50:43.593 INFO:teuthology.orchestra.run.smithi196.stdout:Package python3-rados-2:19.0.0-658.gd4ed20cf.el9.x86_64 is already installed. 2024-02-05T04:50:43.604 INFO:teuthology.orchestra.run.smithi150.stdout:Package ceph-mgr-2:19.0.0-658.gd4ed20cf.el9.x86_64 is already installed. 2024-02-05T04:50:43.645 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-02-05T04:50:43.646 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-02-05T04:50:43.646 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-02-05T04:50:43.655 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-02-05T04:50:43.656 INFO:teuthology.orchestra.run.smithi150.stdout:Nothing to do. 2024-02-05T04:50:43.656 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-02-05T04:50:43.689 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-volume-19.0.0-658.gd4ed20cf.el9.noarch.rpm 1.0 MB/s | 262 kB 00:00 2024-02-05T04:50:43.690 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:50:43.690 INFO:teuthology.orchestra.run.smithi062.stdout:Total 1.0 MB/s | 262 kB 00:00 2024-02-05T04:50:43.691 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2024-02-05T04:50:43.699 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2024-02-05T04:50:43.699 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2024-02-05T04:50:43.713 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install python3-rgw 2024-02-05T04:50:43.723 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-mgr-dashboard 2024-02-05T04:50:43.731 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2024-02-05T04:50:43.732 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2024-02-05T04:50:43.960 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2024-02-05T04:50:43.982 INFO:teuthology.orchestra.run.smithi091.stdout:Last metadata expiration check: 0:01:34 ago on Mon 05 Feb 2024 04:49:09 AM UTC. 2024-02-05T04:50:44.016 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-volume-2:19.0.0-658.gd4ed20cf.el9.noarch 1/1 2024-02-05T04:50:44.031 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-volume-2:19.0.0-658.gd4ed20cf.el9.noarch 1/1 2024-02-05T04:50:44.031 INFO:teuthology.orchestra.run.smithi062.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T04:50:44.032 INFO:teuthology.orchestra.run.smithi062.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-02-05T04:50:44.032 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:50:44.137 INFO:teuthology.orchestra.run.smithi091.stdout:Dependencies resolved. 2024-02-05T04:50:44.138 INFO:teuthology.orchestra.run.smithi091.stdout:================================================================================ 2024-02-05T04:50:44.138 INFO:teuthology.orchestra.run.smithi091.stdout: Package Arch Version Repository Size 2024-02-05T04:50:44.138 INFO:teuthology.orchestra.run.smithi091.stdout:================================================================================ 2024-02-05T04:50:44.138 INFO:teuthology.orchestra.run.smithi091.stdout:Installing: 2024-02-05T04:50:44.138 INFO:teuthology.orchestra.run.smithi091.stdout: ceph-volume noarch 2:19.0.0-658.gd4ed20cf.el9 ceph-noarch 262 k 2024-02-05T04:50:44.138 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:50:44.138 INFO:teuthology.orchestra.run.smithi091.stdout:Transaction Summary 2024-02-05T04:50:44.138 INFO:teuthology.orchestra.run.smithi091.stdout:================================================================================ 2024-02-05T04:50:44.138 INFO:teuthology.orchestra.run.smithi091.stdout:Install 1 Package 2024-02-05T04:50:44.138 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:50:44.139 INFO:teuthology.orchestra.run.smithi091.stdout:Total download size: 262 k 2024-02-05T04:50:44.139 INFO:teuthology.orchestra.run.smithi091.stdout:Installed size: 1.2 M 2024-02-05T04:50:44.139 INFO:teuthology.orchestra.run.smithi091.stdout:Downloading Packages: 2024-02-05T04:50:44.285 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:45 ago on Mon 05 Feb 2024 04:48:59 AM UTC. 2024-02-05T04:50:44.289 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:05 ago on Mon 05 Feb 2024 04:49:39 AM UTC. 2024-02-05T04:50:44.354 INFO:teuthology.orchestra.run.smithi091.stdout:ceph-volume-19.0.0-658.gd4ed20cf.el9.noarch.rpm 1.2 MB/s | 262 kB 00:00 2024-02-05T04:50:44.354 INFO:teuthology.orchestra.run.smithi091.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:50:44.355 INFO:teuthology.orchestra.run.smithi091.stdout:Total 1.2 MB/s | 262 kB 00:00 2024-02-05T04:50:44.355 INFO:teuthology.orchestra.run.smithi091.stdout:Running transaction check 2024-02-05T04:50:44.363 INFO:teuthology.orchestra.run.smithi091.stdout:Transaction check succeeded. 2024-02-05T04:50:44.363 INFO:teuthology.orchestra.run.smithi091.stdout:Running transaction test 2024-02-05T04:50:44.389 INFO:teuthology.orchestra.run.smithi196.stdout:Package python3-rgw-2:19.0.0-658.gd4ed20cf.el9.x86_64 is already installed. 2024-02-05T04:50:44.401 INFO:teuthology.orchestra.run.smithi091.stdout:Transaction test succeeded. 2024-02-05T04:50:44.401 INFO:teuthology.orchestra.run.smithi091.stdout:Running transaction 2024-02-05T04:50:44.442 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-02-05T04:50:44.443 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-02-05T04:50:44.443 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-02-05T04:50:44.447 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-02-05T04:50:44.448 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-02-05T04:50:44.448 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2024-02-05T04:50:44.448 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-02-05T04:50:44.449 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2024-02-05T04:50:44.449 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-dashboard noarch 2:19.0.0-658.gd4ed20cf.el9 ceph-noarch 1.7 M 2024-02-05T04:50:44.449 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2024-02-05T04:50:44.449 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-grafana-dashboards noarch 2:19.0.0-658.gd4ed20cf.el9 ceph-noarch 28 k 2024-02-05T04:50:44.449 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-prometheus-alerts noarch 2:19.0.0-658.gd4ed20cf.el9 ceph-noarch 15 k 2024-02-05T04:50:44.449 INFO:teuthology.orchestra.run.smithi150.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-02-05T04:50:44.449 INFO:teuthology.orchestra.run.smithi150.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-02-05T04:50:44.449 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:50:44.449 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-02-05T04:50:44.449 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-02-05T04:50:44.449 INFO:teuthology.orchestra.run.smithi150.stdout:Install 5 Packages 2024-02-05T04:50:44.449 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:50:44.449 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 2.0 M 2024-02-05T04:50:44.450 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 8.4 M 2024-02-05T04:50:44.450 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-02-05T04:50:44.513 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install python3-cephfs 2024-02-05T04:50:44.616 INFO:teuthology.orchestra.run.smithi091.stdout: Preparing : 1/1 2024-02-05T04:50:44.681 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : ceph-volume-2:19.0.0-658.gd4ed20cf.el9.noarch 1/1 2024-02-05T04:50:44.696 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: ceph-volume-2:19.0.0-658.gd4ed20cf.el9.noarch 1/1 2024-02-05T04:50:44.697 INFO:teuthology.orchestra.run.smithi091.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T04:50:44.697 INFO:teuthology.orchestra.run.smithi091.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-02-05T04:50:44.697 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:50:44.725 INFO:teuthology.orchestra.run.smithi150.stdout:(1/5): ceph-grafana-dashboards-19.0.0-658.gd4ed 141 kB/s | 28 kB 00:00 2024-02-05T04:50:44.750 INFO:teuthology.orchestra.run.smithi150.stdout:(2/5): ceph-prometheus-alerts-19.0.0-658.gd4ed2 67 kB/s | 15 kB 00:00 2024-02-05T04:50:45.084 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:46 ago on Mon 05 Feb 2024 04:48:59 AM UTC. 2024-02-05T04:50:45.101 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-volume-2:19.0.0-658.gd4ed20cf.el9.noarch 1/1 2024-02-05T04:50:45.101 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:50:45.101 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2024-02-05T04:50:45.101 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-volume-2:19.0.0-658.gd4ed20cf.el9.noarch 2024-02-05T04:50:45.101 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:50:45.101 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-02-05T04:50:45.109 INFO:teuthology.orchestra.run.smithi150.stdout:(3/5): python3-repoze-lru-0.7-16.el9.noarch.rpm 80 kB/s | 31 kB 00:00 2024-02-05T04:50:45.188 INFO:teuthology.orchestra.run.smithi196.stdout:Package python3-cephfs-2:19.0.0-658.gd4ed20cf.el9.x86_64 is already installed. 2024-02-05T04:50:45.239 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-02-05T04:50:45.240 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-02-05T04:50:45.240 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-02-05T04:50:45.312 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install python3-rbd 2024-02-05T04:50:45.334 INFO:teuthology.orchestra.run.smithi150.stdout:(4/5): python3-routes-2.5.1-5.el9.noarch.rpm 323 kB/s | 188 kB 00:00 2024-02-05T04:50:45.423 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install librados-devel 2024-02-05T04:50:45.501 INFO:teuthology.orchestra.run.smithi150.stdout:(5/5): ceph-mgr-dashboard-19.0.0-658.gd4ed20cf. 1.8 MB/s | 1.7 MB 00:00 2024-02-05T04:50:45.504 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:50:45.504 INFO:teuthology.orchestra.run.smithi150.stdout:Total 1.9 MB/s | 2.0 MB 00:01 2024-02-05T04:50:45.505 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-02-05T04:50:45.512 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-02-05T04:50:45.512 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-02-05T04:50:45.571 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-02-05T04:50:45.571 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-02-05T04:50:45.702 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : ceph-volume-2:19.0.0-658.gd4ed20cf.el9.noarch 1/1 2024-02-05T04:50:45.702 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:50:45.702 INFO:teuthology.orchestra.run.smithi091.stdout:Installed: 2024-02-05T04:50:45.703 INFO:teuthology.orchestra.run.smithi091.stdout: ceph-volume-2:19.0.0-658.gd4ed20cf.el9.noarch 2024-02-05T04:50:45.703 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:50:45.703 INFO:teuthology.orchestra.run.smithi091.stdout:Complete! 2024-02-05T04:50:45.818 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-02-05T04:50:45.888 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:46 ago on Mon 05 Feb 2024 04:48:59 AM UTC. 2024-02-05T04:50:45.899 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/5 2024-02-05T04:50:45.931 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/5 2024-02-05T04:50:45.961 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-658.gd4ed20cf.el9.no 3/5 2024-02-05T04:50:45.993 INFO:teuthology.orchestra.run.smithi196.stdout:Package python3-rbd-2:19.0.0-658.gd4ed20cf.el9.x86_64 is already installed. 2024-02-05T04:50:46.007 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:01:39 ago on Mon 05 Feb 2024 04:49:07 AM UTC. 2024-02-05T04:50:46.019 DEBUG:teuthology.orchestra.run.smithi091:> sudo yum -y install librados-devel 2024-02-05T04:50:46.044 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-02-05T04:50:46.046 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-02-05T04:50:46.046 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-02-05T04:50:46.113 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install rbd-fuse 2024-02-05T04:50:46.173 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-02-05T04:50:46.173 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-02-05T04:50:46.173 INFO:teuthology.orchestra.run.smithi062.stdout: Package Arch Version Repo Size 2024-02-05T04:50:46.174 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-02-05T04:50:46.174 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2024-02-05T04:50:46.174 INFO:teuthology.orchestra.run.smithi062.stdout: librados-devel x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 127 k 2024-02-05T04:50:46.174 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:50:46.174 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2024-02-05T04:50:46.174 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-02-05T04:50:46.174 INFO:teuthology.orchestra.run.smithi062.stdout:Install 1 Package 2024-02-05T04:50:46.174 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:50:46.175 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 127 k 2024-02-05T04:50:46.175 INFO:teuthology.orchestra.run.smithi062.stdout:Installed size: 456 k 2024-02-05T04:50:46.175 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2024-02-05T04:50:46.189 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-658.gd4ed20cf.el9.n 4/5 2024-02-05T04:50:46.229 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-658.gd4ed20cf.el9.noarch 5/5 2024-02-05T04:50:46.599 INFO:teuthology.orchestra.run.smithi091.stdout:Last metadata expiration check: 0:01:37 ago on Mon 05 Feb 2024 04:49:09 AM UTC. 2024-02-05T04:50:46.677 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:47 ago on Mon 05 Feb 2024 04:48:59 AM UTC. 2024-02-05T04:50:46.686 INFO:teuthology.orchestra.run.smithi062.stdout:librados-devel-19.0.0-658.gd4ed20cf.el9.x86_64. 248 kB/s | 127 kB 00:00 2024-02-05T04:50:46.686 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:50:46.687 INFO:teuthology.orchestra.run.smithi062.stdout:Total 247 kB/s | 127 kB 00:00 2024-02-05T04:50:46.687 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2024-02-05T04:50:46.693 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2024-02-05T04:50:46.693 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2024-02-05T04:50:46.751 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2024-02-05T04:50:46.751 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2024-02-05T04:50:46.755 INFO:teuthology.orchestra.run.smithi091.stdout:Dependencies resolved. 2024-02-05T04:50:46.756 INFO:teuthology.orchestra.run.smithi091.stdout:================================================================================ 2024-02-05T04:50:46.756 INFO:teuthology.orchestra.run.smithi091.stdout: Package Arch Version Repo Size 2024-02-05T04:50:46.756 INFO:teuthology.orchestra.run.smithi091.stdout:================================================================================ 2024-02-05T04:50:46.756 INFO:teuthology.orchestra.run.smithi091.stdout:Installing: 2024-02-05T04:50:46.756 INFO:teuthology.orchestra.run.smithi091.stdout: librados-devel x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 127 k 2024-02-05T04:50:46.756 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:50:46.756 INFO:teuthology.orchestra.run.smithi091.stdout:Transaction Summary 2024-02-05T04:50:46.756 INFO:teuthology.orchestra.run.smithi091.stdout:================================================================================ 2024-02-05T04:50:46.756 INFO:teuthology.orchestra.run.smithi091.stdout:Install 1 Package 2024-02-05T04:50:46.756 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:50:46.756 INFO:teuthology.orchestra.run.smithi091.stdout:Total download size: 127 k 2024-02-05T04:50:46.757 INFO:teuthology.orchestra.run.smithi091.stdout:Installed size: 456 k 2024-02-05T04:50:46.757 INFO:teuthology.orchestra.run.smithi091.stdout:Downloading Packages: 2024-02-05T04:50:46.784 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-658.gd4ed20cf.el9.noarch 5/5 2024-02-05T04:50:46.784 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-658.gd4ed20cf.el9.n 1/5 2024-02-05T04:50:46.784 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-658.gd4ed20cf.el9.noarch 2/5 2024-02-05T04:50:46.784 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-658.gd4ed20cf.el9.no 3/5 2024-02-05T04:50:46.784 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 4/5 2024-02-05T04:50:46.836 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-02-05T04:50:46.837 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-02-05T04:50:46.837 INFO:teuthology.orchestra.run.smithi196.stdout: Package Architecture Version Repository Size 2024-02-05T04:50:46.837 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-02-05T04:50:46.837 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-02-05T04:50:46.837 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-fuse x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 85 k 2024-02-05T04:50:46.837 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:50:46.837 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-02-05T04:50:46.838 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-02-05T04:50:46.838 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-02-05T04:50:46.838 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:50:46.838 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 85 k 2024-02-05T04:50:46.838 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 231 k 2024-02-05T04:50:46.838 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-02-05T04:50:46.903 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2024-02-05T04:50:47.003 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : librados-devel-2:19.0.0-658.gd4ed20cf.el9.x86_64 1/1 2024-02-05T04:50:47.049 INFO:teuthology.orchestra.run.smithi196.stdout:rbd-fuse-19.0.0-658.gd4ed20cf.el9.x86_64.rpm 405 kB/s | 85 kB 00:00 2024-02-05T04:50:47.049 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:50:47.050 INFO:teuthology.orchestra.run.smithi196.stdout:Total 402 kB/s | 85 kB 00:00 2024-02-05T04:50:47.050 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-02-05T04:50:47.056 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-02-05T04:50:47.057 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-02-05T04:50:47.067 INFO:teuthology.orchestra.run.smithi091.stdout:librados-devel-19.0.0-658.gd4ed20cf.el9.x86_64. 408 kB/s | 127 kB 00:00 2024-02-05T04:50:47.068 INFO:teuthology.orchestra.run.smithi091.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:50:47.068 INFO:teuthology.orchestra.run.smithi091.stdout:Total 406 kB/s | 127 kB 00:00 2024-02-05T04:50:47.068 INFO:teuthology.orchestra.run.smithi091.stdout:Running transaction check 2024-02-05T04:50:47.074 INFO:teuthology.orchestra.run.smithi091.stdout:Transaction check succeeded. 2024-02-05T04:50:47.074 INFO:teuthology.orchestra.run.smithi091.stdout:Running transaction test 2024-02-05T04:50:47.084 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 5/5 2024-02-05T04:50:47.085 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:50:47.085 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2024-02-05T04:50:47.085 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-grafana-dashboards-2:19.0.0-658.gd4ed20cf.el9.noarch 2024-02-05T04:50:47.085 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-dashboard-2:19.0.0-658.gd4ed20cf.el9.noarch 2024-02-05T04:50:47.085 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-prometheus-alerts-2:19.0.0-658.gd4ed20cf.el9.noarch 2024-02-05T04:50:47.085 INFO:teuthology.orchestra.run.smithi150.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-02-05T04:50:47.085 INFO:teuthology.orchestra.run.smithi150.stdout: python3-routes-2.5.1-5.el9.noarch 2024-02-05T04:50:47.085 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:50:47.085 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-02-05T04:50:47.111 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-02-05T04:50:47.112 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-02-05T04:50:47.129 INFO:teuthology.orchestra.run.smithi091.stdout:Transaction test succeeded. 2024-02-05T04:50:47.130 INFO:teuthology.orchestra.run.smithi091.stdout:Running transaction 2024-02-05T04:50:47.259 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-02-05T04:50:47.273 INFO:teuthology.orchestra.run.smithi091.stdout: Preparing : 1/1 2024-02-05T04:50:47.361 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-05T04:50:47.366 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : librados-devel-2:19.0.0-658.gd4ed20cf.el9.x86_64 1/1 2024-02-05T04:50:47.371 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : rbd-fuse-2:19.0.0-658.gd4ed20cf.el9.x86_64 1/1 2024-02-05T04:50:47.424 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: librados-devel-2:19.0.0-658.gd4ed20cf.el9.x86_64 1/1 2024-02-05T04:50:47.709 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : librados-devel-2:19.0.0-658.gd4ed20cf.el9.x86_64 1/1 2024-02-05T04:50:47.709 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:50:47.709 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2024-02-05T04:50:47.709 INFO:teuthology.orchestra.run.smithi062.stdout: librados-devel-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:50:47.709 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:50:47.709 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-02-05T04:50:47.759 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: rbd-fuse-2:19.0.0-658.gd4ed20cf.el9.x86_64 1/1 2024-02-05T04:50:47.827 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: librados-devel-2:19.0.0-658.gd4ed20cf.el9.x86_64 1/1 2024-02-05T04:50:47.924 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:08 ago on Mon 05 Feb 2024 04:49:39 AM UTC. 2024-02-05T04:50:47.952 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install libcephfs2 2024-02-05T04:50:48.061 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : rbd-fuse-2:19.0.0-658.gd4ed20cf.el9.x86_64 1/1 2024-02-05T04:50:48.061 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:50:48.061 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-02-05T04:50:48.061 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-fuse-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:50:48.061 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:50:48.061 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-02-05T04:50:48.082 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-02-05T04:50:48.083 INFO:teuthology.orchestra.run.smithi150.stdout:=========================================================================================== 2024-02-05T04:50:48.083 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2024-02-05T04:50:48.084 INFO:teuthology.orchestra.run.smithi150.stdout:=========================================================================================== 2024-02-05T04:50:48.084 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2024-02-05T04:50:48.084 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-658.gd4ed20cf.el9 ceph-noarch 7.4 M 2024-02-05T04:50:48.084 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2024-02-05T04:50:48.087 INFO:teuthology.orchestra.run.smithi150.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-02-05T04:50:48.087 INFO:teuthology.orchestra.run.smithi150.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-02-05T04:50:48.087 INFO:teuthology.orchestra.run.smithi150.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-02-05T04:50:48.087 INFO:teuthology.orchestra.run.smithi150.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-02-05T04:50:48.087 INFO:teuthology.orchestra.run.smithi150.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-02-05T04:50:48.087 INFO:teuthology.orchestra.run.smithi150.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-02-05T04:50:48.088 INFO:teuthology.orchestra.run.smithi150.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-02-05T04:50:48.088 INFO:teuthology.orchestra.run.smithi150.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-02-05T04:50:48.088 INFO:teuthology.orchestra.run.smithi150.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-02-05T04:50:48.088 INFO:teuthology.orchestra.run.smithi150.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-02-05T04:50:48.088 INFO:teuthology.orchestra.run.smithi150.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-02-05T04:50:48.088 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:50:48.088 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-02-05T04:50:48.088 INFO:teuthology.orchestra.run.smithi150.stdout:=========================================================================================== 2024-02-05T04:50:48.088 INFO:teuthology.orchestra.run.smithi150.stdout:Install 12 Packages 2024-02-05T04:50:48.088 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:50:48.089 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 37 M 2024-02-05T04:50:48.089 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 209 M 2024-02-05T04:50:48.089 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-02-05T04:50:48.142 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : librados-devel-2:19.0.0-658.gd4ed20cf.el9.x86_64 1/1 2024-02-05T04:50:48.142 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:50:48.142 INFO:teuthology.orchestra.run.smithi091.stdout:Installed: 2024-02-05T04:50:48.142 INFO:teuthology.orchestra.run.smithi091.stdout: librados-devel-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:50:48.142 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:50:48.142 INFO:teuthology.orchestra.run.smithi091.stdout:Complete! 2024-02-05T04:50:48.269 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install rbd-mirror 2024-02-05T04:50:48.340 INFO:teuthology.orchestra.run.smithi150.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 225 kB/s | 32 kB 00:00 2024-02-05T04:50:48.362 DEBUG:teuthology.orchestra.run.smithi091:> sudo yum -y install libcephfs2 2024-02-05T04:50:48.374 INFO:teuthology.orchestra.run.smithi150.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 539 kB/s | 17 kB 00:00 2024-02-05T04:50:48.416 INFO:teuthology.orchestra.run.smithi150.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 843 kB/s | 34 kB 00:00 2024-02-05T04:50:48.466 INFO:teuthology.orchestra.run.smithi150.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-02-05T04:50:48.537 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:01:41 ago on Mon 05 Feb 2024 04:49:07 AM UTC. 2024-02-05T04:50:48.646 INFO:teuthology.orchestra.run.smithi062.stdout:Package libcephfs2-2:19.0.0-658.gd4ed20cf.el9.x86_64 is already installed. 2024-02-05T04:50:48.691 INFO:teuthology.orchestra.run.smithi150.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 21 MB/s | 4.8 MB 00:00 2024-02-05T04:50:48.700 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-02-05T04:50:48.702 INFO:teuthology.orchestra.run.smithi062.stdout:Nothing to do. 2024-02-05T04:50:48.702 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-02-05T04:50:48.758 INFO:teuthology.orchestra.run.smithi150.stdout:(6/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 14 MB/s | 4.7 MB 00:00 2024-02-05T04:50:48.770 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install libcephfs-devel 2024-02-05T04:50:48.792 INFO:teuthology.orchestra.run.smithi150.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.1 MB/s | 217 kB 00:00 2024-02-05T04:50:48.860 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:49 ago on Mon 05 Feb 2024 04:48:59 AM UTC. 2024-02-05T04:50:48.934 INFO:teuthology.orchestra.run.smithi150.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 5.6 MB/s | 802 kB 00:00 2024-02-05T04:50:48.946 INFO:teuthology.orchestra.run.smithi091.stdout:Last metadata expiration check: 0:01:39 ago on Mon 05 Feb 2024 04:49:09 AM UTC. 2024-02-05T04:50:49.027 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-02-05T04:50:49.028 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-02-05T04:50:49.028 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repo Size 2024-02-05T04:50:49.028 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-02-05T04:50:49.028 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-02-05T04:50:49.028 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-mirror x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 3.1 M 2024-02-05T04:50:49.028 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:50:49.028 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-02-05T04:50:49.028 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-02-05T04:50:49.028 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-02-05T04:50:49.028 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:50:49.029 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 3.1 M 2024-02-05T04:50:49.029 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 13 M 2024-02-05T04:50:49.029 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-02-05T04:50:49.050 INFO:teuthology.orchestra.run.smithi091.stdout:Package libcephfs2-2:19.0.0-658.gd4ed20cf.el9.x86_64 is already installed. 2024-02-05T04:50:49.101 INFO:teuthology.orchestra.run.smithi150.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 46 MB/s | 16 MB 00:00 2024-02-05T04:50:49.102 INFO:teuthology.orchestra.run.smithi091.stdout:Dependencies resolved. 2024-02-05T04:50:49.103 INFO:teuthology.orchestra.run.smithi091.stdout:Nothing to do. 2024-02-05T04:50:49.103 INFO:teuthology.orchestra.run.smithi091.stdout:Complete! 2024-02-05T04:50:49.126 INFO:teuthology.orchestra.run.smithi150.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 194 kB 00:00 2024-02-05T04:50:49.174 DEBUG:teuthology.orchestra.run.smithi091:> sudo yum -y install libcephfs-devel 2024-02-05T04:50:49.210 INFO:teuthology.orchestra.run.smithi150.stdout:(11/12): ceph-mgr-diskprediction-local-19.0.0-6 7.3 MB/s | 7.4 MB 00:01 2024-02-05T04:50:49.348 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:01:42 ago on Mon 05 Feb 2024 04:49:07 AM UTC. 2024-02-05T04:50:49.463 INFO:teuthology.orchestra.run.smithi196.stdout:rbd-mirror-19.0.0-658.gd4ed20cf.el9.x86_64.rpm 7.1 MB/s | 3.1 MB 00:00 2024-02-05T04:50:49.463 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:50:49.464 INFO:teuthology.orchestra.run.smithi196.stdout:Total 7.1 MB/s | 3.1 MB 00:00 2024-02-05T04:50:49.464 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-02-05T04:50:49.473 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-02-05T04:50:49.473 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-02-05T04:50:49.510 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-02-05T04:50:49.511 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-02-05T04:50:49.511 INFO:teuthology.orchestra.run.smithi062.stdout: Package Arch Version Repo Size 2024-02-05T04:50:49.511 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-02-05T04:50:49.511 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2024-02-05T04:50:49.511 INFO:teuthology.orchestra.run.smithi062.stdout: libcephfs-devel x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 31 k 2024-02-05T04:50:49.511 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:50:49.511 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2024-02-05T04:50:49.511 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-02-05T04:50:49.511 INFO:teuthology.orchestra.run.smithi062.stdout:Install 1 Package 2024-02-05T04:50:49.511 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:50:49.512 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 31 k 2024-02-05T04:50:49.512 INFO:teuthology.orchestra.run.smithi062.stdout:Installed size: 140 k 2024-02-05T04:50:49.512 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2024-02-05T04:50:49.549 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-02-05T04:50:49.549 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-02-05T04:50:49.681 INFO:teuthology.orchestra.run.smithi062.stdout:libcephfs-devel-19.0.0-658.gd4ed20cf.el9.x86_64 184 kB/s | 31 kB 00:00 2024-02-05T04:50:49.682 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:50:49.682 INFO:teuthology.orchestra.run.smithi062.stdout:Total 183 kB/s | 31 kB 00:00 2024-02-05T04:50:49.682 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2024-02-05T04:50:49.685 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2024-02-05T04:50:49.685 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2024-02-05T04:50:49.720 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2024-02-05T04:50:49.721 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2024-02-05T04:50:49.744 INFO:teuthology.orchestra.run.smithi091.stdout:Last metadata expiration check: 0:01:40 ago on Mon 05 Feb 2024 04:49:09 AM UTC. 2024-02-05T04:50:49.816 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2024-02-05T04:50:49.900 INFO:teuthology.orchestra.run.smithi091.stdout:Dependencies resolved. 2024-02-05T04:50:49.901 INFO:teuthology.orchestra.run.smithi091.stdout:================================================================================ 2024-02-05T04:50:49.901 INFO:teuthology.orchestra.run.smithi091.stdout: Package Arch Version Repo Size 2024-02-05T04:50:49.901 INFO:teuthology.orchestra.run.smithi091.stdout:================================================================================ 2024-02-05T04:50:49.901 INFO:teuthology.orchestra.run.smithi091.stdout:Installing: 2024-02-05T04:50:49.901 INFO:teuthology.orchestra.run.smithi091.stdout: libcephfs-devel x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 31 k 2024-02-05T04:50:49.902 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:50:49.902 INFO:teuthology.orchestra.run.smithi091.stdout:Transaction Summary 2024-02-05T04:50:49.902 INFO:teuthology.orchestra.run.smithi091.stdout:================================================================================ 2024-02-05T04:50:49.902 INFO:teuthology.orchestra.run.smithi091.stdout:Install 1 Package 2024-02-05T04:50:49.902 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:50:49.902 INFO:teuthology.orchestra.run.smithi091.stdout:Total download size: 31 k 2024-02-05T04:50:49.902 INFO:teuthology.orchestra.run.smithi091.stdout:Installed size: 140 k 2024-02-05T04:50:49.902 INFO:teuthology.orchestra.run.smithi091.stdout:Downloading Packages: 2024-02-05T04:50:49.924 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libcephfs-devel-2:19.0.0-658.gd4ed20cf.el9.x86_64 1/1 2024-02-05T04:50:49.993 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-02-05T04:50:50.062 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : rbd-mirror-2:19.0.0-658.gd4ed20cf.el9.x86_64 1/1 2024-02-05T04:50:50.087 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: rbd-mirror-2:19.0.0-658.gd4ed20cf.el9.x86_64 1/1 2024-02-05T04:50:50.087 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T04:50:50.087 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-02-05T04:50:50.087 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-05T04:50:50.087 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-05T04:50:50.088 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:50:50.110 INFO:teuthology.orchestra.run.smithi150.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 243 kB/s | 245 kB 00:01 2024-02-05T04:50:50.113 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:50:50.113 INFO:teuthology.orchestra.run.smithi150.stdout:Total 18 MB/s | 37 MB 00:02 2024-02-05T04:50:50.126 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-02-05T04:50:50.151 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-02-05T04:50:50.151 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-02-05T04:50:50.153 INFO:teuthology.orchestra.run.smithi091.stdout:libcephfs-devel-19.0.0-658.gd4ed20cf.el9.x86_64 124 kB/s | 31 kB 00:00 2024-02-05T04:50:50.154 INFO:teuthology.orchestra.run.smithi091.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:50:50.154 INFO:teuthology.orchestra.run.smithi091.stdout:Total 123 kB/s | 31 kB 00:00 2024-02-05T04:50:50.154 INFO:teuthology.orchestra.run.smithi091.stdout:Running transaction check 2024-02-05T04:50:50.157 INFO:teuthology.orchestra.run.smithi091.stdout:Transaction check succeeded. 2024-02-05T04:50:50.158 INFO:teuthology.orchestra.run.smithi091.stdout:Running transaction test 2024-02-05T04:50:50.187 INFO:teuthology.orchestra.run.smithi091.stdout:Transaction test succeeded. 2024-02-05T04:50:50.187 INFO:teuthology.orchestra.run.smithi091.stdout:Running transaction 2024-02-05T04:50:50.271 INFO:teuthology.orchestra.run.smithi091.stdout: Preparing : 1/1 2024-02-05T04:50:50.294 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-658.gd4ed20cf.el9.x86_64 1/1 2024-02-05T04:50:50.384 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : libcephfs-devel-2:19.0.0-658.gd4ed20cf.el9.x86_64 1/1 2024-02-05T04:50:50.530 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-02-05T04:50:50.530 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-02-05T04:50:50.567 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libcephfs-devel-2:19.0.0-658.gd4ed20cf.el9.x86_64 1/1 2024-02-05T04:50:50.567 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:50:50.567 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2024-02-05T04:50:50.567 INFO:teuthology.orchestra.run.smithi062.stdout: libcephfs-devel-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:50:50.567 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:50:50.567 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-02-05T04:50:50.749 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install librados2 2024-02-05T04:50:50.757 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-658.gd4ed20cf.el9.x86_64 1/1 2024-02-05T04:50:51.019 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : libcephfs-devel-2:19.0.0-658.gd4ed20cf.el9.x86_64 1/1 2024-02-05T04:50:51.020 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:50:51.020 INFO:teuthology.orchestra.run.smithi091.stdout:Installed: 2024-02-05T04:50:51.020 INFO:teuthology.orchestra.run.smithi091.stdout: libcephfs-devel-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:50:51.020 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:50:51.020 INFO:teuthology.orchestra.run.smithi091.stdout:Complete! 2024-02-05T04:50:51.191 DEBUG:teuthology.orchestra.run.smithi091:> sudo yum -y install librados2 2024-02-05T04:50:51.229 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-02-05T04:50:51.250 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : rbd-mirror-2:19.0.0-658.gd4ed20cf.el9.x86_64 1/1 2024-02-05T04:50:51.250 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:50:51.250 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-02-05T04:50:51.250 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-mirror-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:50:51.250 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:50:51.250 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-02-05T04:50:51.310 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-02-05T04:50:51.319 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:01:44 ago on Mon 05 Feb 2024 04:49:07 AM UTC. 2024-02-05T04:50:51.339 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-02-05T04:50:51.428 INFO:teuthology.orchestra.run.smithi062.stdout:Package librados2-2:19.0.0-658.gd4ed20cf.el9.x86_64 is already installed. 2024-02-05T04:50:51.432 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-02-05T04:50:51.476 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-02-05T04:50:51.478 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install rbd-nbd 2024-02-05T04:50:51.484 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-02-05T04:50:51.485 INFO:teuthology.orchestra.run.smithi062.stdout:Nothing to do. 2024-02-05T04:50:51.486 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-02-05T04:50:51.561 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install librbd1 2024-02-05T04:50:51.665 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-02-05T04:50:51.692 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-02-05T04:50:51.754 INFO:teuthology.orchestra.run.smithi091.stdout:Last metadata expiration check: 0:01:42 ago on Mon 05 Feb 2024 04:49:09 AM UTC. 2024-02-05T04:50:51.769 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-02-05T04:50:51.860 INFO:teuthology.orchestra.run.smithi091.stdout:Package librados2-2:19.0.0-658.gd4ed20cf.el9.x86_64 is already installed. 2024-02-05T04:50:51.913 INFO:teuthology.orchestra.run.smithi091.stdout:Dependencies resolved. 2024-02-05T04:50:51.914 INFO:teuthology.orchestra.run.smithi091.stdout:Nothing to do. 2024-02-05T04:50:51.914 INFO:teuthology.orchestra.run.smithi091.stdout:Complete! 2024-02-05T04:50:51.980 DEBUG:teuthology.orchestra.run.smithi091:> sudo yum -y install librbd1 2024-02-05T04:50:52.050 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:53 ago on Mon 05 Feb 2024 04:48:59 AM UTC. 2024-02-05T04:50:52.135 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:01:45 ago on Mon 05 Feb 2024 04:49:07 AM UTC. 2024-02-05T04:50:52.210 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-02-05T04:50:52.210 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-02-05T04:50:52.210 INFO:teuthology.orchestra.run.smithi196.stdout: Package Architecture Version Repository Size 2024-02-05T04:50:52.211 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-02-05T04:50:52.211 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-02-05T04:50:52.211 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-nbd x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 170 k 2024-02-05T04:50:52.211 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:50:52.211 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-02-05T04:50:52.211 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-02-05T04:50:52.211 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-02-05T04:50:52.211 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:50:52.211 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 170 k 2024-02-05T04:50:52.211 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 490 k 2024-02-05T04:50:52.211 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-02-05T04:50:52.245 INFO:teuthology.orchestra.run.smithi062.stdout:Package librbd1-2:19.0.0-658.gd4ed20cf.el9.x86_64 is already installed. 2024-02-05T04:50:52.302 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-02-05T04:50:52.303 INFO:teuthology.orchestra.run.smithi062.stdout:Nothing to do. 2024-02-05T04:50:52.303 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-02-05T04:50:52.375 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install python3-rados 2024-02-05T04:50:52.431 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-02-05T04:50:52.537 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-02-05T04:50:52.547 INFO:teuthology.orchestra.run.smithi091.stdout:Last metadata expiration check: 0:01:43 ago on Mon 05 Feb 2024 04:49:09 AM UTC. 2024-02-05T04:50:52.557 INFO:teuthology.orchestra.run.smithi196.stdout:rbd-nbd-19.0.0-658.gd4ed20cf.el9.x86_64.rpm 494 kB/s | 170 kB 00:00 2024-02-05T04:50:52.557 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:50:52.557 INFO:teuthology.orchestra.run.smithi196.stdout:Total 492 kB/s | 170 kB 00:00 2024-02-05T04:50:52.558 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-02-05T04:50:52.565 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-02-05T04:50:52.565 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-02-05T04:50:52.619 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-02-05T04:50:52.620 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-02-05T04:50:52.650 INFO:teuthology.orchestra.run.smithi091.stdout:Package librbd1-2:19.0.0-658.gd4ed20cf.el9.x86_64 is already installed. 2024-02-05T04:50:52.703 INFO:teuthology.orchestra.run.smithi091.stdout:Dependencies resolved. 2024-02-05T04:50:52.704 INFO:teuthology.orchestra.run.smithi091.stdout:Nothing to do. 2024-02-05T04:50:52.705 INFO:teuthology.orchestra.run.smithi091.stdout:Complete! 2024-02-05T04:50:52.772 DEBUG:teuthology.orchestra.run.smithi091:> sudo yum -y install python3-rados 2024-02-05T04:50:52.775 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-02-05T04:50:52.879 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : rbd-nbd-2:19.0.0-658.gd4ed20cf.el9.x86_64 1/1 2024-02-05T04:50:52.957 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:01:45 ago on Mon 05 Feb 2024 04:49:07 AM UTC. 2024-02-05T04:50:53.066 INFO:teuthology.orchestra.run.smithi062.stdout:Package python3-rados-2:19.0.0-658.gd4ed20cf.el9.x86_64 is already installed. 2024-02-05T04:50:53.121 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-02-05T04:50:53.122 INFO:teuthology.orchestra.run.smithi062.stdout:Nothing to do. 2024-02-05T04:50:53.122 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-02-05T04:50:53.200 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install python3-rgw 2024-02-05T04:50:53.337 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: rbd-nbd-2:19.0.0-658.gd4ed20cf.el9.x86_64 1/1 2024-02-05T04:50:53.360 INFO:teuthology.orchestra.run.smithi091.stdout:Last metadata expiration check: 0:01:44 ago on Mon 05 Feb 2024 04:49:09 AM UTC. 2024-02-05T04:50:53.475 INFO:teuthology.orchestra.run.smithi091.stdout:Package python3-rados-2:19.0.0-658.gd4ed20cf.el9.x86_64 is already installed. 2024-02-05T04:50:53.528 INFO:teuthology.orchestra.run.smithi091.stdout:Dependencies resolved. 2024-02-05T04:50:53.529 INFO:teuthology.orchestra.run.smithi091.stdout:Nothing to do. 2024-02-05T04:50:53.529 INFO:teuthology.orchestra.run.smithi091.stdout:Complete! 2024-02-05T04:50:53.595 DEBUG:teuthology.orchestra.run.smithi091:> sudo yum -y install python3-rgw 2024-02-05T04:50:53.638 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : rbd-nbd-2:19.0.0-658.gd4ed20cf.el9.x86_64 1/1 2024-02-05T04:50:53.638 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:50:53.638 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-02-05T04:50:53.638 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-nbd-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:50:53.638 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:50:53.638 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-02-05T04:50:53.765 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-02-05T04:50:53.781 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:01:46 ago on Mon 05 Feb 2024 04:49:07 AM UTC. 2024-02-05T04:50:53.871 DEBUG:teuthology.parallel:result is None 2024-02-05T04:50:53.889 INFO:teuthology.orchestra.run.smithi062.stdout:Package python3-rgw-2:19.0.0-658.gd4ed20cf.el9.x86_64 is already installed. 2024-02-05T04:50:53.943 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-02-05T04:50:53.944 INFO:teuthology.orchestra.run.smithi062.stdout:Nothing to do. 2024-02-05T04:50:53.944 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-02-05T04:50:54.025 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install python3-cephfs 2024-02-05T04:50:54.167 INFO:teuthology.orchestra.run.smithi091.stdout:Last metadata expiration check: 0:01:45 ago on Mon 05 Feb 2024 04:49:09 AM UTC. 2024-02-05T04:50:54.271 INFO:teuthology.orchestra.run.smithi091.stdout:Package python3-rgw-2:19.0.0-658.gd4ed20cf.el9.x86_64 is already installed. 2024-02-05T04:50:54.324 INFO:teuthology.orchestra.run.smithi091.stdout:Dependencies resolved. 2024-02-05T04:50:54.325 INFO:teuthology.orchestra.run.smithi091.stdout:Nothing to do. 2024-02-05T04:50:54.325 INFO:teuthology.orchestra.run.smithi091.stdout:Complete! 2024-02-05T04:50:54.388 DEBUG:teuthology.orchestra.run.smithi091:> sudo yum -y install python3-cephfs 2024-02-05T04:50:54.606 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:01:47 ago on Mon 05 Feb 2024 04:49:07 AM UTC. 2024-02-05T04:50:54.714 INFO:teuthology.orchestra.run.smithi062.stdout:Package python3-cephfs-2:19.0.0-658.gd4ed20cf.el9.x86_64 is already installed. 2024-02-05T04:50:54.769 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-02-05T04:50:54.770 INFO:teuthology.orchestra.run.smithi062.stdout:Nothing to do. 2024-02-05T04:50:54.770 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-02-05T04:50:54.803 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-02-05T04:50:54.832 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-658.gd4ed20 12/12 2024-02-05T04:50:54.853 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install python3-rbd 2024-02-05T04:50:54.953 INFO:teuthology.orchestra.run.smithi091.stdout:Last metadata expiration check: 0:01:45 ago on Mon 05 Feb 2024 04:49:09 AM UTC. 2024-02-05T04:50:55.057 INFO:teuthology.orchestra.run.smithi091.stdout:Package python3-cephfs-2:19.0.0-658.gd4ed20cf.el9.x86_64 is already installed. 2024-02-05T04:50:55.109 INFO:teuthology.orchestra.run.smithi091.stdout:Dependencies resolved. 2024-02-05T04:50:55.109 INFO:teuthology.orchestra.run.smithi091.stdout:Nothing to do. 2024-02-05T04:50:55.109 INFO:teuthology.orchestra.run.smithi091.stdout:Complete! 2024-02-05T04:50:55.169 DEBUG:teuthology.orchestra.run.smithi091:> sudo yum -y install python3-rbd 2024-02-05T04:50:55.438 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:01:48 ago on Mon 05 Feb 2024 04:49:07 AM UTC. 2024-02-05T04:50:55.545 INFO:teuthology.orchestra.run.smithi062.stdout:Package python3-rbd-2:19.0.0-658.gd4ed20cf.el9.x86_64 is already installed. 2024-02-05T04:50:55.600 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-02-05T04:50:55.601 INFO:teuthology.orchestra.run.smithi062.stdout:Nothing to do. 2024-02-05T04:50:55.601 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-02-05T04:50:55.666 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install rbd-fuse 2024-02-05T04:50:55.747 INFO:teuthology.orchestra.run.smithi091.stdout:Last metadata expiration check: 0:01:46 ago on Mon 05 Feb 2024 04:49:09 AM UTC. 2024-02-05T04:50:55.851 INFO:teuthology.orchestra.run.smithi091.stdout:Package python3-rbd-2:19.0.0-658.gd4ed20cf.el9.x86_64 is already installed. 2024-02-05T04:50:55.906 INFO:teuthology.orchestra.run.smithi091.stdout:Dependencies resolved. 2024-02-05T04:50:55.906 INFO:teuthology.orchestra.run.smithi091.stdout:Nothing to do. 2024-02-05T04:50:55.906 INFO:teuthology.orchestra.run.smithi091.stdout:Complete! 2024-02-05T04:50:55.980 DEBUG:teuthology.orchestra.run.smithi091:> sudo yum -y install rbd-fuse 2024-02-05T04:50:56.239 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:01:49 ago on Mon 05 Feb 2024 04:49:07 AM UTC. 2024-02-05T04:50:56.403 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-02-05T04:50:56.404 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-02-05T04:50:56.404 INFO:teuthology.orchestra.run.smithi062.stdout: Package Architecture Version Repository Size 2024-02-05T04:50:56.404 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-02-05T04:50:56.404 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2024-02-05T04:50:56.404 INFO:teuthology.orchestra.run.smithi062.stdout: rbd-fuse x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 85 k 2024-02-05T04:50:56.405 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:50:56.405 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2024-02-05T04:50:56.405 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-02-05T04:50:56.405 INFO:teuthology.orchestra.run.smithi062.stdout:Install 1 Package 2024-02-05T04:50:56.405 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:50:56.405 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 85 k 2024-02-05T04:50:56.405 INFO:teuthology.orchestra.run.smithi062.stdout:Installed size: 231 k 2024-02-05T04:50:56.405 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2024-02-05T04:50:56.543 INFO:teuthology.orchestra.run.smithi091.stdout:Last metadata expiration check: 0:01:47 ago on Mon 05 Feb 2024 04:49:09 AM UTC. 2024-02-05T04:50:56.701 INFO:teuthology.orchestra.run.smithi091.stdout:Dependencies resolved. 2024-02-05T04:50:56.702 INFO:teuthology.orchestra.run.smithi091.stdout:================================================================================ 2024-02-05T04:50:56.702 INFO:teuthology.orchestra.run.smithi091.stdout: Package Architecture Version Repository Size 2024-02-05T04:50:56.702 INFO:teuthology.orchestra.run.smithi091.stdout:================================================================================ 2024-02-05T04:50:56.702 INFO:teuthology.orchestra.run.smithi091.stdout:Installing: 2024-02-05T04:50:56.702 INFO:teuthology.orchestra.run.smithi091.stdout: rbd-fuse x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 85 k 2024-02-05T04:50:56.702 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:50:56.702 INFO:teuthology.orchestra.run.smithi091.stdout:Transaction Summary 2024-02-05T04:50:56.702 INFO:teuthology.orchestra.run.smithi091.stdout:================================================================================ 2024-02-05T04:50:56.702 INFO:teuthology.orchestra.run.smithi091.stdout:Install 1 Package 2024-02-05T04:50:56.702 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:50:56.702 INFO:teuthology.orchestra.run.smithi091.stdout:Total download size: 85 k 2024-02-05T04:50:56.702 INFO:teuthology.orchestra.run.smithi091.stdout:Installed size: 231 k 2024-02-05T04:50:56.702 INFO:teuthology.orchestra.run.smithi091.stdout:Downloading Packages: 2024-02-05T04:50:56.842 INFO:teuthology.orchestra.run.smithi062.stdout:rbd-fuse-19.0.0-658.gd4ed20cf.el9.x86_64.rpm 195 kB/s | 85 kB 00:00 2024-02-05T04:50:56.843 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:50:56.843 INFO:teuthology.orchestra.run.smithi062.stdout:Total 194 kB/s | 85 kB 00:00 2024-02-05T04:50:56.843 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2024-02-05T04:50:56.851 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2024-02-05T04:50:56.851 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2024-02-05T04:50:56.908 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2024-02-05T04:50:56.911 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2024-02-05T04:50:57.059 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2024-02-05T04:50:57.151 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : rbd-fuse-2:19.0.0-658.gd4ed20cf.el9.x86_64 1/1 2024-02-05T04:50:57.247 INFO:teuthology.orchestra.run.smithi091.stdout:rbd-fuse-19.0.0-658.gd4ed20cf.el9.x86_64.rpm 156 kB/s | 85 kB 00:00 2024-02-05T04:50:57.248 INFO:teuthology.orchestra.run.smithi091.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:50:57.248 INFO:teuthology.orchestra.run.smithi091.stdout:Total 156 kB/s | 85 kB 00:00 2024-02-05T04:50:57.248 INFO:teuthology.orchestra.run.smithi091.stdout:Running transaction check 2024-02-05T04:50:57.256 INFO:teuthology.orchestra.run.smithi091.stdout:Transaction check succeeded. 2024-02-05T04:50:57.256 INFO:teuthology.orchestra.run.smithi091.stdout:Running transaction test 2024-02-05T04:50:57.295 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-658.gd4ed20 12/12 2024-02-05T04:50:57.295 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-658.gd4ed20 1/12 2024-02-05T04:50:57.295 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-02-05T04:50:57.296 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-02-05T04:50:57.296 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-02-05T04:50:57.296 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-02-05T04:50:57.296 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-02-05T04:50:57.296 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-02-05T04:50:57.296 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-02-05T04:50:57.296 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-02-05T04:50:57.296 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-02-05T04:50:57.296 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-02-05T04:50:57.310 INFO:teuthology.orchestra.run.smithi091.stdout:Transaction test succeeded. 2024-02-05T04:50:57.311 INFO:teuthology.orchestra.run.smithi091.stdout:Running transaction 2024-02-05T04:50:57.458 INFO:teuthology.orchestra.run.smithi091.stdout: Preparing : 1/1 2024-02-05T04:50:57.534 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: rbd-fuse-2:19.0.0-658.gd4ed20cf.el9.x86_64 1/1 2024-02-05T04:50:57.554 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : rbd-fuse-2:19.0.0-658.gd4ed20cf.el9.x86_64 1/1 2024-02-05T04:50:57.780 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-02-05T04:50:57.780 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:50:57.780 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2024-02-05T04:50:57.780 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-diskprediction-local-2:19.0.0-658.gd4ed20cf.el9.noarch 2024-02-05T04:50:57.780 INFO:teuthology.orchestra.run.smithi150.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-02-05T04:50:57.780 INFO:teuthology.orchestra.run.smithi150.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-02-05T04:50:57.780 INFO:teuthology.orchestra.run.smithi150.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-02-05T04:50:57.780 INFO:teuthology.orchestra.run.smithi150.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-02-05T04:50:57.780 INFO:teuthology.orchestra.run.smithi150.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-02-05T04:50:57.780 INFO:teuthology.orchestra.run.smithi150.stdout: openblas-0.3.21-2.el9.x86_64 2024-02-05T04:50:57.781 INFO:teuthology.orchestra.run.smithi150.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-02-05T04:50:57.781 INFO:teuthology.orchestra.run.smithi150.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-02-05T04:50:57.781 INFO:teuthology.orchestra.run.smithi150.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-02-05T04:50:57.781 INFO:teuthology.orchestra.run.smithi150.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-02-05T04:50:57.781 INFO:teuthology.orchestra.run.smithi150.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-02-05T04:50:57.781 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:50:57.781 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-02-05T04:50:57.833 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : rbd-fuse-2:19.0.0-658.gd4ed20cf.el9.x86_64 1/1 2024-02-05T04:50:57.834 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:50:57.834 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2024-02-05T04:50:57.834 INFO:teuthology.orchestra.run.smithi062.stdout: rbd-fuse-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:50:57.834 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:50:57.834 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-02-05T04:50:57.979 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: rbd-fuse-2:19.0.0-658.gd4ed20cf.el9.x86_64 1/1 2024-02-05T04:50:58.076 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install rbd-mirror 2024-02-05T04:50:58.128 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-mgr-rook 2024-02-05T04:50:58.300 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : rbd-fuse-2:19.0.0-658.gd4ed20cf.el9.x86_64 1/1 2024-02-05T04:50:58.300 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:50:58.301 INFO:teuthology.orchestra.run.smithi091.stdout:Installed: 2024-02-05T04:50:58.301 INFO:teuthology.orchestra.run.smithi091.stdout: rbd-fuse-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:50:58.301 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:50:58.301 INFO:teuthology.orchestra.run.smithi091.stdout:Complete! 2024-02-05T04:50:58.544 DEBUG:teuthology.orchestra.run.smithi091:> sudo yum -y install rbd-mirror 2024-02-05T04:50:58.659 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:01:51 ago on Mon 05 Feb 2024 04:49:07 AM UTC. 2024-02-05T04:50:58.701 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:19 ago on Mon 05 Feb 2024 04:49:39 AM UTC. 2024-02-05T04:50:58.821 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-02-05T04:50:58.822 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-02-05T04:50:58.822 INFO:teuthology.orchestra.run.smithi062.stdout: Package Arch Version Repo Size 2024-02-05T04:50:58.822 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-02-05T04:50:58.822 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2024-02-05T04:50:58.822 INFO:teuthology.orchestra.run.smithi062.stdout: rbd-mirror x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 3.1 M 2024-02-05T04:50:58.822 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:50:58.822 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2024-02-05T04:50:58.822 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-02-05T04:50:58.822 INFO:teuthology.orchestra.run.smithi062.stdout:Install 1 Package 2024-02-05T04:50:58.823 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:50:58.823 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 3.1 M 2024-02-05T04:50:58.823 INFO:teuthology.orchestra.run.smithi062.stdout:Installed size: 13 M 2024-02-05T04:50:58.823 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2024-02-05T04:50:58.861 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-02-05T04:50:58.862 INFO:teuthology.orchestra.run.smithi150.stdout:======================================================================================= 2024-02-05T04:50:58.862 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2024-02-05T04:50:58.862 INFO:teuthology.orchestra.run.smithi150.stdout:======================================================================================= 2024-02-05T04:50:58.862 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2024-02-05T04:50:58.862 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-rook noarch 2:19.0.0-658.gd4ed20cf.el9 ceph-noarch 50 k 2024-02-05T04:50:58.862 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2024-02-05T04:50:58.862 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-02-05T04:50:58.862 INFO:teuthology.orchestra.run.smithi150.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-02-05T04:50:58.863 INFO:teuthology.orchestra.run.smithi150.stdout: python3-google-auth noarch 1:2.27.0-1.el9 epel 203 k 2024-02-05T04:50:58.863 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-02-05T04:50:58.863 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-02-05T04:50:58.863 INFO:teuthology.orchestra.run.smithi150.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-02-05T04:50:58.863 INFO:teuthology.orchestra.run.smithi150.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-02-05T04:50:58.863 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-02-05T04:50:58.863 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-02-05T04:50:58.863 INFO:teuthology.orchestra.run.smithi150.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-02-05T04:50:58.863 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-02-05T04:50:58.863 INFO:teuthology.orchestra.run.smithi150.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-02-05T04:50:58.863 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:50:58.863 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-02-05T04:50:58.863 INFO:teuthology.orchestra.run.smithi150.stdout:======================================================================================= 2024-02-05T04:50:58.863 INFO:teuthology.orchestra.run.smithi150.stdout:Install 13 Packages 2024-02-05T04:50:58.863 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:50:58.864 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 2.0 M 2024-02-05T04:50:58.864 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 26 M 2024-02-05T04:50:58.864 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-02-05T04:50:59.154 INFO:teuthology.orchestra.run.smithi091.stdout:Last metadata expiration check: 0:01:50 ago on Mon 05 Feb 2024 04:49:09 AM UTC. 2024-02-05T04:50:59.317 INFO:teuthology.orchestra.run.smithi091.stdout:Dependencies resolved. 2024-02-05T04:50:59.317 INFO:teuthology.orchestra.run.smithi091.stdout:================================================================================ 2024-02-05T04:50:59.317 INFO:teuthology.orchestra.run.smithi091.stdout: Package Arch Version Repo Size 2024-02-05T04:50:59.317 INFO:teuthology.orchestra.run.smithi091.stdout:================================================================================ 2024-02-05T04:50:59.318 INFO:teuthology.orchestra.run.smithi091.stdout:Installing: 2024-02-05T04:50:59.318 INFO:teuthology.orchestra.run.smithi091.stdout: rbd-mirror x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 3.1 M 2024-02-05T04:50:59.318 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:50:59.318 INFO:teuthology.orchestra.run.smithi091.stdout:Transaction Summary 2024-02-05T04:50:59.318 INFO:teuthology.orchestra.run.smithi091.stdout:================================================================================ 2024-02-05T04:50:59.318 INFO:teuthology.orchestra.run.smithi091.stdout:Install 1 Package 2024-02-05T04:50:59.318 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:50:59.318 INFO:teuthology.orchestra.run.smithi091.stdout:Total download size: 3.1 M 2024-02-05T04:50:59.318 INFO:teuthology.orchestra.run.smithi091.stdout:Installed size: 13 M 2024-02-05T04:50:59.318 INFO:teuthology.orchestra.run.smithi091.stdout:Downloading Packages: 2024-02-05T04:50:59.365 INFO:teuthology.orchestra.run.smithi150.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 159 kB/s | 24 kB 00:00 2024-02-05T04:50:59.390 INFO:teuthology.orchestra.run.smithi150.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 101 kB/s | 18 kB 00:00 2024-02-05T04:50:59.441 INFO:teuthology.orchestra.run.smithi150.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-02-05T04:50:59.485 INFO:teuthology.orchestra.run.smithi150.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 132 kB 00:00 2024-02-05T04:50:59.508 INFO:teuthology.orchestra.run.smithi150.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.1 MB/s | 211 kB 00:00 2024-02-05T04:50:59.533 INFO:teuthology.orchestra.run.smithi150.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 904 kB/s | 43 kB 00:00 2024-02-05T04:50:59.725 INFO:teuthology.orchestra.run.smithi150.stdout:(7/13): ceph-mgr-rook-19.0.0-658.gd4ed20cf.el9. 98 kB/s | 50 kB 00:00 2024-02-05T04:50:59.792 INFO:teuthology.orchestra.run.smithi150.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 58 kB/s | 15 kB 00:00 2024-02-05T04:51:00.067 INFO:teuthology.orchestra.run.smithi150.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 58 kB/s | 32 kB 00:00 2024-02-05T04:51:01.129 INFO:teuthology.orchestra.run.smithi062.stdout:rbd-mirror-19.0.0-658.gd4ed20cf.el9.x86_64.rpm 1.3 MB/s | 3.1 MB 00:02 2024-02-05T04:51:01.129 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:51:01.129 INFO:teuthology.orchestra.run.smithi062.stdout:Total 1.3 MB/s | 3.1 MB 00:02 2024-02-05T04:51:01.130 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2024-02-05T04:51:01.142 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2024-02-05T04:51:01.142 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2024-02-05T04:51:01.221 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2024-02-05T04:51:01.221 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2024-02-05T04:51:01.678 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2024-02-05T04:51:01.726 INFO:teuthology.orchestra.run.smithi150.stdout:(10/13): python3-google-auth-2.27.0-1.el9.noarc 101 kB/s | 203 kB 00:02 2024-02-05T04:51:01.729 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : rbd-mirror-2:19.0.0-658.gd4ed20cf.el9.x86_64 1/1 2024-02-05T04:51:01.753 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: rbd-mirror-2:19.0.0-658.gd4ed20cf.el9.x86_64 1/1 2024-02-05T04:51:01.754 INFO:teuthology.orchestra.run.smithi062.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T04:51:01.754 INFO:teuthology.orchestra.run.smithi062.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-02-05T04:51:01.754 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-05T04:51:01.754 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-05T04:51:01.754 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:51:02.376 INFO:teuthology.orchestra.run.smithi091.stdout:rbd-mirror-19.0.0-658.gd4ed20cf.el9.x86_64.rpm 1.0 MB/s | 3.1 MB 00:03 2024-02-05T04:51:02.377 INFO:teuthology.orchestra.run.smithi091.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:51:02.378 INFO:teuthology.orchestra.run.smithi091.stdout:Total 1.0 MB/s | 3.1 MB 00:03 2024-02-05T04:51:02.378 INFO:teuthology.orchestra.run.smithi091.stdout:Running transaction check 2024-02-05T04:51:02.389 INFO:teuthology.orchestra.run.smithi091.stdout:Transaction check succeeded. 2024-02-05T04:51:02.389 INFO:teuthology.orchestra.run.smithi091.stdout:Running transaction test 2024-02-05T04:51:02.465 INFO:teuthology.orchestra.run.smithi091.stdout:Transaction test succeeded. 2024-02-05T04:51:02.465 INFO:teuthology.orchestra.run.smithi091.stdout:Running transaction 2024-02-05T04:51:02.883 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : rbd-mirror-2:19.0.0-658.gd4ed20cf.el9.x86_64 1/1 2024-02-05T04:51:02.883 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:51:02.883 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2024-02-05T04:51:02.884 INFO:teuthology.orchestra.run.smithi062.stdout: rbd-mirror-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:51:02.884 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:51:02.884 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-02-05T04:51:02.896 INFO:teuthology.orchestra.run.smithi091.stdout: Preparing : 1/1 2024-02-05T04:51:02.960 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : rbd-mirror-2:19.0.0-658.gd4ed20cf.el9.x86_64 1/1 2024-02-05T04:51:02.984 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: rbd-mirror-2:19.0.0-658.gd4ed20cf.el9.x86_64 1/1 2024-02-05T04:51:02.985 INFO:teuthology.orchestra.run.smithi091.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T04:51:02.985 INFO:teuthology.orchestra.run.smithi091.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-02-05T04:51:02.985 INFO:teuthology.orchestra.run.smithi091.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-05T04:51:02.985 INFO:teuthology.orchestra.run.smithi091.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-05T04:51:02.985 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:51:03.142 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install rbd-nbd 2024-02-05T04:51:03.724 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:01:56 ago on Mon 05 Feb 2024 04:49:07 AM UTC. 2024-02-05T04:51:03.892 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-02-05T04:51:03.892 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-02-05T04:51:03.892 INFO:teuthology.orchestra.run.smithi062.stdout: Package Architecture Version Repository Size 2024-02-05T04:51:03.892 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-02-05T04:51:03.892 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2024-02-05T04:51:03.892 INFO:teuthology.orchestra.run.smithi062.stdout: rbd-nbd x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 170 k 2024-02-05T04:51:03.893 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:51:03.893 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2024-02-05T04:51:03.893 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-02-05T04:51:03.893 INFO:teuthology.orchestra.run.smithi062.stdout:Install 1 Package 2024-02-05T04:51:03.893 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:51:03.893 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 170 k 2024-02-05T04:51:03.893 INFO:teuthology.orchestra.run.smithi062.stdout:Installed size: 490 k 2024-02-05T04:51:03.893 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2024-02-05T04:51:04.009 INFO:teuthology.orchestra.run.smithi150.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 15 kB/s | 59 kB 00:03 2024-02-05T04:51:04.099 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : rbd-mirror-2:19.0.0-658.gd4ed20cf.el9.x86_64 1/1 2024-02-05T04:51:04.100 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:51:04.100 INFO:teuthology.orchestra.run.smithi091.stdout:Installed: 2024-02-05T04:51:04.100 INFO:teuthology.orchestra.run.smithi091.stdout: rbd-mirror-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:51:04.100 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:51:04.100 INFO:teuthology.orchestra.run.smithi091.stdout:Complete! 2024-02-05T04:51:04.357 INFO:teuthology.orchestra.run.smithi062.stdout:rbd-nbd-19.0.0-658.gd4ed20cf.el9.x86_64.rpm 365 kB/s | 170 kB 00:00 2024-02-05T04:51:04.357 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:51:04.357 INFO:teuthology.orchestra.run.smithi062.stdout:Total 364 kB/s | 170 kB 00:00 2024-02-05T04:51:04.357 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2024-02-05T04:51:04.366 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2024-02-05T04:51:04.366 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2024-02-05T04:51:04.380 DEBUG:teuthology.orchestra.run.smithi091:> sudo yum -y install rbd-nbd 2024-02-05T04:51:04.424 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2024-02-05T04:51:04.424 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2024-02-05T04:51:04.591 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2024-02-05T04:51:04.684 INFO:teuthology.orchestra.run.smithi150.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 30 kB/s | 90 kB 00:02 2024-02-05T04:51:04.691 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : rbd-nbd-2:19.0.0-658.gd4ed20cf.el9.x86_64 1/1 2024-02-05T04:51:04.958 INFO:teuthology.orchestra.run.smithi091.stdout:Last metadata expiration check: 0:01:55 ago on Mon 05 Feb 2024 04:49:09 AM UTC. 2024-02-05T04:51:05.121 INFO:teuthology.orchestra.run.smithi091.stdout:Dependencies resolved. 2024-02-05T04:51:05.121 INFO:teuthology.orchestra.run.smithi091.stdout:================================================================================ 2024-02-05T04:51:05.121 INFO:teuthology.orchestra.run.smithi091.stdout: Package Architecture Version Repository Size 2024-02-05T04:51:05.121 INFO:teuthology.orchestra.run.smithi091.stdout:================================================================================ 2024-02-05T04:51:05.121 INFO:teuthology.orchestra.run.smithi091.stdout:Installing: 2024-02-05T04:51:05.121 INFO:teuthology.orchestra.run.smithi091.stdout: rbd-nbd x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 170 k 2024-02-05T04:51:05.121 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:51:05.121 INFO:teuthology.orchestra.run.smithi091.stdout:Transaction Summary 2024-02-05T04:51:05.121 INFO:teuthology.orchestra.run.smithi091.stdout:================================================================================ 2024-02-05T04:51:05.121 INFO:teuthology.orchestra.run.smithi091.stdout:Install 1 Package 2024-02-05T04:51:05.121 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:51:05.122 INFO:teuthology.orchestra.run.smithi091.stdout:Total download size: 170 k 2024-02-05T04:51:05.122 INFO:teuthology.orchestra.run.smithi091.stdout:Installed size: 490 k 2024-02-05T04:51:05.122 INFO:teuthology.orchestra.run.smithi091.stdout:Downloading Packages: 2024-02-05T04:51:05.159 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: rbd-nbd-2:19.0.0-658.gd4ed20cf.el9.x86_64 1/1 2024-02-05T04:51:05.467 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : rbd-nbd-2:19.0.0-658.gd4ed20cf.el9.x86_64 1/1 2024-02-05T04:51:05.467 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:51:05.467 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2024-02-05T04:51:05.467 INFO:teuthology.orchestra.run.smithi062.stdout: rbd-nbd-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:51:05.467 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:51:05.467 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-02-05T04:51:05.472 INFO:teuthology.orchestra.run.smithi091.stdout:rbd-nbd-19.0.0-658.gd4ed20cf.el9.x86_64.rpm 482 kB/s | 170 kB 00:00 2024-02-05T04:51:05.473 INFO:teuthology.orchestra.run.smithi091.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:51:05.473 INFO:teuthology.orchestra.run.smithi091.stdout:Total 480 kB/s | 170 kB 00:00 2024-02-05T04:51:05.473 INFO:teuthology.orchestra.run.smithi091.stdout:Running transaction check 2024-02-05T04:51:05.481 INFO:teuthology.orchestra.run.smithi091.stdout:Transaction check succeeded. 2024-02-05T04:51:05.481 INFO:teuthology.orchestra.run.smithi091.stdout:Running transaction test 2024-02-05T04:51:05.537 INFO:teuthology.orchestra.run.smithi091.stdout:Transaction test succeeded. 2024-02-05T04:51:05.537 INFO:teuthology.orchestra.run.smithi091.stdout:Running transaction 2024-02-05T04:51:05.690 INFO:teuthology.orchestra.run.smithi091.stdout: Preparing : 1/1 2024-02-05T04:51:05.709 DEBUG:teuthology.parallel:result is None 2024-02-05T04:51:05.787 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : rbd-nbd-2:19.0.0-658.gd4ed20cf.el9.x86_64 1/1 2024-02-05T04:51:06.257 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: rbd-nbd-2:19.0.0-658.gd4ed20cf.el9.x86_64 1/1 2024-02-05T04:51:06.575 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : rbd-nbd-2:19.0.0-658.gd4ed20cf.el9.x86_64 1/1 2024-02-05T04:51:06.575 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:51:06.575 INFO:teuthology.orchestra.run.smithi091.stdout:Installed: 2024-02-05T04:51:06.575 INFO:teuthology.orchestra.run.smithi091.stdout: rbd-nbd-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:51:06.575 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:51:06.575 INFO:teuthology.orchestra.run.smithi091.stdout:Complete! 2024-02-05T04:51:06.809 INFO:teuthology.orchestra.run.smithi150.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 149 kB/s | 1.0 MB 00:07 2024-02-05T04:51:06.813 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:51:06.814 INFO:teuthology.orchestra.run.smithi150.stdout:Total 264 kB/s | 2.0 MB 00:07 2024-02-05T04:51:06.814 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-02-05T04:51:06.825 DEBUG:teuthology.parallel:result is None 2024-02-05T04:51:06.828 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-02-05T04:51:06.829 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-02-05T04:51:06.936 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-02-05T04:51:06.936 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-02-05T04:51:07.462 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-02-05T04:51:07.629 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-02-05T04:51:07.677 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-02-05T04:51:07.731 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-02-05T04:51:07.761 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-02-05T04:51:07.802 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-02-05T04:51:07.894 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-02-05T04:51:08.004 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-google-auth-1:2.27.0-1.el9.noarch 7/13 2024-02-05T04:51:08.062 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-02-05T04:51:08.776 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-02-05T04:51:08.818 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-02-05T04:51:08.851 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-02-05T04:51:08.899 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-02-05T04:51:08.922 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-rook-2:19.0.0-658.gd4ed20cf.el9.noarch 13/13 2024-02-05T04:51:09.789 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-658.gd4ed20cf.el9.noarch 13/13 2024-02-05T04:51:09.790 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-rook-2:19.0.0-658.gd4ed20cf.el9.noarch 1/13 2024-02-05T04:51:09.790 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-02-05T04:51:09.790 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-02-05T04:51:09.790 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-02-05T04:51:09.790 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-02-05T04:51:09.790 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-02-05T04:51:09.790 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-02-05T04:51:09.790 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-02-05T04:51:09.790 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-02-05T04:51:09.790 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-google-auth-1:2.27.0-1.el9.noarch 10/13 2024-02-05T04:51:09.790 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-02-05T04:51:09.790 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-02-05T04:51:10.163 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-02-05T04:51:10.163 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:51:10.163 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2024-02-05T04:51:10.163 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-rook-2:19.0.0-658.gd4ed20cf.el9.noarch 2024-02-05T04:51:10.163 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-02-05T04:51:10.163 INFO:teuthology.orchestra.run.smithi150.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-02-05T04:51:10.163 INFO:teuthology.orchestra.run.smithi150.stdout: python3-google-auth-1:2.27.0-1.el9.noarch 2024-02-05T04:51:10.163 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-02-05T04:51:10.163 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-02-05T04:51:10.164 INFO:teuthology.orchestra.run.smithi150.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-02-05T04:51:10.164 INFO:teuthology.orchestra.run.smithi150.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-02-05T04:51:10.164 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-02-05T04:51:10.164 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-02-05T04:51:10.164 INFO:teuthology.orchestra.run.smithi150.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-02-05T04:51:10.164 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rsa-4.9-2.el9.noarch 2024-02-05T04:51:10.164 INFO:teuthology.orchestra.run.smithi150.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-02-05T04:51:10.164 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:51:10.164 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-02-05T04:51:10.447 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-mgr-cephadm 2024-02-05T04:51:11.023 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:32 ago on Mon 05 Feb 2024 04:49:39 AM UTC. 2024-02-05T04:51:11.183 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-02-05T04:51:11.184 INFO:teuthology.orchestra.run.smithi150.stdout:======================================================================================= 2024-02-05T04:51:11.184 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2024-02-05T04:51:11.184 INFO:teuthology.orchestra.run.smithi150.stdout:======================================================================================= 2024-02-05T04:51:11.184 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2024-02-05T04:51:11.184 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-cephadm noarch 2:19.0.0-658.gd4ed20cf.el9 ceph-noarch 133 k 2024-02-05T04:51:11.184 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2024-02-05T04:51:11.184 INFO:teuthology.orchestra.run.smithi150.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2024-02-05T04:51:11.184 INFO:teuthology.orchestra.run.smithi150.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-02-05T04:51:11.185 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2024-02-05T04:51:11.185 INFO:teuthology.orchestra.run.smithi150.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-02-05T04:51:11.185 INFO:teuthology.orchestra.run.smithi150.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-02-05T04:51:11.185 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:51:11.185 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-02-05T04:51:11.185 INFO:teuthology.orchestra.run.smithi150.stdout:======================================================================================= 2024-02-05T04:51:11.185 INFO:teuthology.orchestra.run.smithi150.stdout:Install 6 Packages 2024-02-05T04:51:11.185 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:51:11.186 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 6.8 M 2024-02-05T04:51:11.186 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 33 M 2024-02-05T04:51:11.186 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-02-05T04:51:11.512 INFO:teuthology.orchestra.run.smithi150.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.3 MB/s | 228 kB 00:00 2024-02-05T04:51:11.746 INFO:teuthology.orchestra.run.smithi150.stdout:(2/6): ceph-mgr-cephadm-19.0.0-658.gd4ed20cf.el 327 kB/s | 133 kB 00:00 2024-02-05T04:51:11.796 INFO:teuthology.orchestra.run.smithi150.stdout:(3/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 1.9 MB/s | 548 kB 00:00 2024-02-05T04:51:11.872 INFO:teuthology.orchestra.run.smithi150.stdout:(4/6): python3-babel-2.9.1-2.el9.noarch.rpm 11 MB/s | 5.8 MB 00:00 2024-02-05T04:51:11.896 INFO:teuthology.orchestra.run.smithi150.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 386 kB/s | 58 kB 00:00 2024-02-05T04:51:11.922 INFO:teuthology.orchestra.run.smithi150.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 407 kB/s | 51 kB 00:00 2024-02-05T04:51:11.923 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:51:11.924 INFO:teuthology.orchestra.run.smithi150.stdout:Total 9.2 MB/s | 6.8 MB 00:00 2024-02-05T04:51:11.924 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-02-05T04:51:11.933 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-02-05T04:51:11.933 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-02-05T04:51:12.024 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-02-05T04:51:12.024 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-02-05T04:51:12.327 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-02-05T04:51:12.488 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-02-05T04:51:12.540 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2024-02-05T04:51:13.054 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-02-05T04:51:13.142 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-02-05T04:51:13.203 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2024-02-05T04:51:13.227 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-658.gd4ed20cf.el9.noarch 6/6 2024-02-05T04:51:14.107 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-658.gd4ed20cf.el9.noarch 6/6 2024-02-05T04:51:14.107 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-658.gd4ed20cf.el9.noarch 1/6 2024-02-05T04:51:14.107 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-02-05T04:51:14.108 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2024-02-05T04:51:14.108 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2024-02-05T04:51:14.108 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-02-05T04:51:14.425 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-02-05T04:51:14.425 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:51:14.425 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2024-02-05T04:51:14.426 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-cephadm-2:19.0.0-658.gd4ed20cf.el9.noarch 2024-02-05T04:51:14.426 INFO:teuthology.orchestra.run.smithi150.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2024-02-05T04:51:14.426 INFO:teuthology.orchestra.run.smithi150.stdout: python3-babel-2.9.1-2.el9.noarch 2024-02-05T04:51:14.426 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jinja2-2.11.3-4.el9.noarch 2024-02-05T04:51:14.426 INFO:teuthology.orchestra.run.smithi150.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-02-05T04:51:14.426 INFO:teuthology.orchestra.run.smithi150.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-02-05T04:51:14.426 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:51:14.426 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-02-05T04:51:14.706 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-fuse 2024-02-05T04:51:15.269 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:36 ago on Mon 05 Feb 2024 04:49:39 AM UTC. 2024-02-05T04:51:15.426 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-02-05T04:51:15.427 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-02-05T04:51:15.428 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2024-02-05T04:51:15.428 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-02-05T04:51:15.428 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2024-02-05T04:51:15.428 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-fuse x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 887 k 2024-02-05T04:51:15.428 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2024-02-05T04:51:15.428 INFO:teuthology.orchestra.run.smithi150.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-02-05T04:51:15.428 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:51:15.428 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-02-05T04:51:15.429 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-02-05T04:51:15.429 INFO:teuthology.orchestra.run.smithi150.stdout:Install 2 Packages 2024-02-05T04:51:15.429 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:51:15.429 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 966 k 2024-02-05T04:51:15.429 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 2.8 M 2024-02-05T04:51:15.429 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-02-05T04:51:15.682 INFO:teuthology.orchestra.run.smithi150.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 310 kB/s | 79 kB 00:00 2024-02-05T04:51:15.973 INFO:teuthology.orchestra.run.smithi150.stdout:(2/2): ceph-fuse-19.0.0-658.gd4ed20cf.el9.x86_6 1.6 MB/s | 887 kB 00:00 2024-02-05T04:51:15.975 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:51:15.975 INFO:teuthology.orchestra.run.smithi150.stdout:Total 1.7 MB/s | 966 kB 00:00 2024-02-05T04:51:15.975 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-02-05T04:51:15.995 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-02-05T04:51:15.995 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-02-05T04:51:16.071 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-02-05T04:51:16.071 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-02-05T04:51:16.266 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-02-05T04:51:16.400 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-02-05T04:51:16.490 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-fuse-2:19.0.0-658.gd4ed20cf.el9.x86_64 2/2 2024-02-05T04:51:17.247 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-fuse-2:19.0.0-658.gd4ed20cf.el9.x86_64 2/2 2024-02-05T04:51:17.247 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-fuse-2:19.0.0-658.gd4ed20cf.el9.x86_64 1/2 2024-02-05T04:51:17.599 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-02-05T04:51:17.599 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:51:17.599 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2024-02-05T04:51:17.599 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-fuse-2:19.0.0-658.gd4ed20cf.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-02-05T04:51:17.599 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:51:17.599 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-02-05T04:51:17.862 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-volume 2024-02-05T04:51:18.452 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:39 ago on Mon 05 Feb 2024 04:49:39 AM UTC. 2024-02-05T04:51:18.608 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-02-05T04:51:18.609 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-02-05T04:51:18.609 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2024-02-05T04:51:18.609 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-02-05T04:51:18.609 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2024-02-05T04:51:18.609 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-volume noarch 2:19.0.0-658.gd4ed20cf.el9 ceph-noarch 262 k 2024-02-05T04:51:18.609 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:51:18.609 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-02-05T04:51:18.609 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-02-05T04:51:18.609 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2024-02-05T04:51:18.609 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:51:18.610 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 262 k 2024-02-05T04:51:18.610 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 1.2 M 2024-02-05T04:51:18.610 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-02-05T04:51:18.878 INFO:teuthology.orchestra.run.smithi150.stdout:ceph-volume-19.0.0-658.gd4ed20cf.el9.noarch.rpm 979 kB/s | 262 kB 00:00 2024-02-05T04:51:18.878 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:51:18.879 INFO:teuthology.orchestra.run.smithi150.stdout:Total 974 kB/s | 262 kB 00:00 2024-02-05T04:51:18.879 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-02-05T04:51:18.887 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-02-05T04:51:18.887 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-02-05T04:51:18.926 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-02-05T04:51:18.926 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-02-05T04:51:19.139 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-02-05T04:51:19.213 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-volume-2:19.0.0-658.gd4ed20cf.el9.noarch 1/1 2024-02-05T04:51:19.229 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-volume-2:19.0.0-658.gd4ed20cf.el9.noarch 1/1 2024-02-05T04:51:19.229 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T04:51:19.229 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-02-05T04:51:19.229 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:51:20.300 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-volume-2:19.0.0-658.gd4ed20cf.el9.noarch 1/1 2024-02-05T04:51:20.300 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:51:20.300 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2024-02-05T04:51:20.300 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-volume-2:19.0.0-658.gd4ed20cf.el9.noarch 2024-02-05T04:51:20.300 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:51:20.300 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-02-05T04:51:20.660 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install librados-devel 2024-02-05T04:51:21.246 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:42 ago on Mon 05 Feb 2024 04:49:39 AM UTC. 2024-02-05T04:51:21.406 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-02-05T04:51:21.407 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-02-05T04:51:21.407 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repo Size 2024-02-05T04:51:21.407 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-02-05T04:51:21.407 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2024-02-05T04:51:21.407 INFO:teuthology.orchestra.run.smithi150.stdout: librados-devel x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 127 k 2024-02-05T04:51:21.407 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:51:21.407 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-02-05T04:51:21.407 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-02-05T04:51:21.407 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2024-02-05T04:51:21.407 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:51:21.407 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 127 k 2024-02-05T04:51:21.408 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 456 k 2024-02-05T04:51:21.408 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-02-05T04:51:21.716 INFO:teuthology.orchestra.run.smithi150.stdout:librados-devel-19.0.0-658.gd4ed20cf.el9.x86_64. 411 kB/s | 127 kB 00:00 2024-02-05T04:51:21.716 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:51:21.717 INFO:teuthology.orchestra.run.smithi150.stdout:Total 409 kB/s | 127 kB 00:00 2024-02-05T04:51:21.717 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-02-05T04:51:21.722 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-02-05T04:51:21.722 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-02-05T04:51:21.780 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-02-05T04:51:21.780 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-02-05T04:51:21.927 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-02-05T04:51:22.031 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : librados-devel-2:19.0.0-658.gd4ed20cf.el9.x86_64 1/1 2024-02-05T04:51:22.532 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librados-devel-2:19.0.0-658.gd4ed20cf.el9.x86_64 1/1 2024-02-05T04:51:22.840 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librados-devel-2:19.0.0-658.gd4ed20cf.el9.x86_64 1/1 2024-02-05T04:51:22.840 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:51:22.840 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2024-02-05T04:51:22.840 INFO:teuthology.orchestra.run.smithi150.stdout: librados-devel-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:51:22.841 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:51:22.841 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-02-05T04:51:23.095 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install libcephfs2 2024-02-05T04:51:23.680 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:44 ago on Mon 05 Feb 2024 04:49:39 AM UTC. 2024-02-05T04:51:23.788 INFO:teuthology.orchestra.run.smithi150.stdout:Package libcephfs2-2:19.0.0-658.gd4ed20cf.el9.x86_64 is already installed. 2024-02-05T04:51:23.840 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-02-05T04:51:23.841 INFO:teuthology.orchestra.run.smithi150.stdout:Nothing to do. 2024-02-05T04:51:23.841 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-02-05T04:51:23.918 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install libcephfs-devel 2024-02-05T04:51:24.489 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:45 ago on Mon 05 Feb 2024 04:49:39 AM UTC. 2024-02-05T04:51:24.645 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-02-05T04:51:24.645 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-02-05T04:51:24.646 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repo Size 2024-02-05T04:51:24.646 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-02-05T04:51:24.646 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2024-02-05T04:51:24.646 INFO:teuthology.orchestra.run.smithi150.stdout: libcephfs-devel x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 31 k 2024-02-05T04:51:24.646 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:51:24.646 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-02-05T04:51:24.646 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-02-05T04:51:24.646 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2024-02-05T04:51:24.646 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:51:24.646 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 31 k 2024-02-05T04:51:24.646 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 140 k 2024-02-05T04:51:24.647 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-02-05T04:51:24.819 INFO:teuthology.orchestra.run.smithi150.stdout:libcephfs-devel-19.0.0-658.gd4ed20cf.el9.x86_64 181 kB/s | 31 kB 00:00 2024-02-05T04:51:24.820 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:51:24.820 INFO:teuthology.orchestra.run.smithi150.stdout:Total 179 kB/s | 31 kB 00:00 2024-02-05T04:51:24.820 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-02-05T04:51:24.823 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-02-05T04:51:24.823 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-02-05T04:51:24.855 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-02-05T04:51:24.856 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-02-05T04:51:24.951 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-02-05T04:51:25.060 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libcephfs-devel-2:19.0.0-658.gd4ed20cf.el9.x86_64 1/1 2024-02-05T04:51:25.477 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-658.gd4ed20cf.el9.x86_64 1/1 2024-02-05T04:51:25.746 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libcephfs-devel-2:19.0.0-658.gd4ed20cf.el9.x86_64 1/1 2024-02-05T04:51:25.746 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:51:25.746 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2024-02-05T04:51:25.746 INFO:teuthology.orchestra.run.smithi150.stdout: libcephfs-devel-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:51:25.746 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:51:25.746 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-02-05T04:51:25.925 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install librados2 2024-02-05T04:51:26.496 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:47 ago on Mon 05 Feb 2024 04:49:39 AM UTC. 2024-02-05T04:51:26.600 INFO:teuthology.orchestra.run.smithi150.stdout:Package librados2-2:19.0.0-658.gd4ed20cf.el9.x86_64 is already installed. 2024-02-05T04:51:26.653 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-02-05T04:51:26.654 INFO:teuthology.orchestra.run.smithi150.stdout:Nothing to do. 2024-02-05T04:51:26.654 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-02-05T04:51:26.735 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install librbd1 2024-02-05T04:51:27.302 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:48 ago on Mon 05 Feb 2024 04:49:39 AM UTC. 2024-02-05T04:51:27.408 INFO:teuthology.orchestra.run.smithi150.stdout:Package librbd1-2:19.0.0-658.gd4ed20cf.el9.x86_64 is already installed. 2024-02-05T04:51:27.462 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-02-05T04:51:27.463 INFO:teuthology.orchestra.run.smithi150.stdout:Nothing to do. 2024-02-05T04:51:27.463 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-02-05T04:51:27.541 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install python3-rados 2024-02-05T04:51:28.116 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:49 ago on Mon 05 Feb 2024 04:49:39 AM UTC. 2024-02-05T04:51:28.221 INFO:teuthology.orchestra.run.smithi150.stdout:Package python3-rados-2:19.0.0-658.gd4ed20cf.el9.x86_64 is already installed. 2024-02-05T04:51:28.273 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-02-05T04:51:28.274 INFO:teuthology.orchestra.run.smithi150.stdout:Nothing to do. 2024-02-05T04:51:28.275 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-02-05T04:51:28.355 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install python3-rgw 2024-02-05T04:51:28.931 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:49 ago on Mon 05 Feb 2024 04:49:39 AM UTC. 2024-02-05T04:51:29.035 INFO:teuthology.orchestra.run.smithi150.stdout:Package python3-rgw-2:19.0.0-658.gd4ed20cf.el9.x86_64 is already installed. 2024-02-05T04:51:29.088 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-02-05T04:51:29.089 INFO:teuthology.orchestra.run.smithi150.stdout:Nothing to do. 2024-02-05T04:51:29.089 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-02-05T04:51:29.176 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install python3-cephfs 2024-02-05T04:51:29.752 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:50 ago on Mon 05 Feb 2024 04:49:39 AM UTC. 2024-02-05T04:51:29.855 INFO:teuthology.orchestra.run.smithi150.stdout:Package python3-cephfs-2:19.0.0-658.gd4ed20cf.el9.x86_64 is already installed. 2024-02-05T04:51:29.907 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-02-05T04:51:29.909 INFO:teuthology.orchestra.run.smithi150.stdout:Nothing to do. 2024-02-05T04:51:29.909 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-02-05T04:51:29.993 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install python3-rbd 2024-02-05T04:51:30.571 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:51 ago on Mon 05 Feb 2024 04:49:39 AM UTC. 2024-02-05T04:51:30.675 INFO:teuthology.orchestra.run.smithi150.stdout:Package python3-rbd-2:19.0.0-658.gd4ed20cf.el9.x86_64 is already installed. 2024-02-05T04:51:30.727 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-02-05T04:51:30.727 INFO:teuthology.orchestra.run.smithi150.stdout:Nothing to do. 2024-02-05T04:51:30.727 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-02-05T04:51:30.818 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install rbd-fuse 2024-02-05T04:51:31.388 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:52 ago on Mon 05 Feb 2024 04:49:39 AM UTC. 2024-02-05T04:51:31.545 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-02-05T04:51:31.546 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-02-05T04:51:31.546 INFO:teuthology.orchestra.run.smithi150.stdout: Package Architecture Version Repository Size 2024-02-05T04:51:31.546 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-02-05T04:51:31.546 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2024-02-05T04:51:31.547 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-fuse x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 85 k 2024-02-05T04:51:31.547 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:51:31.547 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-02-05T04:51:31.547 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-02-05T04:51:31.547 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2024-02-05T04:51:31.547 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:51:31.547 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 85 k 2024-02-05T04:51:31.547 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 231 k 2024-02-05T04:51:31.547 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-02-05T04:51:31.755 INFO:teuthology.orchestra.run.smithi150.stdout:rbd-fuse-19.0.0-658.gd4ed20cf.el9.x86_64.rpm 409 kB/s | 85 kB 00:00 2024-02-05T04:51:31.756 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:51:31.756 INFO:teuthology.orchestra.run.smithi150.stdout:Total 407 kB/s | 85 kB 00:00 2024-02-05T04:51:31.756 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-02-05T04:51:31.763 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-02-05T04:51:31.763 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-02-05T04:51:31.819 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-02-05T04:51:31.819 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-02-05T04:51:31.984 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-02-05T04:51:32.087 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : rbd-fuse-2:19.0.0-658.gd4ed20cf.el9.x86_64 1/1 2024-02-05T04:51:32.506 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: rbd-fuse-2:19.0.0-658.gd4ed20cf.el9.x86_64 1/1 2024-02-05T04:51:32.796 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : rbd-fuse-2:19.0.0-658.gd4ed20cf.el9.x86_64 1/1 2024-02-05T04:51:32.797 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:51:32.797 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2024-02-05T04:51:32.797 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-fuse-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:51:32.797 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:51:32.797 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-02-05T04:51:33.042 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install rbd-mirror 2024-02-05T04:51:33.630 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:54 ago on Mon 05 Feb 2024 04:49:39 AM UTC. 2024-02-05T04:51:33.793 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-02-05T04:51:33.793 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-02-05T04:51:33.794 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repo Size 2024-02-05T04:51:33.794 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-02-05T04:51:33.794 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2024-02-05T04:51:33.794 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-mirror x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 3.1 M 2024-02-05T04:51:33.794 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:51:33.794 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-02-05T04:51:33.794 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-02-05T04:51:33.794 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2024-02-05T04:51:33.794 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:51:33.794 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 3.1 M 2024-02-05T04:51:33.794 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 13 M 2024-02-05T04:51:33.794 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-02-05T04:51:34.558 INFO:teuthology.orchestra.run.smithi150.stdout:rbd-mirror-19.0.0-658.gd4ed20cf.el9.x86_64.rpm 4.0 MB/s | 3.1 MB 00:00 2024-02-05T04:51:34.559 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:51:34.559 INFO:teuthology.orchestra.run.smithi150.stdout:Total 4.0 MB/s | 3.1 MB 00:00 2024-02-05T04:51:34.559 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-02-05T04:51:34.568 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-02-05T04:51:34.569 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-02-05T04:51:34.642 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-02-05T04:51:34.642 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-02-05T04:51:35.071 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-02-05T04:51:35.134 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : rbd-mirror-2:19.0.0-658.gd4ed20cf.el9.x86_64 1/1 2024-02-05T04:51:35.158 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: rbd-mirror-2:19.0.0-658.gd4ed20cf.el9.x86_64 1/1 2024-02-05T04:51:35.158 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T04:51:35.159 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-02-05T04:51:35.159 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-05T04:51:35.159 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-05T04:51:35.159 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:51:36.322 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : rbd-mirror-2:19.0.0-658.gd4ed20cf.el9.x86_64 1/1 2024-02-05T04:51:36.322 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:51:36.322 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2024-02-05T04:51:36.322 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-mirror-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:51:36.322 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:51:36.323 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-02-05T04:51:36.562 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install rbd-nbd 2024-02-05T04:51:37.144 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:58 ago on Mon 05 Feb 2024 04:49:39 AM UTC. 2024-02-05T04:51:37.302 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-02-05T04:51:37.303 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-02-05T04:51:37.304 INFO:teuthology.orchestra.run.smithi150.stdout: Package Architecture Version Repository Size 2024-02-05T04:51:37.304 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-02-05T04:51:37.304 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2024-02-05T04:51:37.304 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-nbd x86_64 2:19.0.0-658.gd4ed20cf.el9 ceph 170 k 2024-02-05T04:51:37.304 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:51:37.304 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-02-05T04:51:37.304 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-02-05T04:51:37.304 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2024-02-05T04:51:37.305 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:51:37.305 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 170 k 2024-02-05T04:51:37.305 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 490 k 2024-02-05T04:51:37.305 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-02-05T04:51:37.537 INFO:teuthology.orchestra.run.smithi150.stdout:rbd-nbd-19.0.0-658.gd4ed20cf.el9.x86_64.rpm 733 kB/s | 170 kB 00:00 2024-02-05T04:51:37.537 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:51:37.537 INFO:teuthology.orchestra.run.smithi150.stdout:Total 728 kB/s | 170 kB 00:00 2024-02-05T04:51:37.538 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-02-05T04:51:37.545 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-02-05T04:51:37.546 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-02-05T04:51:37.601 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-02-05T04:51:37.601 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-02-05T04:51:37.767 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-02-05T04:51:37.869 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : rbd-nbd-2:19.0.0-658.gd4ed20cf.el9.x86_64 1/1 2024-02-05T04:51:38.337 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: rbd-nbd-2:19.0.0-658.gd4ed20cf.el9.x86_64 1/1 2024-02-05T04:51:38.639 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : rbd-nbd-2:19.0.0-658.gd4ed20cf.el9.x86_64 1/1 2024-02-05T04:51:38.639 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:51:38.639 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2024-02-05T04:51:38.639 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-nbd-2:19.0.0-658.gd4ed20cf.el9.x86_64 2024-02-05T04:51:38.639 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:51:38.639 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-02-05T04:51:38.918 DEBUG:teuthology.parallel:result is None 2024-02-05T04:51:38.919 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d4ed20cfb238ef2c47d64ca4533eca00b8968b98 2024-02-05T04:51:39.071 DEBUG:teuthology.orchestra.run.smithi062:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-05T04:51:39.096 INFO:teuthology.orchestra.run.smithi062.stdout:19.0.0-658.gd4ed20cf.el9 2024-02-05T04:51:39.097 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-658.gd4ed20cf.el9 2024-02-05T04:51:39.097 INFO:teuthology.task.install:The correct ceph version 19.0.0-658.gd4ed20cf is installed. 2024-02-05T04:51:39.099 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d4ed20cfb238ef2c47d64ca4533eca00b8968b98 2024-02-05T04:51:39.314 DEBUG:teuthology.orchestra.run.smithi091:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-05T04:51:39.339 INFO:teuthology.orchestra.run.smithi091.stdout:19.0.0-658.gd4ed20cf.el9 2024-02-05T04:51:39.339 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-658.gd4ed20cf.el9 2024-02-05T04:51:39.339 INFO:teuthology.task.install:The correct ceph version 19.0.0-658.gd4ed20cf is installed. 2024-02-05T04:51:39.341 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d4ed20cfb238ef2c47d64ca4533eca00b8968b98 2024-02-05T04:51:39.480 DEBUG:teuthology.orchestra.run.smithi150:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-05T04:51:39.501 INFO:teuthology.orchestra.run.smithi150.stdout:19.0.0-658.gd4ed20cf.el9 2024-02-05T04:51:39.501 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-658.gd4ed20cf.el9 2024-02-05T04:51:39.501 INFO:teuthology.task.install:The correct ceph version 19.0.0-658.gd4ed20cf is installed. 2024-02-05T04:51:39.503 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d4ed20cfb238ef2c47d64ca4533eca00b8968b98 2024-02-05T04:51:39.719 DEBUG:teuthology.orchestra.run.smithi196:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-05T04:51:39.744 INFO:teuthology.orchestra.run.smithi196.stdout:19.0.0-658.gd4ed20cf.el9 2024-02-05T04:51:39.744 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-658.gd4ed20cf.el9 2024-02-05T04:51:39.744 INFO:teuthology.task.install:The correct ceph version 19.0.0-658.gd4ed20cf is installed. 2024-02-05T04:51:39.747 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-02-05T04:51:39.747 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-02-05T04:51:39.747 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-05T04:51:39.780 DEBUG:teuthology.orchestra.run.smithi091:> set -ex 2024-02-05T04:51:39.780 DEBUG:teuthology.orchestra.run.smithi091:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-05T04:51:39.813 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-02-05T04:51:39.813 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-05T04:51:39.843 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-02-05T04:51:39.844 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-05T04:51:39.876 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-02-05T04:51:39.876 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-02-05T04:51:39.876 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/usr/bin/daemon-helper 2024-02-05T04:51:39.912 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-05T04:51:39.980 DEBUG:teuthology.orchestra.run.smithi091:> set -ex 2024-02-05T04:51:39.981 DEBUG:teuthology.orchestra.run.smithi091:> sudo dd of=/usr/bin/daemon-helper 2024-02-05T04:51:40.011 DEBUG:teuthology.orchestra.run.smithi091:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-05T04:51:40.079 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-02-05T04:51:40.079 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/usr/bin/daemon-helper 2024-02-05T04:51:40.107 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-05T04:51:40.174 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-02-05T04:51:40.174 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/usr/bin/daemon-helper 2024-02-05T04:51:40.201 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-05T04:51:40.269 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-02-05T04:51:40.269 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-02-05T04:51:40.270 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-05T04:51:40.297 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-05T04:51:40.363 DEBUG:teuthology.orchestra.run.smithi091:> set -ex 2024-02-05T04:51:40.363 DEBUG:teuthology.orchestra.run.smithi091:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-05T04:51:40.390 DEBUG:teuthology.orchestra.run.smithi091:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-05T04:51:40.457 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-02-05T04:51:40.457 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-05T04:51:40.484 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-05T04:51:40.551 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-02-05T04:51:40.551 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-05T04:51:40.579 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-05T04:51:40.646 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-02-05T04:51:40.646 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-02-05T04:51:40.646 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/usr/bin/stdin-killer 2024-02-05T04:51:40.673 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-05T04:51:40.738 DEBUG:teuthology.orchestra.run.smithi091:> set -ex 2024-02-05T04:51:40.739 DEBUG:teuthology.orchestra.run.smithi091:> sudo dd of=/usr/bin/stdin-killer 2024-02-05T04:51:40.765 DEBUG:teuthology.orchestra.run.smithi091:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-05T04:51:40.832 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-02-05T04:51:40.832 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/usr/bin/stdin-killer 2024-02-05T04:51:40.859 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-05T04:51:40.927 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-02-05T04:51:40.927 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/usr/bin/stdin-killer 2024-02-05T04:51:40.954 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-05T04:51:41.022 INFO:teuthology.run_tasks:Running task ceph... 2024-02-05T04:51:41.119 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-02-05T04:51:41.119 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 777 /var/log/ceph 2024-02-05T04:51:41.122 DEBUG:teuthology.orchestra.run.smithi091:> sudo chmod 777 /var/log/ceph 2024-02-05T04:51:41.124 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 777 /var/log/ceph 2024-02-05T04:51:41.127 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod 777 /var/log/ceph 2024-02-05T04:51:41.154 INFO:tasks.ceph:Disabling ceph logrotate... 2024-02-05T04:51:41.155 DEBUG:teuthology.orchestra.run.smithi062:> sudo rm -f -- /etc/logrotate.d/ceph 2024-02-05T04:51:41.189 DEBUG:teuthology.orchestra.run.smithi091:> sudo rm -f -- /etc/logrotate.d/ceph 2024-02-05T04:51:41.191 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -f -- /etc/logrotate.d/ceph 2024-02-05T04:51:41.195 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -f -- /etc/logrotate.d/ceph 2024-02-05T04:51:41.221 INFO:tasks.ceph:Creating extra log directories... 2024-02-05T04:51:41.221 DEBUG:teuthology.orchestra.run.smithi062:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-02-05T04:51:41.255 DEBUG:teuthology.orchestra.run.smithi091:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-02-05T04:51:41.257 DEBUG:teuthology.orchestra.run.smithi150:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-02-05T04:51:41.261 DEBUG:teuthology.orchestra.run.smithi196:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-02-05T04:51:41.302 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-02-05T04:51:41.302 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.1, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject socket failures': 2500, 'ms tcp read timeout': 5, 'osd_async_recovery_min_cost': 1, 'osd_object_clean_region_max_num_intervals': 1000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-02-05T04:51:41.303 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/nmordech-2024-01-31_07:15:30-rados-wip-nitzan-max-pglog-size-distro-default-smithi/7539667', 'branch': 'wip-nitzan-max-pglog-size', 'description': 'rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2}', 'email': None, 'first_in_suite': False, 'job_id': '7539667', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'nmordech-2024-01-31_07:15:30-rados-wip-nitzan-max-pglog-size-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-nitzan-max-pglog-size'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject socket failures': 2500, 'ms tcp read timeout': 5, 'osd_async_recovery_min_cost': 1, 'osd_object_clean_region_max_num_intervals': 1000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.1, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from'], 'sha1': 'd4ed20cfb238ef2c47d64ca4533eca00b8968b98'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'd4ed20cfb238ef2c47d64ca4533eca00b8968b98'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-nitzan-max-pglog-size', 'sha1': 'd4ed20cfb238ef2c47d64ca4533eca00b8968b98'}}, 'owner': 'scheduled_nmordech@teuthology', 'priority': 1000, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'osd.0', 'osd.4', 'osd.8', 'osd.12'], ['mon.b', 'osd.1', 'osd.5', 'osd.9', 'osd.13'], ['mon.c', 'osd.2', 'osd.6', 'osd.10', 'osd.14'], ['mgr.x', 'osd.3', 'osd.7', 'osd.11', 'osd.15', 'client.0']], 'seed': 8695, 'sha1': 'd4ed20cfb238ef2c47d64ca4533eca00b8968b98', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-nitzan-max-pglog-size', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': 'd4ed20cfb238ef2c47d64ca4533eca00b8968b98', 'targets': {'smithi062.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLrCXX9FGrvpb2YY//9WUXmiMbUrI6W2c1RxGH7NTG8iNneJEkvdg41pWixfxXaNkgRiaumItanuQ+E6EfZtsuQ=', 'smithi091.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKACiH2/ZV+aC13a4lw3tv6KdmbuvnXSsmKFzugeTLDBQHhxcPZCUScqyjU02uLKZMsAiDzBiR/NEjXhmt//V1Q=', 'smithi150.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDkiF69tNRQpFmCKo4Q2yleP7Jc1Pgc2YXFX0TcAZ3/oJIAxt4oc0AJkq+b2psJRwqrnXI9Z/uxNWwNPz7ZO4Fo=', 'smithi196.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOOMfC5VqynyKF0c2ZrwRiL5Aavhag9Mt0amEBga9yehMArhs/XBirD15Vfk+fhSedG28dlp3w/xd36KPqL4jPg='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.1, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject socket failures': 2500, 'ms tcp read timeout': 5, 'osd_async_recovery_min_cost': 1, 'osd_object_clean_region_max_num_intervals': 1000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from'], 'sha1': 'd4ed20cfb238ef2c47d64ca4533eca00b8968b98', 'cluster': 'ceph'}}, {'thrashosds': {'bluestore_new_sharding': 'random', 'chance_bluestore_reshard': 1, 'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'min_in': 8, 'timeout': 1200}}, {'rados': {'clients': ['client.0'], 'ec_pool': True, 'erasure_code_profile': {'c': 2, 'crush-failure-domain': 'osd', 'k': 4, 'm': 3, 'name': 'shecprofile', 'plugin': 'shec'}, 'objects': 50, 'op_weights': {'append': 100, 'copy_from': 50, 'delete': 50, 'read': 100, 'rmattr': 25, 'rollback': 50, 'setattr': 25, 'snap_create': 50, 'snap_remove': 50, 'write': 0}, 'ops': 400, 'write_append_excl': False}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'd9fdb2209e15b39d9f061fd85399f352ce0f0894', 'timestamp': '2024-01-31_07:15:30', 'tube': 'smithi', 'user': 'nmordech', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-02-05T04:51:41.303 DEBUG:teuthology.orchestra.run.smithi062:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-02-05T04:51:41.321 DEBUG:teuthology.orchestra.run.smithi091:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-02-05T04:51:41.324 DEBUG:teuthology.orchestra.run.smithi150:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-02-05T04:51:41.328 DEBUG:teuthology.orchestra.run.smithi196:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-02-05T04:51:41.344 DEBUG:teuthology.orchestra.run.smithi062:> sudo install -d -m0777 -- /var/run/ceph 2024-02-05T04:51:41.376 DEBUG:teuthology.orchestra.run.smithi091:> sudo install -d -m0777 -- /var/run/ceph 2024-02-05T04:51:41.379 DEBUG:teuthology.orchestra.run.smithi150:> sudo install -d -m0777 -- /var/run/ceph 2024-02-05T04:51:41.383 DEBUG:teuthology.orchestra.run.smithi196:> sudo install -d -m0777 -- /var/run/ceph 2024-02-05T04:51:41.412 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-02-05T04:51:41.412 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/scratch_devs of=/dev/stdout 2024-02-05T04:51:41.456 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-05T04:51:41.456 DEBUG:teuthology.orchestra.run.smithi062:> stat /dev/vg_nvme/lv_1 2024-02-05T04:51:41.510 INFO:teuthology.orchestra.run.smithi062.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-05T04:51:41.510 INFO:teuthology.orchestra.run.smithi062.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-05T04:51:41.510 INFO:teuthology.orchestra.run.smithi062.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-02-05T04:51:41.510 INFO:teuthology.orchestra.run.smithi062.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-05T04:51:41.510 INFO:teuthology.orchestra.run.smithi062.stdout:Context: system_u:object_r:device_t:s0 2024-02-05T04:51:41.510 INFO:teuthology.orchestra.run.smithi062.stdout:Access: 2024-02-05 04:51:02.006123093 +0000 2024-02-05T04:51:41.511 INFO:teuthology.orchestra.run.smithi062.stdout:Modify: 2024-02-05 04:48:14.225077329 +0000 2024-02-05T04:51:41.511 INFO:teuthology.orchestra.run.smithi062.stdout:Change: 2024-02-05 04:48:14.225077329 +0000 2024-02-05T04:51:41.511 INFO:teuthology.orchestra.run.smithi062.stdout: Birth: 2024-02-05 04:48:14.225077329 +0000 2024-02-05T04:51:41.511 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-05T04:51:41.576 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records in 2024-02-05T04:51:41.576 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records out 2024-02-05T04:51:41.576 INFO:teuthology.orchestra.run.smithi062.stderr:512 bytes copied, 0.000250098 s, 2.0 MB/s 2024-02-05T04:51:41.577 DEBUG:teuthology.orchestra.run.smithi062:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-05T04:51:41.633 DEBUG:teuthology.orchestra.run.smithi062:> stat /dev/vg_nvme/lv_2 2024-02-05T04:51:41.687 INFO:teuthology.orchestra.run.smithi062.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-05T04:51:41.687 INFO:teuthology.orchestra.run.smithi062.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-05T04:51:41.687 INFO:teuthology.orchestra.run.smithi062.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-02-05T04:51:41.687 INFO:teuthology.orchestra.run.smithi062.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-05T04:51:41.687 INFO:teuthology.orchestra.run.smithi062.stdout:Context: system_u:object_r:device_t:s0 2024-02-05T04:51:41.687 INFO:teuthology.orchestra.run.smithi062.stdout:Access: 2024-02-05 04:51:02.007123076 +0000 2024-02-05T04:51:41.687 INFO:teuthology.orchestra.run.smithi062.stdout:Modify: 2024-02-05 04:48:14.571071452 +0000 2024-02-05T04:51:41.687 INFO:teuthology.orchestra.run.smithi062.stdout:Change: 2024-02-05 04:48:14.571071452 +0000 2024-02-05T04:51:41.687 INFO:teuthology.orchestra.run.smithi062.stdout: Birth: 2024-02-05 04:48:14.571071452 +0000 2024-02-05T04:51:41.688 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-05T04:51:41.751 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records in 2024-02-05T04:51:41.752 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records out 2024-02-05T04:51:41.752 INFO:teuthology.orchestra.run.smithi062.stderr:512 bytes copied, 0.000192081 s, 2.7 MB/s 2024-02-05T04:51:41.753 DEBUG:teuthology.orchestra.run.smithi062:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-05T04:51:41.807 DEBUG:teuthology.orchestra.run.smithi062:> stat /dev/vg_nvme/lv_3 2024-02-05T04:51:41.862 INFO:teuthology.orchestra.run.smithi062.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-05T04:51:41.862 INFO:teuthology.orchestra.run.smithi062.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-05T04:51:41.862 INFO:teuthology.orchestra.run.smithi062.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-02-05T04:51:41.862 INFO:teuthology.orchestra.run.smithi062.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-05T04:51:41.862 INFO:teuthology.orchestra.run.smithi062.stdout:Context: system_u:object_r:device_t:s0 2024-02-05T04:51:41.862 INFO:teuthology.orchestra.run.smithi062.stdout:Access: 2024-02-05 04:51:02.007123076 +0000 2024-02-05T04:51:41.862 INFO:teuthology.orchestra.run.smithi062.stdout:Modify: 2024-02-05 04:48:14.907065744 +0000 2024-02-05T04:51:41.862 INFO:teuthology.orchestra.run.smithi062.stdout:Change: 2024-02-05 04:48:14.907065744 +0000 2024-02-05T04:51:41.862 INFO:teuthology.orchestra.run.smithi062.stdout: Birth: 2024-02-05 04:48:14.907065744 +0000 2024-02-05T04:51:41.863 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-05T04:51:41.926 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records in 2024-02-05T04:51:41.927 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records out 2024-02-05T04:51:41.927 INFO:teuthology.orchestra.run.smithi062.stderr:512 bytes copied, 0.000184638 s, 2.8 MB/s 2024-02-05T04:51:41.927 DEBUG:teuthology.orchestra.run.smithi062:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-05T04:51:41.983 DEBUG:teuthology.orchestra.run.smithi062:> stat /dev/vg_nvme/lv_4 2024-02-05T04:51:42.038 INFO:teuthology.orchestra.run.smithi062.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-05T04:51:42.038 INFO:teuthology.orchestra.run.smithi062.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-05T04:51:42.038 INFO:teuthology.orchestra.run.smithi062.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-02-05T04:51:42.038 INFO:teuthology.orchestra.run.smithi062.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-05T04:51:42.038 INFO:teuthology.orchestra.run.smithi062.stdout:Context: system_u:object_r:device_t:s0 2024-02-05T04:51:42.038 INFO:teuthology.orchestra.run.smithi062.stdout:Access: 2024-02-05 04:51:02.007123076 +0000 2024-02-05T04:51:42.038 INFO:teuthology.orchestra.run.smithi062.stdout:Modify: 2024-02-05 04:48:15.259059765 +0000 2024-02-05T04:51:42.039 INFO:teuthology.orchestra.run.smithi062.stdout:Change: 2024-02-05 04:48:15.259059765 +0000 2024-02-05T04:51:42.039 INFO:teuthology.orchestra.run.smithi062.stdout: Birth: 2024-02-05 04:48:15.259059765 +0000 2024-02-05T04:51:42.039 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-05T04:51:42.103 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records in 2024-02-05T04:51:42.103 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records out 2024-02-05T04:51:42.103 INFO:teuthology.orchestra.run.smithi062.stderr:512 bytes copied, 0.000161192 s, 3.2 MB/s 2024-02-05T04:51:42.104 DEBUG:teuthology.orchestra.run.smithi062:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-05T04:51:42.158 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-02-05T04:51:42.159 DEBUG:teuthology.orchestra.run.smithi091:> set -ex 2024-02-05T04:51:42.159 DEBUG:teuthology.orchestra.run.smithi091:> dd if=/scratch_devs of=/dev/stdout 2024-02-05T04:51:42.177 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-05T04:51:42.177 DEBUG:teuthology.orchestra.run.smithi091:> stat /dev/vg_nvme/lv_1 2024-02-05T04:51:42.234 INFO:teuthology.orchestra.run.smithi091.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-05T04:51:42.234 INFO:teuthology.orchestra.run.smithi091.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-05T04:51:42.234 INFO:teuthology.orchestra.run.smithi091.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-02-05T04:51:42.234 INFO:teuthology.orchestra.run.smithi091.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-05T04:51:42.234 INFO:teuthology.orchestra.run.smithi091.stdout:Context: system_u:object_r:device_t:s0 2024-02-05T04:51:42.234 INFO:teuthology.orchestra.run.smithi091.stdout:Access: 2024-02-05 04:51:03.203772526 +0000 2024-02-05T04:51:42.234 INFO:teuthology.orchestra.run.smithi091.stdout:Modify: 2024-02-05 04:48:12.704947185 +0000 2024-02-05T04:51:42.234 INFO:teuthology.orchestra.run.smithi091.stdout:Change: 2024-02-05 04:48:12.704947185 +0000 2024-02-05T04:51:42.234 INFO:teuthology.orchestra.run.smithi091.stdout: Birth: 2024-02-05 04:48:12.704947185 +0000 2024-02-05T04:51:42.235 DEBUG:teuthology.orchestra.run.smithi091:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-05T04:51:42.303 INFO:teuthology.orchestra.run.smithi091.stderr:1+0 records in 2024-02-05T04:51:42.303 INFO:teuthology.orchestra.run.smithi091.stderr:1+0 records out 2024-02-05T04:51:42.303 INFO:teuthology.orchestra.run.smithi091.stderr:512 bytes copied, 0.000213303 s, 2.4 MB/s 2024-02-05T04:51:42.304 DEBUG:teuthology.orchestra.run.smithi091:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-05T04:51:42.363 DEBUG:teuthology.orchestra.run.smithi091:> stat /dev/vg_nvme/lv_2 2024-02-05T04:51:42.419 INFO:teuthology.orchestra.run.smithi091.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-05T04:51:42.419 INFO:teuthology.orchestra.run.smithi091.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-05T04:51:42.419 INFO:teuthology.orchestra.run.smithi091.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-02-05T04:51:42.419 INFO:teuthology.orchestra.run.smithi091.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-05T04:51:42.419 INFO:teuthology.orchestra.run.smithi091.stdout:Context: system_u:object_r:device_t:s0 2024-02-05T04:51:42.419 INFO:teuthology.orchestra.run.smithi091.stdout:Access: 2024-02-05 04:51:03.204772501 +0000 2024-02-05T04:51:42.419 INFO:teuthology.orchestra.run.smithi091.stdout:Modify: 2024-02-05 04:48:13.045938519 +0000 2024-02-05T04:51:42.419 INFO:teuthology.orchestra.run.smithi091.stdout:Change: 2024-02-05 04:48:13.045938519 +0000 2024-02-05T04:51:42.419 INFO:teuthology.orchestra.run.smithi091.stdout: Birth: 2024-02-05 04:48:13.045938519 +0000 2024-02-05T04:51:42.420 DEBUG:teuthology.orchestra.run.smithi091:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-05T04:51:42.488 INFO:teuthology.orchestra.run.smithi091.stderr:1+0 records in 2024-02-05T04:51:42.488 INFO:teuthology.orchestra.run.smithi091.stderr:1+0 records out 2024-02-05T04:51:42.488 INFO:teuthology.orchestra.run.smithi091.stderr:512 bytes copied, 0.000215278 s, 2.4 MB/s 2024-02-05T04:51:42.489 DEBUG:teuthology.orchestra.run.smithi091:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-05T04:51:42.546 DEBUG:teuthology.orchestra.run.smithi091:> stat /dev/vg_nvme/lv_3 2024-02-05T04:51:42.602 INFO:teuthology.orchestra.run.smithi091.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-05T04:51:42.603 INFO:teuthology.orchestra.run.smithi091.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-05T04:51:42.603 INFO:teuthology.orchestra.run.smithi091.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-02-05T04:51:42.603 INFO:teuthology.orchestra.run.smithi091.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-05T04:51:42.603 INFO:teuthology.orchestra.run.smithi091.stdout:Context: system_u:object_r:device_t:s0 2024-02-05T04:51:42.603 INFO:teuthology.orchestra.run.smithi091.stdout:Access: 2024-02-05 04:51:03.204772501 +0000 2024-02-05T04:51:42.603 INFO:teuthology.orchestra.run.smithi091.stdout:Modify: 2024-02-05 04:48:13.370930259 +0000 2024-02-05T04:51:42.603 INFO:teuthology.orchestra.run.smithi091.stdout:Change: 2024-02-05 04:48:13.370930259 +0000 2024-02-05T04:51:42.603 INFO:teuthology.orchestra.run.smithi091.stdout: Birth: 2024-02-05 04:48:13.370930259 +0000 2024-02-05T04:51:42.603 DEBUG:teuthology.orchestra.run.smithi091:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-05T04:51:42.670 INFO:teuthology.orchestra.run.smithi091.stderr:1+0 records in 2024-02-05T04:51:42.671 INFO:teuthology.orchestra.run.smithi091.stderr:1+0 records out 2024-02-05T04:51:42.671 INFO:teuthology.orchestra.run.smithi091.stderr:512 bytes copied, 0.000213852 s, 2.4 MB/s 2024-02-05T04:51:42.672 DEBUG:teuthology.orchestra.run.smithi091:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-05T04:51:42.729 DEBUG:teuthology.orchestra.run.smithi091:> stat /dev/vg_nvme/lv_4 2024-02-05T04:51:42.786 INFO:teuthology.orchestra.run.smithi091.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-05T04:51:42.786 INFO:teuthology.orchestra.run.smithi091.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-05T04:51:42.786 INFO:teuthology.orchestra.run.smithi091.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-02-05T04:51:42.786 INFO:teuthology.orchestra.run.smithi091.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-05T04:51:42.786 INFO:teuthology.orchestra.run.smithi091.stdout:Context: system_u:object_r:device_t:s0 2024-02-05T04:51:42.787 INFO:teuthology.orchestra.run.smithi091.stdout:Access: 2024-02-05 04:51:03.205772475 +0000 2024-02-05T04:51:42.787 INFO:teuthology.orchestra.run.smithi091.stdout:Modify: 2024-02-05 04:48:13.708921669 +0000 2024-02-05T04:51:42.787 INFO:teuthology.orchestra.run.smithi091.stdout:Change: 2024-02-05 04:48:13.708921669 +0000 2024-02-05T04:51:42.787 INFO:teuthology.orchestra.run.smithi091.stdout: Birth: 2024-02-05 04:48:13.708921669 +0000 2024-02-05T04:51:42.787 DEBUG:teuthology.orchestra.run.smithi091:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-05T04:51:42.853 INFO:teuthology.orchestra.run.smithi091.stderr:1+0 records in 2024-02-05T04:51:42.853 INFO:teuthology.orchestra.run.smithi091.stderr:1+0 records out 2024-02-05T04:51:42.853 INFO:teuthology.orchestra.run.smithi091.stderr:512 bytes copied, 0.00019224 s, 2.7 MB/s 2024-02-05T04:51:42.854 DEBUG:teuthology.orchestra.run.smithi091:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-05T04:51:42.911 INFO:tasks.ceph:osd dev map: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-02-05T04:51:42.911 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-02-05T04:51:42.911 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/scratch_devs of=/dev/stdout 2024-02-05T04:51:42.930 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-05T04:51:42.930 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_1 2024-02-05T04:51:42.988 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-05T04:51:42.988 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-05T04:51:42.988 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-02-05T04:51:42.988 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-05T04:51:42.989 INFO:teuthology.orchestra.run.smithi150.stdout:Context: system_u:object_r:device_t:s0 2024-02-05T04:51:42.989 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2024-02-05 04:51:35.425222389 +0000 2024-02-05T04:51:42.989 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2024-02-05 04:48:14.097991522 +0000 2024-02-05T04:51:42.989 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2024-02-05 04:48:14.097991522 +0000 2024-02-05T04:51:42.989 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: 2024-02-05 04:48:14.097991522 +0000 2024-02-05T04:51:42.989 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-05T04:51:43.058 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2024-02-05T04:51:43.058 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2024-02-05T04:51:43.058 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.000254656 s, 2.0 MB/s 2024-02-05T04:51:43.059 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-05T04:51:43.118 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_2 2024-02-05T04:51:43.175 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-05T04:51:43.175 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-05T04:51:43.175 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-02-05T04:51:43.175 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-05T04:51:43.175 INFO:teuthology.orchestra.run.smithi150.stdout:Context: system_u:object_r:device_t:s0 2024-02-05T04:51:43.176 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2024-02-05 04:51:35.426222370 +0000 2024-02-05T04:51:43.176 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2024-02-05 04:48:14.425985258 +0000 2024-02-05T04:51:43.176 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2024-02-05 04:48:14.425985258 +0000 2024-02-05T04:51:43.176 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: 2024-02-05 04:48:14.425985258 +0000 2024-02-05T04:51:43.176 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-05T04:51:43.244 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2024-02-05T04:51:43.244 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2024-02-05T04:51:43.244 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.000204534 s, 2.5 MB/s 2024-02-05T04:51:43.245 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-05T04:51:43.303 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_3 2024-02-05T04:51:43.361 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-05T04:51:43.361 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-05T04:51:43.361 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-02-05T04:51:43.361 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-05T04:51:43.361 INFO:teuthology.orchestra.run.smithi150.stdout:Context: system_u:object_r:device_t:s0 2024-02-05T04:51:43.361 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2024-02-05 04:51:35.426222370 +0000 2024-02-05T04:51:43.361 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2024-02-05 04:48:14.765978764 +0000 2024-02-05T04:51:43.361 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2024-02-05 04:48:14.765978764 +0000 2024-02-05T04:51:43.361 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: 2024-02-05 04:48:14.765978764 +0000 2024-02-05T04:51:43.362 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-05T04:51:43.429 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2024-02-05T04:51:43.429 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2024-02-05T04:51:43.429 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.000213524 s, 2.4 MB/s 2024-02-05T04:51:43.430 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-05T04:51:43.488 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_4 2024-02-05T04:51:43.544 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-05T04:51:43.544 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-05T04:51:43.544 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-02-05T04:51:43.544 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-05T04:51:43.544 INFO:teuthology.orchestra.run.smithi150.stdout:Context: system_u:object_r:device_t:s0 2024-02-05T04:51:43.544 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2024-02-05 04:51:35.426222370 +0000 2024-02-05T04:51:43.544 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2024-02-05 04:48:15.107972232 +0000 2024-02-05T04:51:43.544 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2024-02-05 04:48:15.107972232 +0000 2024-02-05T04:51:43.544 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: 2024-02-05 04:48:15.107972232 +0000 2024-02-05T04:51:43.545 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-05T04:51:43.611 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2024-02-05T04:51:43.611 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2024-02-05T04:51:43.611 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.000168597 s, 3.0 MB/s 2024-02-05T04:51:43.612 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-05T04:51:43.668 INFO:tasks.ceph:osd dev map: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-02-05T04:51:43.669 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-02-05T04:51:43.669 DEBUG:teuthology.orchestra.run.smithi196:> dd if=/scratch_devs of=/dev/stdout 2024-02-05T04:51:43.687 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-05T04:51:43.687 DEBUG:teuthology.orchestra.run.smithi196:> stat /dev/vg_nvme/lv_1 2024-02-05T04:51:43.745 INFO:teuthology.orchestra.run.smithi196.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-05T04:51:43.745 INFO:teuthology.orchestra.run.smithi196.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-05T04:51:43.745 INFO:teuthology.orchestra.run.smithi196.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-02-05T04:51:43.745 INFO:teuthology.orchestra.run.smithi196.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-05T04:51:43.745 INFO:teuthology.orchestra.run.smithi196.stdout:Context: system_u:object_r:device_t:s0 2024-02-05T04:51:43.745 INFO:teuthology.orchestra.run.smithi196.stdout:Access: 2024-02-05 04:50:50.335690094 +0000 2024-02-05T04:51:43.745 INFO:teuthology.orchestra.run.smithi196.stdout:Modify: 2024-02-05 04:48:14.480228041 +0000 2024-02-05T04:51:43.745 INFO:teuthology.orchestra.run.smithi196.stdout:Change: 2024-02-05 04:48:14.480228041 +0000 2024-02-05T04:51:43.746 INFO:teuthology.orchestra.run.smithi196.stdout: Birth: 2024-02-05 04:48:14.480228041 +0000 2024-02-05T04:51:43.746 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-05T04:51:43.814 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records in 2024-02-05T04:51:43.814 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records out 2024-02-05T04:51:43.814 INFO:teuthology.orchestra.run.smithi196.stderr:512 bytes copied, 0.000279539 s, 1.8 MB/s 2024-02-05T04:51:43.815 DEBUG:teuthology.orchestra.run.smithi196:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-05T04:51:43.873 DEBUG:teuthology.orchestra.run.smithi196:> stat /dev/vg_nvme/lv_2 2024-02-05T04:51:43.930 INFO:teuthology.orchestra.run.smithi196.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-05T04:51:43.930 INFO:teuthology.orchestra.run.smithi196.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-05T04:51:43.930 INFO:teuthology.orchestra.run.smithi196.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-02-05T04:51:43.931 INFO:teuthology.orchestra.run.smithi196.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-05T04:51:43.931 INFO:teuthology.orchestra.run.smithi196.stdout:Context: system_u:object_r:device_t:s0 2024-02-05T04:51:43.931 INFO:teuthology.orchestra.run.smithi196.stdout:Access: 2024-02-05 04:50:50.335690094 +0000 2024-02-05T04:51:43.931 INFO:teuthology.orchestra.run.smithi196.stdout:Modify: 2024-02-05 04:48:14.818220437 +0000 2024-02-05T04:51:43.931 INFO:teuthology.orchestra.run.smithi196.stdout:Change: 2024-02-05 04:48:14.818220437 +0000 2024-02-05T04:51:43.931 INFO:teuthology.orchestra.run.smithi196.stdout: Birth: 2024-02-05 04:48:14.818220437 +0000 2024-02-05T04:51:43.931 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-05T04:51:43.999 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records in 2024-02-05T04:51:43.999 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records out 2024-02-05T04:51:43.999 INFO:teuthology.orchestra.run.smithi196.stderr:512 bytes copied, 0.000280644 s, 1.8 MB/s 2024-02-05T04:51:44.001 DEBUG:teuthology.orchestra.run.smithi196:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-05T04:51:44.058 DEBUG:teuthology.orchestra.run.smithi196:> stat /dev/vg_nvme/lv_3 2024-02-05T04:51:44.113 INFO:teuthology.orchestra.run.smithi196.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-05T04:51:44.114 INFO:teuthology.orchestra.run.smithi196.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-05T04:51:44.114 INFO:teuthology.orchestra.run.smithi196.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-02-05T04:51:44.114 INFO:teuthology.orchestra.run.smithi196.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-05T04:51:44.114 INFO:teuthology.orchestra.run.smithi196.stdout:Context: system_u:object_r:device_t:s0 2024-02-05T04:51:44.114 INFO:teuthology.orchestra.run.smithi196.stdout:Access: 2024-02-05 04:50:50.336690072 +0000 2024-02-05T04:51:44.114 INFO:teuthology.orchestra.run.smithi196.stdout:Modify: 2024-02-05 04:48:15.157212811 +0000 2024-02-05T04:51:44.114 INFO:teuthology.orchestra.run.smithi196.stdout:Change: 2024-02-05 04:48:15.157212811 +0000 2024-02-05T04:51:44.114 INFO:teuthology.orchestra.run.smithi196.stdout: Birth: 2024-02-05 04:48:15.157212811 +0000 2024-02-05T04:51:44.114 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-05T04:51:44.180 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records in 2024-02-05T04:51:44.181 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records out 2024-02-05T04:51:44.181 INFO:teuthology.orchestra.run.smithi196.stderr:512 bytes copied, 0.000210577 s, 2.4 MB/s 2024-02-05T04:51:44.182 DEBUG:teuthology.orchestra.run.smithi196:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-05T04:51:44.238 DEBUG:teuthology.orchestra.run.smithi196:> stat /dev/vg_nvme/lv_4 2024-02-05T04:51:44.293 INFO:teuthology.orchestra.run.smithi196.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-05T04:51:44.293 INFO:teuthology.orchestra.run.smithi196.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-05T04:51:44.293 INFO:teuthology.orchestra.run.smithi196.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-02-05T04:51:44.293 INFO:teuthology.orchestra.run.smithi196.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-05T04:51:44.293 INFO:teuthology.orchestra.run.smithi196.stdout:Context: system_u:object_r:device_t:s0 2024-02-05T04:51:44.293 INFO:teuthology.orchestra.run.smithi196.stdout:Access: 2024-02-05 04:50:50.336690072 +0000 2024-02-05T04:51:44.293 INFO:teuthology.orchestra.run.smithi196.stdout:Modify: 2024-02-05 04:48:15.514204779 +0000 2024-02-05T04:51:44.293 INFO:teuthology.orchestra.run.smithi196.stdout:Change: 2024-02-05 04:48:15.514204779 +0000 2024-02-05T04:51:44.293 INFO:teuthology.orchestra.run.smithi196.stdout: Birth: 2024-02-05 04:48:15.514204779 +0000 2024-02-05T04:51:44.293 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-05T04:51:44.359 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records in 2024-02-05T04:51:44.359 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records out 2024-02-05T04:51:44.359 INFO:teuthology.orchestra.run.smithi196.stderr:512 bytes copied, 0.000189438 s, 2.7 MB/s 2024-02-05T04:51:44.360 DEBUG:teuthology.orchestra.run.smithi196:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-05T04:51:44.416 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-02-05T04:51:44.416 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi062.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi091.front.sepia.ceph.com'): {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi150.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi196.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'}} 2024-02-05T04:51:44.416 INFO:tasks.ceph:Generating config... 2024-02-05T04:51:44.417 INFO:tasks.ceph:[osd] debug monc = 20 2024-02-05T04:51:44.417 INFO:tasks.ceph:[osd] bdev async discard = True 2024-02-05T04:51:44.417 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-02-05T04:51:44.417 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-02-05T04:51:44.418 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-02-05T04:51:44.418 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-02-05T04:51:44.418 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-02-05T04:51:44.418 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-02-05T04:51:44.418 INFO:tasks.ceph:[osd] debug ms = 1 2024-02-05T04:51:44.418 INFO:tasks.ceph:[osd] debug osd = 20 2024-02-05T04:51:44.418 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-02-05T04:51:44.418 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-02-05T04:51:44.418 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-02-05T04:51:44.418 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-02-05T04:51:44.418 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-02-05T04:51:44.418 INFO:tasks.ceph:[osd] osd debug reject backfill probability = 0.1 2024-02-05T04:51:44.418 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-02-05T04:51:44.418 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-02-05T04:51:44.418 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-02-05T04:51:44.419 INFO:tasks.ceph:[osd] osd max backfills = 3 2024-02-05T04:51:44.419 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-02-05T04:51:44.419 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-02-05T04:51:44.419 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-02-05T04:51:44.419 INFO:tasks.ceph:[osd] osd memory target = 1610612736 2024-02-05T04:51:44.419 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-02-05T04:51:44.419 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-02-05T04:51:44.419 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-02-05T04:51:44.419 INFO:tasks.ceph:[osd] osd scrub max interval = 120 2024-02-05T04:51:44.419 INFO:tasks.ceph:[osd] osd scrub min interval = 60 2024-02-05T04:51:44.419 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-02-05T04:51:44.419 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-02-05T04:51:44.419 INFO:tasks.ceph:[global] mon election default strategy = 3 2024-02-05T04:51:44.419 INFO:tasks.ceph:[global] ms inject socket failures = 2500 2024-02-05T04:51:44.419 INFO:tasks.ceph:[global] ms tcp read timeout = 5 2024-02-05T04:51:44.419 INFO:tasks.ceph:[global] osd_async_recovery_min_cost = 1 2024-02-05T04:51:44.420 INFO:tasks.ceph:[global] osd_object_clean_region_max_num_intervals = 1000 2024-02-05T04:51:44.420 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-02-05T04:51:44.420 INFO:tasks.ceph:[mgr] debug ms = 1 2024-02-05T04:51:44.420 INFO:tasks.ceph:[mon] debug mon = 20 2024-02-05T04:51:44.420 INFO:tasks.ceph:[mon] debug ms = 1 2024-02-05T04:51:44.420 INFO:tasks.ceph:[mon] debug paxos = 20 2024-02-05T04:51:44.420 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-02-05T04:51:44.420 INFO:tasks.ceph:Setting up mon.a... 2024-02-05T04:51:44.420 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-02-05T04:51:44.583 INFO:teuthology.orchestra.run.smithi062.stdout:creating /etc/ceph/ceph.keyring 2024-02-05T04:51:44.586 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-02-05T04:51:44.634 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-02-05T04:51:44.662 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.62'), ('mon.b', '172.21.15.91'), ('mon.c', '172.21.15.150')] 2024-02-05T04:51:44.663 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.62,172.21.15.91,172.21.15.150', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject socket failures': 2500, 'ms tcp read timeout': 5, 'osd_async_recovery_min_cost': 1, 'osd_object_clean_region_max_num_intervals': 1000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': 120, 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.1, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon scrub interval': 300}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}} 2024-02-05T04:51:44.664 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-02-05T04:51:44.665 DEBUG:teuthology.orchestra.run.smithi062:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-02-05T04:51:44.718 DEBUG:teuthology.orchestra.run.smithi062:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.62 --add b 172.21.15.91 --add c 172.21.15.150 --print /home/ubuntu/cephtest/ceph.monmap 2024-02-05T04:51:44.803 INFO:teuthology.orchestra.run.smithi062.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-02-05T04:51:44.803 INFO:teuthology.orchestra.run.smithi062.stdout:monmaptool: generated fsid 88c11a2d-382e-4f21-b8af-45ab9c504b93 2024-02-05T04:51:44.803 INFO:teuthology.orchestra.run.smithi062.stdout:setting min_mon_release = quincy 2024-02-05T04:51:44.803 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 0 2024-02-05T04:51:44.803 INFO:teuthology.orchestra.run.smithi062.stdout:fsid 88c11a2d-382e-4f21-b8af-45ab9c504b93 2024-02-05T04:51:44.803 INFO:teuthology.orchestra.run.smithi062.stdout:last_changed 2024-02-05T04:51:44.803005+0000 2024-02-05T04:51:44.803 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-02-05T04:51:44.803005+0000 2024-02-05T04:51:44.803 INFO:teuthology.orchestra.run.smithi062.stdout:min_mon_release 17 (quincy) 2024-02-05T04:51:44.803 INFO:teuthology.orchestra.run.smithi062.stdout:election_strategy: 3 2024-02-05T04:51:44.805 INFO:teuthology.orchestra.run.smithi062.stdout:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-02-05T04:51:44.806 INFO:teuthology.orchestra.run.smithi062.stdout:1: [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] mon.b 2024-02-05T04:51:44.806 INFO:teuthology.orchestra.run.smithi062.stdout:2: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.c 2024-02-05T04:51:44.806 INFO:teuthology.orchestra.run.smithi062.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-02-05T04:51:44.807 DEBUG:teuthology.orchestra.run.smithi062:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-02-05T04:51:44.821 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 88c11a2d-382e-4f21-b8af-45ab9c504b93... 2024-02-05T04:51:44.822 DEBUG:teuthology.orchestra.run.smithi062:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-02-05T04:51:44.864 DEBUG:teuthology.orchestra.run.smithi091:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-02-05T04:51:44.866 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-02-05T04:51:44.869 DEBUG:teuthology.orchestra.run.smithi196:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-02-05T04:51:44.912 INFO:teuthology.orchestra.run.smithi062.stdout:[global] 2024-02-05T04:51:44.912 INFO:teuthology.orchestra.run.smithi062.stdout: chdir = "" 2024-02-05T04:51:44.912 INFO:teuthology.orchestra.run.smithi062.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-02-05T04:51:44.912 INFO:teuthology.orchestra.run.smithi062.stdout: auth supported = cephx 2024-02-05T04:51:44.912 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:51:44.912 INFO:teuthology.orchestra.run.smithi062.stdout: filestore xattr use omap = true 2024-02-05T04:51:44.912 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:51:44.912 INFO:teuthology.orchestra.run.smithi062.stdout: mon clock drift allowed = 1.000 2024-02-05T04:51:44.913 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:51:44.913 INFO:teuthology.orchestra.run.smithi062.stdout: osd crush chooseleaf type = 0 2024-02-05T04:51:44.913 INFO:teuthology.orchestra.run.smithi062.stdout: auth debug = true 2024-02-05T04:51:44.913 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:51:44.913 INFO:teuthology.orchestra.run.smithi062.stdout: ms die on old message = true 2024-02-05T04:51:44.913 INFO:teuthology.orchestra.run.smithi062.stdout: ms die on bug = true 2024-02-05T04:51:44.913 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:51:44.913 INFO:teuthology.orchestra.run.smithi062.stdout: mon max pg per osd = 10000 # >= luminous 2024-02-05T04:51:44.913 INFO:teuthology.orchestra.run.smithi062.stdout: mon pg warn max object skew = 0 2024-02-05T04:51:44.913 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:51:44.913 INFO:teuthology.orchestra.run.smithi062.stdout: # disable pg_autoscaler by default for new pools 2024-02-05T04:51:44.913 INFO:teuthology.orchestra.run.smithi062.stdout: osd_pool_default_pg_autoscale_mode = off 2024-02-05T04:51:44.913 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:51:44.914 INFO:teuthology.orchestra.run.smithi062.stdout: osd pool default size = 2 2024-02-05T04:51:44.914 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:51:44.914 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd allow primary affinity = true 2024-02-05T04:51:44.914 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd allow pg remap = true 2024-02-05T04:51:44.914 INFO:teuthology.orchestra.run.smithi062.stdout: mon warn on legacy crush tunables = false 2024-02-05T04:51:44.914 INFO:teuthology.orchestra.run.smithi062.stdout: mon warn on crush straw calc version zero = false 2024-02-05T04:51:44.914 INFO:teuthology.orchestra.run.smithi062.stdout: mon warn on no sortbitwise = false 2024-02-05T04:51:44.914 INFO:teuthology.orchestra.run.smithi062.stdout: mon warn on osd down out interval zero = false 2024-02-05T04:51:44.914 INFO:teuthology.orchestra.run.smithi062.stdout: mon warn on too few osds = false 2024-02-05T04:51:44.914 INFO:teuthology.orchestra.run.smithi062.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-02-05T04:51:44.914 INFO:teuthology.orchestra.run.smithi062.stdout: mon_warn_on_pool_no_redundancy = false 2024-02-05T04:51:44.914 INFO:teuthology.orchestra.run.smithi062.stdout: mon_allow_pool_size_one = true 2024-02-05T04:51:44.914 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:51:44.914 INFO:teuthology.orchestra.run.smithi062.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-02-05T04:51:44.915 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:51:44.915 INFO:teuthology.orchestra.run.smithi062.stdout: osd default data pool replay window = 5 2024-02-05T04:51:44.915 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:51:44.915 INFO:teuthology.orchestra.run.smithi062.stdout: mon allow pool delete = true 2024-02-05T04:51:44.915 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:51:44.915 INFO:teuthology.orchestra.run.smithi062.stdout: mon cluster log file level = debug 2024-02-05T04:51:44.915 INFO:teuthology.orchestra.run.smithi062.stdout: debug asserts on shutdown = true 2024-02-05T04:51:44.915 INFO:teuthology.orchestra.run.smithi062.stdout: mon health detail to clog = false 2024-02-05T04:51:44.915 INFO:teuthology.orchestra.run.smithi062.stdout: mon host = "172.21.15.62,172.21.15.91,172.21.15.150" 2024-02-05T04:51:44.915 INFO:teuthology.orchestra.run.smithi062.stdout: mon client directed command retry = 5 2024-02-05T04:51:44.915 INFO:teuthology.orchestra.run.smithi062.stdout: mon election default strategy = 3 2024-02-05T04:51:44.915 INFO:teuthology.orchestra.run.smithi062.stdout: ms inject socket failures = 2500 2024-02-05T04:51:44.915 INFO:teuthology.orchestra.run.smithi062.stdout: ms tcp read timeout = 5 2024-02-05T04:51:44.915 INFO:teuthology.orchestra.run.smithi062.stdout: osd_async_recovery_min_cost = 1 2024-02-05T04:51:44.915 INFO:teuthology.orchestra.run.smithi062.stdout: osd_object_clean_region_max_num_intervals = 1000 2024-02-05T04:51:44.916 INFO:teuthology.orchestra.run.smithi062.stdout: fsid = 88c11a2d-382e-4f21-b8af-45ab9c504b93 2024-02-05T04:51:44.916 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:51:44.916 INFO:teuthology.orchestra.run.smithi062.stdout:[osd] 2024-02-05T04:51:44.916 INFO:teuthology.orchestra.run.smithi062.stdout: osd journal size = 100 2024-02-05T04:51:44.916 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:51:44.916 INFO:teuthology.orchestra.run.smithi062.stdout: osd scrub load threshold = 5.0 2024-02-05T04:51:44.916 INFO:teuthology.orchestra.run.smithi062.stdout: osd scrub max interval = 120 2024-02-05T04:51:44.916 INFO:teuthology.orchestra.run.smithi062.stdout: osd mclock profile = high_recovery_ops 2024-02-05T04:51:44.916 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:51:44.916 INFO:teuthology.orchestra.run.smithi062.stdout: osd recover clone overlap = true 2024-02-05T04:51:44.917 INFO:teuthology.orchestra.run.smithi062.stdout: osd recovery max chunk = 1048576 2024-02-05T04:51:44.917 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:51:44.917 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug shutdown = true 2024-02-05T04:51:44.917 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug op order = true 2024-02-05T04:51:44.917 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug verify stray on activate = true 2024-02-05T04:51:44.917 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug trim objects = true 2024-02-05T04:51:44.917 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:51:44.917 INFO:teuthology.orchestra.run.smithi062.stdout: osd open classes on start = true 2024-02-05T04:51:44.917 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug pg log writeout = true 2024-02-05T04:51:44.917 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:51:44.917 INFO:teuthology.orchestra.run.smithi062.stdout: osd deep scrub update digest min age = 30 2024-02-05T04:51:44.917 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:51:44.917 INFO:teuthology.orchestra.run.smithi062.stdout: osd map max advance = 10 2024-02-05T04:51:44.917 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:51:44.917 INFO:teuthology.orchestra.run.smithi062.stdout: journal zero on create = true 2024-02-05T04:51:44.917 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:51:44.918 INFO:teuthology.orchestra.run.smithi062.stdout: filestore ondisk finisher threads = 3 2024-02-05T04:51:44.918 INFO:teuthology.orchestra.run.smithi062.stdout: filestore apply finisher threads = 3 2024-02-05T04:51:44.918 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:51:44.918 INFO:teuthology.orchestra.run.smithi062.stdout: bdev debug aio = true 2024-02-05T04:51:44.918 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug misdirected ops = true 2024-02-05T04:51:44.918 INFO:teuthology.orchestra.run.smithi062.stdout: debug monc = 20 2024-02-05T04:51:44.918 INFO:teuthology.orchestra.run.smithi062.stdout: bdev async discard = True 2024-02-05T04:51:44.918 INFO:teuthology.orchestra.run.smithi062.stdout: bdev enable discard = True 2024-02-05T04:51:44.918 INFO:teuthology.orchestra.run.smithi062.stdout: bluestore block size = 96636764160 2024-02-05T04:51:44.918 INFO:teuthology.orchestra.run.smithi062.stdout: bluestore fsck on mount = True 2024-02-05T04:51:44.918 INFO:teuthology.orchestra.run.smithi062.stdout: bluestore zero block detection = True 2024-02-05T04:51:44.918 INFO:teuthology.orchestra.run.smithi062.stdout: debug bluefs = 20 2024-02-05T04:51:44.918 INFO:teuthology.orchestra.run.smithi062.stdout: debug bluestore = 20 2024-02-05T04:51:44.918 INFO:teuthology.orchestra.run.smithi062.stdout: debug ms = 1 2024-02-05T04:51:44.919 INFO:teuthology.orchestra.run.smithi062.stdout: debug osd = 20 2024-02-05T04:51:44.919 INFO:teuthology.orchestra.run.smithi062.stdout: debug rocksdb = 10 2024-02-05T04:51:44.919 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd backfillfull_ratio = 0.85 2024-02-05T04:51:44.919 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd full ratio = 0.9 2024-02-05T04:51:44.919 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd nearfull ratio = 0.8 2024-02-05T04:51:44.919 INFO:teuthology.orchestra.run.smithi062.stdout: osd blocked scrub grace period = 3600 2024-02-05T04:51:44.919 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug reject backfill probability = 0.1 2024-02-05T04:51:44.919 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug verify cached snaps = True 2024-02-05T04:51:44.919 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug verify missing on start = True 2024-02-05T04:51:44.919 INFO:teuthology.orchestra.run.smithi062.stdout: osd failsafe full ratio = 0.95 2024-02-05T04:51:44.919 INFO:teuthology.orchestra.run.smithi062.stdout: osd max backfills = 3 2024-02-05T04:51:44.919 INFO:teuthology.orchestra.run.smithi062.stdout: osd max markdown count = 1000 2024-02-05T04:51:44.919 INFO:teuthology.orchestra.run.smithi062.stdout: osd mclock override recovery settings = True 2024-02-05T04:51:44.919 INFO:teuthology.orchestra.run.smithi062.stdout: osd memory target = 1610612736 2024-02-05T04:51:44.919 INFO:teuthology.orchestra.run.smithi062.stdout: osd objectstore = bluestore 2024-02-05T04:51:44.920 INFO:teuthology.orchestra.run.smithi062.stdout: osd op queue = debug_random 2024-02-05T04:51:44.920 INFO:teuthology.orchestra.run.smithi062.stdout: osd op queue cut off = debug_random 2024-02-05T04:51:44.920 INFO:teuthology.orchestra.run.smithi062.stdout: osd scrub min interval = 60 2024-02-05T04:51:44.920 INFO:teuthology.orchestra.run.smithi062.stdout: osd shutdown pgref assert = True 2024-02-05T04:51:44.920 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:51:44.920 INFO:teuthology.orchestra.run.smithi062.stdout:[mgr] 2024-02-05T04:51:44.920 INFO:teuthology.orchestra.run.smithi062.stdout: debug ms = 1 2024-02-05T04:51:44.920 INFO:teuthology.orchestra.run.smithi062.stdout: debug mgr = 20 2024-02-05T04:51:44.920 INFO:teuthology.orchestra.run.smithi062.stdout: debug mon = 20 2024-02-05T04:51:44.920 INFO:teuthology.orchestra.run.smithi062.stdout: debug auth = 20 2024-02-05T04:51:44.920 INFO:teuthology.orchestra.run.smithi062.stdout: mon reweight min pgs per osd = 4 2024-02-05T04:51:44.920 INFO:teuthology.orchestra.run.smithi062.stdout: mon reweight min bytes per osd = 10 2024-02-05T04:51:44.920 INFO:teuthology.orchestra.run.smithi062.stdout: mgr/telemetry/nag = false 2024-02-05T04:51:44.920 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:51:44.920 INFO:teuthology.orchestra.run.smithi062.stdout:[mon] 2024-02-05T04:51:44.921 INFO:teuthology.orchestra.run.smithi062.stdout: debug ms = 1 2024-02-05T04:51:44.921 INFO:teuthology.orchestra.run.smithi062.stdout: debug mon = 20 2024-02-05T04:51:44.921 INFO:teuthology.orchestra.run.smithi062.stdout: debug paxos = 20 2024-02-05T04:51:44.921 INFO:teuthology.orchestra.run.smithi062.stdout: debug auth = 20 2024-02-05T04:51:44.921 INFO:teuthology.orchestra.run.smithi062.stdout: mon data avail warn = 5 2024-02-05T04:51:44.921 INFO:teuthology.orchestra.run.smithi062.stdout: mon mgr mkfs grace = 240 2024-02-05T04:51:44.921 INFO:teuthology.orchestra.run.smithi062.stdout: mon reweight min pgs per osd = 4 2024-02-05T04:51:44.921 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd reporter subtree level = osd 2024-02-05T04:51:44.921 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd prime pg temp = true 2024-02-05T04:51:44.921 INFO:teuthology.orchestra.run.smithi062.stdout: mon reweight min bytes per osd = 10 2024-02-05T04:51:44.921 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:51:44.921 INFO:teuthology.orchestra.run.smithi062.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-02-05T04:51:44.921 INFO:teuthology.orchestra.run.smithi062.stdout: auth mon ticket ttl = 660 # 11m 2024-02-05T04:51:44.921 INFO:teuthology.orchestra.run.smithi062.stdout: auth service ticket ttl = 240 # 4m 2024-02-05T04:51:44.921 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:51:44.922 INFO:teuthology.orchestra.run.smithi062.stdout: # don't complain about insecure global_id in the test suite 2024-02-05T04:51:44.922 INFO:teuthology.orchestra.run.smithi062.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-02-05T04:51:44.922 INFO:teuthology.orchestra.run.smithi062.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-02-05T04:51:44.922 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:51:44.922 INFO:teuthology.orchestra.run.smithi062.stdout: # 1m isn't quite enough 2024-02-05T04:51:44.922 INFO:teuthology.orchestra.run.smithi062.stdout: mon_down_mkfs_grace = 2m 2024-02-05T04:51:44.922 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:51:44.922 INFO:teuthology.orchestra.run.smithi062.stdout: mon_warn_on_filestore_osds = false 2024-02-05T04:51:44.922 INFO:teuthology.orchestra.run.smithi062.stdout: mon scrub interval = 300 2024-02-05T04:51:44.922 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:51:44.922 INFO:teuthology.orchestra.run.smithi062.stdout:[client] 2024-02-05T04:51:44.922 INFO:teuthology.orchestra.run.smithi062.stdout: rgw cache enabled = true 2024-02-05T04:51:44.922 INFO:teuthology.orchestra.run.smithi062.stdout: rgw enable ops log = true 2024-02-05T04:51:44.923 INFO:teuthology.orchestra.run.smithi062.stdout: rgw enable usage log = true 2024-02-05T04:51:44.923 INFO:teuthology.orchestra.run.smithi062.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-02-05T04:51:44.923 INFO:teuthology.orchestra.run.smithi062.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-02-05T04:51:44.923 INFO:teuthology.orchestra.run.smithi062.stdout:[mon.a] 2024-02-05T04:51:44.923 INFO:teuthology.orchestra.run.smithi062.stdout:[mon.b] 2024-02-05T04:51:44.923 INFO:teuthology.orchestra.run.smithi062.stdout:[mon.c] 2024-02-05T04:51:44.923 INFO:teuthology.orchestra.run.smithi091.stdout:[global] 2024-02-05T04:51:44.924 INFO:teuthology.orchestra.run.smithi091.stdout: chdir = "" 2024-02-05T04:51:44.924 INFO:teuthology.orchestra.run.smithi091.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-02-05T04:51:44.924 INFO:teuthology.orchestra.run.smithi091.stdout: auth supported = cephx 2024-02-05T04:51:44.924 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:51:44.924 INFO:teuthology.orchestra.run.smithi091.stdout: filestore xattr use omap = true 2024-02-05T04:51:44.924 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:51:44.924 INFO:teuthology.orchestra.run.smithi091.stdout: mon clock drift allowed = 1.000 2024-02-05T04:51:44.924 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:51:44.924 INFO:teuthology.orchestra.run.smithi091.stdout: osd crush chooseleaf type = 0 2024-02-05T04:51:44.924 INFO:teuthology.orchestra.run.smithi091.stdout: auth debug = true 2024-02-05T04:51:44.924 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:51:44.924 INFO:teuthology.orchestra.run.smithi091.stdout: ms die on old message = true 2024-02-05T04:51:44.924 INFO:teuthology.orchestra.run.smithi091.stdout: ms die on bug = true 2024-02-05T04:51:44.925 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:51:44.925 INFO:teuthology.orchestra.run.smithi091.stdout: mon max pg per osd = 10000 # >= luminous 2024-02-05T04:51:44.925 INFO:teuthology.orchestra.run.smithi091.stdout: mon pg warn max object skew = 0 2024-02-05T04:51:44.925 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:51:44.925 INFO:teuthology.orchestra.run.smithi091.stdout: # disable pg_autoscaler by default for new pools 2024-02-05T04:51:44.925 INFO:teuthology.orchestra.run.smithi091.stdout: osd_pool_default_pg_autoscale_mode = off 2024-02-05T04:51:44.925 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:51:44.925 INFO:teuthology.orchestra.run.smithi091.stdout: osd pool default size = 2 2024-02-05T04:51:44.925 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:51:44.925 INFO:teuthology.orchestra.run.smithi091.stdout: mon osd allow primary affinity = true 2024-02-05T04:51:44.925 INFO:teuthology.orchestra.run.smithi091.stdout: mon osd allow pg remap = true 2024-02-05T04:51:44.925 INFO:teuthology.orchestra.run.smithi091.stdout: mon warn on legacy crush tunables = false 2024-02-05T04:51:44.925 INFO:teuthology.orchestra.run.smithi091.stdout: mon warn on crush straw calc version zero = false 2024-02-05T04:51:44.925 INFO:teuthology.orchestra.run.smithi091.stdout: mon warn on no sortbitwise = false 2024-02-05T04:51:44.925 INFO:teuthology.orchestra.run.smithi091.stdout: mon warn on osd down out interval zero = false 2024-02-05T04:51:44.926 INFO:teuthology.orchestra.run.smithi091.stdout: mon warn on too few osds = false 2024-02-05T04:51:44.926 INFO:teuthology.orchestra.run.smithi091.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-02-05T04:51:44.926 INFO:teuthology.orchestra.run.smithi091.stdout: mon_warn_on_pool_no_redundancy = false 2024-02-05T04:51:44.926 INFO:teuthology.orchestra.run.smithi091.stdout: mon_allow_pool_size_one = true 2024-02-05T04:51:44.926 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:51:44.926 INFO:teuthology.orchestra.run.smithi091.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-02-05T04:51:44.926 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:51:44.926 INFO:teuthology.orchestra.run.smithi091.stdout: osd default data pool replay window = 5 2024-02-05T04:51:44.926 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:51:44.926 INFO:teuthology.orchestra.run.smithi091.stdout: mon allow pool delete = true 2024-02-05T04:51:44.926 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:51:44.926 INFO:teuthology.orchestra.run.smithi091.stdout: mon cluster log file level = debug 2024-02-05T04:51:44.927 INFO:teuthology.orchestra.run.smithi091.stdout: debug asserts on shutdown = true 2024-02-05T04:51:44.927 INFO:teuthology.orchestra.run.smithi091.stdout: mon health detail to clog = false 2024-02-05T04:51:44.927 INFO:teuthology.orchestra.run.smithi091.stdout: mon host = "172.21.15.62,172.21.15.91,172.21.15.150" 2024-02-05T04:51:44.927 INFO:teuthology.orchestra.run.smithi091.stdout: mon client directed command retry = 5 2024-02-05T04:51:44.927 INFO:teuthology.orchestra.run.smithi091.stdout: mon election default strategy = 3 2024-02-05T04:51:44.927 INFO:teuthology.orchestra.run.smithi091.stdout: ms inject socket failures = 2500 2024-02-05T04:51:44.927 INFO:teuthology.orchestra.run.smithi091.stdout: ms tcp read timeout = 5 2024-02-05T04:51:44.927 INFO:teuthology.orchestra.run.smithi091.stdout: osd_async_recovery_min_cost = 1 2024-02-05T04:51:44.927 INFO:teuthology.orchestra.run.smithi091.stdout: osd_object_clean_region_max_num_intervals = 1000 2024-02-05T04:51:44.927 INFO:teuthology.orchestra.run.smithi091.stdout: fsid = 88c11a2d-382e-4f21-b8af-45ab9c504b93 2024-02-05T04:51:44.927 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:51:44.927 INFO:teuthology.orchestra.run.smithi091.stdout:[osd] 2024-02-05T04:51:44.927 INFO:teuthology.orchestra.run.smithi091.stdout: osd journal size = 100 2024-02-05T04:51:44.927 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:51:44.928 INFO:teuthology.orchestra.run.smithi091.stdout: osd scrub load threshold = 5.0 2024-02-05T04:51:44.928 INFO:teuthology.orchestra.run.smithi091.stdout: osd scrub max interval = 120 2024-02-05T04:51:44.928 INFO:teuthology.orchestra.run.smithi091.stdout: osd mclock profile = high_recovery_ops 2024-02-05T04:51:44.928 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:51:44.928 INFO:teuthology.orchestra.run.smithi091.stdout: osd recover clone overlap = true 2024-02-05T04:51:44.928 INFO:teuthology.orchestra.run.smithi091.stdout: osd recovery max chunk = 1048576 2024-02-05T04:51:44.928 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:51:44.928 INFO:teuthology.orchestra.run.smithi091.stdout: osd debug shutdown = true 2024-02-05T04:51:44.928 INFO:teuthology.orchestra.run.smithi091.stdout: osd debug op order = true 2024-02-05T04:51:44.928 INFO:teuthology.orchestra.run.smithi091.stdout: osd debug verify stray on activate = true 2024-02-05T04:51:44.928 INFO:teuthology.orchestra.run.smithi091.stdout: osd debug trim objects = true 2024-02-05T04:51:44.928 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:51:44.928 INFO:teuthology.orchestra.run.smithi091.stdout: osd open classes on start = true 2024-02-05T04:51:44.928 INFO:teuthology.orchestra.run.smithi091.stdout: osd debug pg log writeout = true 2024-02-05T04:51:44.928 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:51:44.929 INFO:teuthology.orchestra.run.smithi091.stdout: osd deep scrub update digest min age = 30 2024-02-05T04:51:44.929 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:51:44.929 INFO:teuthology.orchestra.run.smithi091.stdout: osd map max advance = 10 2024-02-05T04:51:44.929 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:51:44.929 INFO:teuthology.orchestra.run.smithi091.stdout: journal zero on create = true 2024-02-05T04:51:44.929 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:51:44.929 INFO:teuthology.orchestra.run.smithi091.stdout: filestore ondisk finisher threads = 3 2024-02-05T04:51:44.929 INFO:teuthology.orchestra.run.smithi091.stdout: filestore apply finisher threads = 3 2024-02-05T04:51:44.929 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:51:44.929 INFO:teuthology.orchestra.run.smithi091.stdout: bdev debug aio = true 2024-02-05T04:51:44.929 INFO:teuthology.orchestra.run.smithi091.stdout: osd debug misdirected ops = true 2024-02-05T04:51:44.929 INFO:teuthology.orchestra.run.smithi091.stdout: debug monc = 20 2024-02-05T04:51:44.930 INFO:teuthology.orchestra.run.smithi091.stdout: bdev async discard = True 2024-02-05T04:51:44.930 INFO:teuthology.orchestra.run.smithi091.stdout: bdev enable discard = True 2024-02-05T04:51:44.930 INFO:teuthology.orchestra.run.smithi091.stdout: bluestore block size = 96636764160 2024-02-05T04:51:44.930 INFO:teuthology.orchestra.run.smithi091.stdout: bluestore fsck on mount = True 2024-02-05T04:51:44.930 INFO:teuthology.orchestra.run.smithi091.stdout: bluestore zero block detection = True 2024-02-05T04:51:44.930 INFO:teuthology.orchestra.run.smithi091.stdout: debug bluefs = 20 2024-02-05T04:51:44.930 INFO:teuthology.orchestra.run.smithi091.stdout: debug bluestore = 20 2024-02-05T04:51:44.930 INFO:teuthology.orchestra.run.smithi091.stdout: debug ms = 1 2024-02-05T04:51:44.930 INFO:teuthology.orchestra.run.smithi091.stdout: debug osd = 20 2024-02-05T04:51:44.930 INFO:teuthology.orchestra.run.smithi091.stdout: debug rocksdb = 10 2024-02-05T04:51:44.930 INFO:teuthology.orchestra.run.smithi091.stdout: mon osd backfillfull_ratio = 0.85 2024-02-05T04:51:44.930 INFO:teuthology.orchestra.run.smithi091.stdout: mon osd full ratio = 0.9 2024-02-05T04:51:44.930 INFO:teuthology.orchestra.run.smithi091.stdout: mon osd nearfull ratio = 0.8 2024-02-05T04:51:44.930 INFO:teuthology.orchestra.run.smithi091.stdout: osd blocked scrub grace period = 3600 2024-02-05T04:51:44.930 INFO:teuthology.orchestra.run.smithi091.stdout: osd debug reject backfill probability = 0.1 2024-02-05T04:51:44.930 INFO:teuthology.orchestra.run.smithi091.stdout: osd debug verify cached snaps = True 2024-02-05T04:51:44.931 INFO:teuthology.orchestra.run.smithi091.stdout: osd debug verify missing on start = True 2024-02-05T04:51:44.931 INFO:teuthology.orchestra.run.smithi091.stdout: osd failsafe full ratio = 0.95 2024-02-05T04:51:44.931 INFO:teuthology.orchestra.run.smithi091.stdout: osd max backfills = 3 2024-02-05T04:51:44.931 INFO:teuthology.orchestra.run.smithi091.stdout: osd max markdown count = 1000 2024-02-05T04:51:44.931 INFO:teuthology.orchestra.run.smithi091.stdout: osd mclock override recovery settings = True 2024-02-05T04:51:44.931 INFO:teuthology.orchestra.run.smithi091.stdout: osd memory target = 1610612736 2024-02-05T04:51:44.931 INFO:teuthology.orchestra.run.smithi091.stdout: osd objectstore = bluestore 2024-02-05T04:51:44.931 INFO:teuthology.orchestra.run.smithi091.stdout: osd op queue = debug_random 2024-02-05T04:51:44.931 INFO:teuthology.orchestra.run.smithi091.stdout: osd op queue cut off = debug_random 2024-02-05T04:51:44.931 INFO:teuthology.orchestra.run.smithi091.stdout: osd scrub min interval = 60 2024-02-05T04:51:44.931 INFO:teuthology.orchestra.run.smithi091.stdout: osd shutdown pgref assert = True 2024-02-05T04:51:44.931 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:51:44.931 INFO:teuthology.orchestra.run.smithi091.stdout:[mgr] 2024-02-05T04:51:44.931 INFO:teuthology.orchestra.run.smithi091.stdout: debug ms = 1 2024-02-05T04:51:44.931 INFO:teuthology.orchestra.run.smithi091.stdout: debug mgr = 20 2024-02-05T04:51:44.932 INFO:teuthology.orchestra.run.smithi091.stdout: debug mon = 20 2024-02-05T04:51:44.932 INFO:teuthology.orchestra.run.smithi091.stdout: debug auth = 20 2024-02-05T04:51:44.932 INFO:teuthology.orchestra.run.smithi091.stdout: mon reweight min pgs per osd = 4 2024-02-05T04:51:44.932 INFO:teuthology.orchestra.run.smithi091.stdout: mon reweight min bytes per osd = 10 2024-02-05T04:51:44.932 INFO:teuthology.orchestra.run.smithi091.stdout: mgr/telemetry/nag = false 2024-02-05T04:51:44.932 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:51:44.932 INFO:teuthology.orchestra.run.smithi091.stdout:[mon] 2024-02-05T04:51:44.932 INFO:teuthology.orchestra.run.smithi091.stdout: debug ms = 1 2024-02-05T04:51:44.932 INFO:teuthology.orchestra.run.smithi091.stdout: debug mon = 20 2024-02-05T04:51:44.932 INFO:teuthology.orchestra.run.smithi091.stdout: debug paxos = 20 2024-02-05T04:51:44.932 INFO:teuthology.orchestra.run.smithi091.stdout: debug auth = 20 2024-02-05T04:51:44.932 INFO:teuthology.orchestra.run.smithi091.stdout: mon data avail warn = 5 2024-02-05T04:51:44.932 INFO:teuthology.orchestra.run.smithi091.stdout: mon mgr mkfs grace = 240 2024-02-05T04:51:44.932 INFO:teuthology.orchestra.run.smithi091.stdout: mon reweight min pgs per osd = 4 2024-02-05T04:51:44.933 INFO:teuthology.orchestra.run.smithi091.stdout: mon osd reporter subtree level = osd 2024-02-05T04:51:44.933 INFO:teuthology.orchestra.run.smithi091.stdout: mon osd prime pg temp = true 2024-02-05T04:51:44.933 INFO:teuthology.orchestra.run.smithi091.stdout: mon reweight min bytes per osd = 10 2024-02-05T04:51:44.933 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:51:44.933 INFO:teuthology.orchestra.run.smithi091.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-02-05T04:51:44.933 INFO:teuthology.orchestra.run.smithi091.stdout: auth mon ticket ttl = 660 # 11m 2024-02-05T04:51:44.933 INFO:teuthology.orchestra.run.smithi091.stdout: auth service ticket ttl = 240 # 4m 2024-02-05T04:51:44.933 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:51:44.933 INFO:teuthology.orchestra.run.smithi091.stdout: # don't complain about insecure global_id in the test suite 2024-02-05T04:51:44.933 INFO:teuthology.orchestra.run.smithi091.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-02-05T04:51:44.933 INFO:teuthology.orchestra.run.smithi091.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-02-05T04:51:44.933 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:51:44.933 INFO:teuthology.orchestra.run.smithi091.stdout: # 1m isn't quite enough 2024-02-05T04:51:44.933 INFO:teuthology.orchestra.run.smithi091.stdout: mon_down_mkfs_grace = 2m 2024-02-05T04:51:44.933 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:51:44.933 INFO:teuthology.orchestra.run.smithi091.stdout: mon_warn_on_filestore_osds = false 2024-02-05T04:51:44.934 INFO:teuthology.orchestra.run.smithi091.stdout: mon scrub interval = 300 2024-02-05T04:51:44.934 INFO:teuthology.orchestra.run.smithi091.stdout: 2024-02-05T04:51:44.934 INFO:teuthology.orchestra.run.smithi091.stdout:[client] 2024-02-05T04:51:44.934 INFO:teuthology.orchestra.run.smithi091.stdout: rgw cache enabled = true 2024-02-05T04:51:44.934 INFO:teuthology.orchestra.run.smithi091.stdout: rgw enable ops log = true 2024-02-05T04:51:44.934 INFO:teuthology.orchestra.run.smithi091.stdout: rgw enable usage log = true 2024-02-05T04:51:44.934 INFO:teuthology.orchestra.run.smithi091.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-02-05T04:51:44.934 INFO:teuthology.orchestra.run.smithi091.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-02-05T04:51:44.935 INFO:teuthology.orchestra.run.smithi091.stdout:[mon.a] 2024-02-05T04:51:44.935 INFO:teuthology.orchestra.run.smithi091.stdout:[mon.b] 2024-02-05T04:51:44.935 INFO:teuthology.orchestra.run.smithi091.stdout:[mon.c] 2024-02-05T04:51:44.935 INFO:teuthology.orchestra.run.smithi196.stdout:[global] 2024-02-05T04:51:44.935 INFO:teuthology.orchestra.run.smithi196.stdout: chdir = "" 2024-02-05T04:51:44.935 INFO:teuthology.orchestra.run.smithi196.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-02-05T04:51:44.935 INFO:teuthology.orchestra.run.smithi196.stdout: auth supported = cephx 2024-02-05T04:51:44.935 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:51:44.935 INFO:teuthology.orchestra.run.smithi196.stdout: filestore xattr use omap = true 2024-02-05T04:51:44.935 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:51:44.935 INFO:teuthology.orchestra.run.smithi196.stdout: mon clock drift allowed = 1.000 2024-02-05T04:51:44.936 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:51:44.936 INFO:teuthology.orchestra.run.smithi196.stdout: osd crush chooseleaf type = 0 2024-02-05T04:51:44.936 INFO:teuthology.orchestra.run.smithi196.stdout: auth debug = true 2024-02-05T04:51:44.936 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:51:44.936 INFO:teuthology.orchestra.run.smithi196.stdout: ms die on old message = true 2024-02-05T04:51:44.936 INFO:teuthology.orchestra.run.smithi196.stdout: ms die on bug = true 2024-02-05T04:51:44.936 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:51:44.936 INFO:teuthology.orchestra.run.smithi196.stdout: mon max pg per osd = 10000 # >= luminous 2024-02-05T04:51:44.936 INFO:teuthology.orchestra.run.smithi196.stdout: mon pg warn max object skew = 0 2024-02-05T04:51:44.936 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:51:44.936 INFO:teuthology.orchestra.run.smithi196.stdout: # disable pg_autoscaler by default for new pools 2024-02-05T04:51:44.937 INFO:teuthology.orchestra.run.smithi196.stdout: osd_pool_default_pg_autoscale_mode = off 2024-02-05T04:51:44.937 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:51:44.937 INFO:teuthology.orchestra.run.smithi196.stdout: osd pool default size = 2 2024-02-05T04:51:44.937 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:51:44.937 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd allow primary affinity = true 2024-02-05T04:51:44.937 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd allow pg remap = true 2024-02-05T04:51:44.937 INFO:teuthology.orchestra.run.smithi196.stdout: mon warn on legacy crush tunables = false 2024-02-05T04:51:44.937 INFO:teuthology.orchestra.run.smithi196.stdout: mon warn on crush straw calc version zero = false 2024-02-05T04:51:44.937 INFO:teuthology.orchestra.run.smithi196.stdout: mon warn on no sortbitwise = false 2024-02-05T04:51:44.937 INFO:teuthology.orchestra.run.smithi196.stdout: mon warn on osd down out interval zero = false 2024-02-05T04:51:44.937 INFO:teuthology.orchestra.run.smithi196.stdout: mon warn on too few osds = false 2024-02-05T04:51:44.937 INFO:teuthology.orchestra.run.smithi196.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-02-05T04:51:44.937 INFO:teuthology.orchestra.run.smithi196.stdout: mon_warn_on_pool_no_redundancy = false 2024-02-05T04:51:44.937 INFO:teuthology.orchestra.run.smithi196.stdout: mon_allow_pool_size_one = true 2024-02-05T04:51:44.937 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:51:44.937 INFO:teuthology.orchestra.run.smithi196.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-02-05T04:51:44.938 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:51:44.938 INFO:teuthology.orchestra.run.smithi196.stdout: osd default data pool replay window = 5 2024-02-05T04:51:44.938 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:51:44.938 INFO:teuthology.orchestra.run.smithi196.stdout: mon allow pool delete = true 2024-02-05T04:51:44.938 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:51:44.938 INFO:teuthology.orchestra.run.smithi196.stdout: mon cluster log file level = debug 2024-02-05T04:51:44.938 INFO:teuthology.orchestra.run.smithi196.stdout: debug asserts on shutdown = true 2024-02-05T04:51:44.938 INFO:teuthology.orchestra.run.smithi196.stdout: mon health detail to clog = false 2024-02-05T04:51:44.938 INFO:teuthology.orchestra.run.smithi196.stdout: mon host = "172.21.15.62,172.21.15.91,172.21.15.150" 2024-02-05T04:51:44.938 INFO:teuthology.orchestra.run.smithi196.stdout: mon client directed command retry = 5 2024-02-05T04:51:44.938 INFO:teuthology.orchestra.run.smithi196.stdout: mon election default strategy = 3 2024-02-05T04:51:44.938 INFO:teuthology.orchestra.run.smithi196.stdout: ms inject socket failures = 2500 2024-02-05T04:51:44.938 INFO:teuthology.orchestra.run.smithi196.stdout: ms tcp read timeout = 5 2024-02-05T04:51:44.938 INFO:teuthology.orchestra.run.smithi196.stdout: osd_async_recovery_min_cost = 1 2024-02-05T04:51:44.938 INFO:teuthology.orchestra.run.smithi196.stdout: osd_object_clean_region_max_num_intervals = 1000 2024-02-05T04:51:44.939 INFO:teuthology.orchestra.run.smithi196.stdout: fsid = 88c11a2d-382e-4f21-b8af-45ab9c504b93 2024-02-05T04:51:44.939 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:51:44.939 INFO:teuthology.orchestra.run.smithi196.stdout:[osd] 2024-02-05T04:51:44.939 INFO:teuthology.orchestra.run.smithi196.stdout: osd journal size = 100 2024-02-05T04:51:44.939 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:51:44.939 INFO:teuthology.orchestra.run.smithi196.stdout: osd scrub load threshold = 5.0 2024-02-05T04:51:44.939 INFO:teuthology.orchestra.run.smithi196.stdout: osd scrub max interval = 120 2024-02-05T04:51:44.939 INFO:teuthology.orchestra.run.smithi196.stdout: osd mclock profile = high_recovery_ops 2024-02-05T04:51:44.939 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:51:44.939 INFO:teuthology.orchestra.run.smithi196.stdout: osd recover clone overlap = true 2024-02-05T04:51:44.939 INFO:teuthology.orchestra.run.smithi196.stdout: osd recovery max chunk = 1048576 2024-02-05T04:51:44.939 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:51:44.939 INFO:teuthology.orchestra.run.smithi196.stdout: osd debug shutdown = true 2024-02-05T04:51:44.939 INFO:teuthology.orchestra.run.smithi196.stdout: osd debug op order = true 2024-02-05T04:51:44.940 INFO:teuthology.orchestra.run.smithi196.stdout: osd debug verify stray on activate = true 2024-02-05T04:51:44.940 INFO:teuthology.orchestra.run.smithi196.stdout: osd debug trim objects = true 2024-02-05T04:51:44.940 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:51:44.940 INFO:teuthology.orchestra.run.smithi196.stdout: osd open classes on start = true 2024-02-05T04:51:44.940 INFO:teuthology.orchestra.run.smithi196.stdout: osd debug pg log writeout = true 2024-02-05T04:51:44.940 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:51:44.940 INFO:teuthology.orchestra.run.smithi196.stdout: osd deep scrub update digest min age = 30 2024-02-05T04:51:44.940 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:51:44.940 INFO:teuthology.orchestra.run.smithi196.stdout: osd map max advance = 10 2024-02-05T04:51:44.940 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:51:44.940 INFO:teuthology.orchestra.run.smithi196.stdout: journal zero on create = true 2024-02-05T04:51:44.940 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:51:44.940 INFO:teuthology.orchestra.run.smithi196.stdout: filestore ondisk finisher threads = 3 2024-02-05T04:51:44.940 INFO:teuthology.orchestra.run.smithi196.stdout: filestore apply finisher threads = 3 2024-02-05T04:51:44.941 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:51:44.941 INFO:teuthology.orchestra.run.smithi196.stdout: bdev debug aio = true 2024-02-05T04:51:44.941 INFO:teuthology.orchestra.run.smithi196.stdout: osd debug misdirected ops = true 2024-02-05T04:51:44.941 INFO:teuthology.orchestra.run.smithi196.stdout: debug monc = 20 2024-02-05T04:51:44.941 INFO:teuthology.orchestra.run.smithi196.stdout: bdev async discard = True 2024-02-05T04:51:44.941 INFO:teuthology.orchestra.run.smithi196.stdout: bdev enable discard = True 2024-02-05T04:51:44.941 INFO:teuthology.orchestra.run.smithi196.stdout: bluestore block size = 96636764160 2024-02-05T04:51:44.941 INFO:teuthology.orchestra.run.smithi196.stdout: bluestore fsck on mount = True 2024-02-05T04:51:44.941 INFO:teuthology.orchestra.run.smithi196.stdout: bluestore zero block detection = True 2024-02-05T04:51:44.941 INFO:teuthology.orchestra.run.smithi196.stdout: debug bluefs = 20 2024-02-05T04:51:44.941 INFO:teuthology.orchestra.run.smithi196.stdout: debug bluestore = 20 2024-02-05T04:51:44.941 INFO:teuthology.orchestra.run.smithi196.stdout: debug ms = 1 2024-02-05T04:51:44.941 INFO:teuthology.orchestra.run.smithi196.stdout: debug osd = 20 2024-02-05T04:51:44.941 INFO:teuthology.orchestra.run.smithi196.stdout: debug rocksdb = 10 2024-02-05T04:51:44.941 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd backfillfull_ratio = 0.85 2024-02-05T04:51:44.942 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd full ratio = 0.9 2024-02-05T04:51:44.942 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd nearfull ratio = 0.8 2024-02-05T04:51:44.942 INFO:teuthology.orchestra.run.smithi196.stdout: osd blocked scrub grace period = 3600 2024-02-05T04:51:44.942 INFO:teuthology.orchestra.run.smithi196.stdout: osd debug reject backfill probability = 0.1 2024-02-05T04:51:44.942 INFO:teuthology.orchestra.run.smithi196.stdout: osd debug verify cached snaps = True 2024-02-05T04:51:44.942 INFO:teuthology.orchestra.run.smithi196.stdout: osd debug verify missing on start = True 2024-02-05T04:51:44.942 INFO:teuthology.orchestra.run.smithi196.stdout: osd failsafe full ratio = 0.95 2024-02-05T04:51:44.942 INFO:teuthology.orchestra.run.smithi196.stdout: osd max backfills = 3 2024-02-05T04:51:44.942 INFO:teuthology.orchestra.run.smithi196.stdout: osd max markdown count = 1000 2024-02-05T04:51:44.942 INFO:teuthology.orchestra.run.smithi196.stdout: osd mclock override recovery settings = True 2024-02-05T04:51:44.942 INFO:teuthology.orchestra.run.smithi196.stdout: osd memory target = 1610612736 2024-02-05T04:51:44.942 INFO:teuthology.orchestra.run.smithi196.stdout: osd objectstore = bluestore 2024-02-05T04:51:44.942 INFO:teuthology.orchestra.run.smithi196.stdout: osd op queue = debug_random 2024-02-05T04:51:44.942 INFO:teuthology.orchestra.run.smithi196.stdout: osd op queue cut off = debug_random 2024-02-05T04:51:44.942 INFO:teuthology.orchestra.run.smithi196.stdout: osd scrub min interval = 60 2024-02-05T04:51:44.943 INFO:teuthology.orchestra.run.smithi196.stdout: osd shutdown pgref assert = True 2024-02-05T04:51:44.943 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:51:44.943 INFO:teuthology.orchestra.run.smithi196.stdout:[mgr] 2024-02-05T04:51:44.943 INFO:teuthology.orchestra.run.smithi196.stdout: debug ms = 1 2024-02-05T04:51:44.943 INFO:teuthology.orchestra.run.smithi196.stdout: debug mgr = 20 2024-02-05T04:51:44.943 INFO:teuthology.orchestra.run.smithi196.stdout: debug mon = 20 2024-02-05T04:51:44.943 INFO:teuthology.orchestra.run.smithi196.stdout: debug auth = 20 2024-02-05T04:51:44.943 INFO:teuthology.orchestra.run.smithi196.stdout: mon reweight min pgs per osd = 4 2024-02-05T04:51:44.943 INFO:teuthology.orchestra.run.smithi196.stdout: mon reweight min bytes per osd = 10 2024-02-05T04:51:44.943 INFO:teuthology.orchestra.run.smithi196.stdout: mgr/telemetry/nag = false 2024-02-05T04:51:44.943 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:51:44.943 INFO:teuthology.orchestra.run.smithi196.stdout:[mon] 2024-02-05T04:51:44.943 INFO:teuthology.orchestra.run.smithi196.stdout: debug ms = 1 2024-02-05T04:51:44.943 INFO:teuthology.orchestra.run.smithi196.stdout: debug mon = 20 2024-02-05T04:51:44.943 INFO:teuthology.orchestra.run.smithi196.stdout: debug paxos = 20 2024-02-05T04:51:44.943 INFO:teuthology.orchestra.run.smithi196.stdout: debug auth = 20 2024-02-05T04:51:44.944 INFO:teuthology.orchestra.run.smithi196.stdout: mon data avail warn = 5 2024-02-05T04:51:44.944 INFO:teuthology.orchestra.run.smithi196.stdout: mon mgr mkfs grace = 240 2024-02-05T04:51:44.944 INFO:teuthology.orchestra.run.smithi196.stdout: mon reweight min pgs per osd = 4 2024-02-05T04:51:44.944 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd reporter subtree level = osd 2024-02-05T04:51:44.944 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd prime pg temp = true 2024-02-05T04:51:44.944 INFO:teuthology.orchestra.run.smithi196.stdout: mon reweight min bytes per osd = 10 2024-02-05T04:51:44.944 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:51:44.944 INFO:teuthology.orchestra.run.smithi196.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-02-05T04:51:44.944 INFO:teuthology.orchestra.run.smithi196.stdout: auth mon ticket ttl = 660 # 11m 2024-02-05T04:51:44.944 INFO:teuthology.orchestra.run.smithi196.stdout: auth service ticket ttl = 240 # 4m 2024-02-05T04:51:44.944 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:51:44.944 INFO:teuthology.orchestra.run.smithi196.stdout: # don't complain about insecure global_id in the test suite 2024-02-05T04:51:44.944 INFO:teuthology.orchestra.run.smithi196.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-02-05T04:51:44.944 INFO:teuthology.orchestra.run.smithi196.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-02-05T04:51:44.945 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:51:44.945 INFO:teuthology.orchestra.run.smithi196.stdout: # 1m isn't quite enough 2024-02-05T04:51:44.945 INFO:teuthology.orchestra.run.smithi196.stdout: mon_down_mkfs_grace = 2m 2024-02-05T04:51:44.945 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:51:44.945 INFO:teuthology.orchestra.run.smithi196.stdout: mon_warn_on_filestore_osds = false 2024-02-05T04:51:44.945 INFO:teuthology.orchestra.run.smithi196.stdout: mon scrub interval = 300 2024-02-05T04:51:44.945 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T04:51:44.945 INFO:teuthology.orchestra.run.smithi196.stdout:[client] 2024-02-05T04:51:44.945 INFO:teuthology.orchestra.run.smithi196.stdout: rgw cache enabled = true 2024-02-05T04:51:44.945 INFO:teuthology.orchestra.run.smithi196.stdout: rgw enable ops log = true 2024-02-05T04:51:44.945 INFO:teuthology.orchestra.run.smithi196.stdout: rgw enable usage log = true 2024-02-05T04:51:44.945 INFO:teuthology.orchestra.run.smithi196.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-02-05T04:51:44.945 INFO:teuthology.orchestra.run.smithi196.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-02-05T04:51:44.945 INFO:teuthology.orchestra.run.smithi196.stdout:[mon.a] 2024-02-05T04:51:44.945 INFO:teuthology.orchestra.run.smithi196.stdout:[mon.b] 2024-02-05T04:51:44.946 INFO:teuthology.orchestra.run.smithi196.stdout:[mon.c] 2024-02-05T04:51:44.948 INFO:teuthology.orchestra.run.smithi150.stdout:[global] 2024-02-05T04:51:44.948 INFO:teuthology.orchestra.run.smithi150.stdout: chdir = "" 2024-02-05T04:51:44.949 INFO:teuthology.orchestra.run.smithi150.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-02-05T04:51:44.949 INFO:teuthology.orchestra.run.smithi150.stdout: auth supported = cephx 2024-02-05T04:51:44.949 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:51:44.949 INFO:teuthology.orchestra.run.smithi150.stdout: filestore xattr use omap = true 2024-02-05T04:51:44.949 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:51:44.949 INFO:teuthology.orchestra.run.smithi150.stdout: mon clock drift allowed = 1.000 2024-02-05T04:51:44.949 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:51:44.949 INFO:teuthology.orchestra.run.smithi150.stdout: osd crush chooseleaf type = 0 2024-02-05T04:51:44.949 INFO:teuthology.orchestra.run.smithi150.stdout: auth debug = true 2024-02-05T04:51:44.949 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:51:44.949 INFO:teuthology.orchestra.run.smithi150.stdout: ms die on old message = true 2024-02-05T04:51:44.949 INFO:teuthology.orchestra.run.smithi150.stdout: ms die on bug = true 2024-02-05T04:51:44.950 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:51:44.950 INFO:teuthology.orchestra.run.smithi150.stdout: mon max pg per osd = 10000 # >= luminous 2024-02-05T04:51:44.950 INFO:teuthology.orchestra.run.smithi150.stdout: mon pg warn max object skew = 0 2024-02-05T04:51:44.950 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:51:44.950 INFO:teuthology.orchestra.run.smithi150.stdout: # disable pg_autoscaler by default for new pools 2024-02-05T04:51:44.950 INFO:teuthology.orchestra.run.smithi150.stdout: osd_pool_default_pg_autoscale_mode = off 2024-02-05T04:51:44.950 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:51:44.950 INFO:teuthology.orchestra.run.smithi150.stdout: osd pool default size = 2 2024-02-05T04:51:44.950 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:51:44.950 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd allow primary affinity = true 2024-02-05T04:51:44.950 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd allow pg remap = true 2024-02-05T04:51:44.950 INFO:teuthology.orchestra.run.smithi150.stdout: mon warn on legacy crush tunables = false 2024-02-05T04:51:44.951 INFO:teuthology.orchestra.run.smithi150.stdout: mon warn on crush straw calc version zero = false 2024-02-05T04:51:44.951 INFO:teuthology.orchestra.run.smithi150.stdout: mon warn on no sortbitwise = false 2024-02-05T04:51:44.951 INFO:teuthology.orchestra.run.smithi150.stdout: mon warn on osd down out interval zero = false 2024-02-05T04:51:44.952 INFO:teuthology.orchestra.run.smithi150.stdout: mon warn on too few osds = false 2024-02-05T04:51:44.952 INFO:teuthology.orchestra.run.smithi150.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-02-05T04:51:44.952 INFO:teuthology.orchestra.run.smithi150.stdout: mon_warn_on_pool_no_redundancy = false 2024-02-05T04:51:44.952 INFO:teuthology.orchestra.run.smithi150.stdout: mon_allow_pool_size_one = true 2024-02-05T04:51:44.952 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:51:44.952 INFO:teuthology.orchestra.run.smithi150.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-02-05T04:51:44.952 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:51:44.952 INFO:teuthology.orchestra.run.smithi150.stdout: osd default data pool replay window = 5 2024-02-05T04:51:44.952 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:51:44.952 INFO:teuthology.orchestra.run.smithi150.stdout: mon allow pool delete = true 2024-02-05T04:51:44.952 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:51:44.952 INFO:teuthology.orchestra.run.smithi150.stdout: mon cluster log file level = debug 2024-02-05T04:51:44.952 INFO:teuthology.orchestra.run.smithi150.stdout: debug asserts on shutdown = true 2024-02-05T04:51:44.953 INFO:teuthology.orchestra.run.smithi150.stdout: mon health detail to clog = false 2024-02-05T04:51:44.953 INFO:teuthology.orchestra.run.smithi150.stdout: mon host = "172.21.15.62,172.21.15.91,172.21.15.150" 2024-02-05T04:51:44.953 INFO:teuthology.orchestra.run.smithi150.stdout: mon client directed command retry = 5 2024-02-05T04:51:44.953 INFO:teuthology.orchestra.run.smithi150.stdout: mon election default strategy = 3 2024-02-05T04:51:44.953 INFO:teuthology.orchestra.run.smithi150.stdout: ms inject socket failures = 2500 2024-02-05T04:51:44.953 INFO:teuthology.orchestra.run.smithi150.stdout: ms tcp read timeout = 5 2024-02-05T04:51:44.953 INFO:teuthology.orchestra.run.smithi150.stdout: osd_async_recovery_min_cost = 1 2024-02-05T04:51:44.953 INFO:teuthology.orchestra.run.smithi150.stdout: osd_object_clean_region_max_num_intervals = 1000 2024-02-05T04:51:44.953 INFO:teuthology.orchestra.run.smithi150.stdout: fsid = 88c11a2d-382e-4f21-b8af-45ab9c504b93 2024-02-05T04:51:44.953 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:51:44.953 INFO:teuthology.orchestra.run.smithi150.stdout:[osd] 2024-02-05T04:51:44.954 INFO:teuthology.orchestra.run.smithi150.stdout: osd journal size = 100 2024-02-05T04:51:44.954 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:51:44.954 INFO:teuthology.orchestra.run.smithi150.stdout: osd scrub load threshold = 5.0 2024-02-05T04:51:44.954 INFO:teuthology.orchestra.run.smithi150.stdout: osd scrub max interval = 120 2024-02-05T04:51:44.954 INFO:teuthology.orchestra.run.smithi150.stdout: osd mclock profile = high_recovery_ops 2024-02-05T04:51:44.954 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:51:44.954 INFO:teuthology.orchestra.run.smithi150.stdout: osd recover clone overlap = true 2024-02-05T04:51:44.954 INFO:teuthology.orchestra.run.smithi150.stdout: osd recovery max chunk = 1048576 2024-02-05T04:51:44.954 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:51:44.954 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug shutdown = true 2024-02-05T04:51:44.955 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug op order = true 2024-02-05T04:51:44.955 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug verify stray on activate = true 2024-02-05T04:51:44.955 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug trim objects = true 2024-02-05T04:51:44.955 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:51:44.956 INFO:teuthology.orchestra.run.smithi150.stdout: osd open classes on start = true 2024-02-05T04:51:44.956 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug pg log writeout = true 2024-02-05T04:51:44.956 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:51:44.956 INFO:teuthology.orchestra.run.smithi150.stdout: osd deep scrub update digest min age = 30 2024-02-05T04:51:44.956 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:51:44.956 INFO:teuthology.orchestra.run.smithi150.stdout: osd map max advance = 10 2024-02-05T04:51:44.956 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:51:44.956 INFO:teuthology.orchestra.run.smithi150.stdout: journal zero on create = true 2024-02-05T04:51:44.957 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:51:44.957 INFO:teuthology.orchestra.run.smithi150.stdout: filestore ondisk finisher threads = 3 2024-02-05T04:51:44.957 INFO:teuthology.orchestra.run.smithi150.stdout: filestore apply finisher threads = 3 2024-02-05T04:51:44.957 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:51:44.957 INFO:teuthology.orchestra.run.smithi150.stdout: bdev debug aio = true 2024-02-05T04:51:44.957 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug misdirected ops = true 2024-02-05T04:51:44.957 INFO:teuthology.orchestra.run.smithi150.stdout: debug monc = 20 2024-02-05T04:51:44.957 INFO:teuthology.orchestra.run.smithi150.stdout: bdev async discard = True 2024-02-05T04:51:44.957 INFO:teuthology.orchestra.run.smithi150.stdout: bdev enable discard = True 2024-02-05T04:51:44.957 INFO:teuthology.orchestra.run.smithi150.stdout: bluestore block size = 96636764160 2024-02-05T04:51:44.957 INFO:teuthology.orchestra.run.smithi150.stdout: bluestore fsck on mount = True 2024-02-05T04:51:44.957 INFO:teuthology.orchestra.run.smithi150.stdout: bluestore zero block detection = True 2024-02-05T04:51:44.957 INFO:teuthology.orchestra.run.smithi150.stdout: debug bluefs = 20 2024-02-05T04:51:44.957 INFO:teuthology.orchestra.run.smithi150.stdout: debug bluestore = 20 2024-02-05T04:51:44.957 INFO:teuthology.orchestra.run.smithi150.stdout: debug ms = 1 2024-02-05T04:51:44.957 INFO:teuthology.orchestra.run.smithi150.stdout: debug osd = 20 2024-02-05T04:51:44.958 INFO:teuthology.orchestra.run.smithi150.stdout: debug rocksdb = 10 2024-02-05T04:51:44.958 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd backfillfull_ratio = 0.85 2024-02-05T04:51:44.958 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd full ratio = 0.9 2024-02-05T04:51:44.958 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd nearfull ratio = 0.8 2024-02-05T04:51:44.958 INFO:teuthology.orchestra.run.smithi150.stdout: osd blocked scrub grace period = 3600 2024-02-05T04:51:44.958 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug reject backfill probability = 0.1 2024-02-05T04:51:44.958 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug verify cached snaps = True 2024-02-05T04:51:44.958 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug verify missing on start = True 2024-02-05T04:51:44.958 INFO:teuthology.orchestra.run.smithi150.stdout: osd failsafe full ratio = 0.95 2024-02-05T04:51:44.958 INFO:teuthology.orchestra.run.smithi150.stdout: osd max backfills = 3 2024-02-05T04:51:44.958 INFO:teuthology.orchestra.run.smithi150.stdout: osd max markdown count = 1000 2024-02-05T04:51:44.958 INFO:teuthology.orchestra.run.smithi150.stdout: osd mclock override recovery settings = True 2024-02-05T04:51:44.958 INFO:teuthology.orchestra.run.smithi150.stdout: osd memory target = 1610612736 2024-02-05T04:51:44.958 INFO:teuthology.orchestra.run.smithi150.stdout: osd objectstore = bluestore 2024-02-05T04:51:44.958 INFO:teuthology.orchestra.run.smithi150.stdout: osd op queue = debug_random 2024-02-05T04:51:44.958 INFO:teuthology.orchestra.run.smithi150.stdout: osd op queue cut off = debug_random 2024-02-05T04:51:44.959 INFO:teuthology.orchestra.run.smithi150.stdout: osd scrub min interval = 60 2024-02-05T04:51:44.959 INFO:teuthology.orchestra.run.smithi150.stdout: osd shutdown pgref assert = True 2024-02-05T04:51:44.959 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:51:44.959 INFO:teuthology.orchestra.run.smithi150.stdout:[mgr] 2024-02-05T04:51:44.959 INFO:teuthology.orchestra.run.smithi150.stdout: debug ms = 1 2024-02-05T04:51:44.959 INFO:teuthology.orchestra.run.smithi150.stdout: debug mgr = 20 2024-02-05T04:51:44.959 INFO:teuthology.orchestra.run.smithi150.stdout: debug mon = 20 2024-02-05T04:51:44.959 INFO:teuthology.orchestra.run.smithi150.stdout: debug auth = 20 2024-02-05T04:51:44.959 INFO:teuthology.orchestra.run.smithi150.stdout: mon reweight min pgs per osd = 4 2024-02-05T04:51:44.959 INFO:teuthology.orchestra.run.smithi150.stdout: mon reweight min bytes per osd = 10 2024-02-05T04:51:44.959 INFO:teuthology.orchestra.run.smithi150.stdout: mgr/telemetry/nag = false 2024-02-05T04:51:44.959 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:51:44.959 INFO:teuthology.orchestra.run.smithi150.stdout:[mon] 2024-02-05T04:51:44.959 INFO:teuthology.orchestra.run.smithi150.stdout: debug ms = 1 2024-02-05T04:51:44.959 INFO:teuthology.orchestra.run.smithi150.stdout: debug mon = 20 2024-02-05T04:51:44.960 INFO:teuthology.orchestra.run.smithi150.stdout: debug paxos = 20 2024-02-05T04:51:44.960 INFO:teuthology.orchestra.run.smithi150.stdout: debug auth = 20 2024-02-05T04:51:44.960 INFO:teuthology.orchestra.run.smithi150.stdout: mon data avail warn = 5 2024-02-05T04:51:44.960 INFO:teuthology.orchestra.run.smithi150.stdout: mon mgr mkfs grace = 240 2024-02-05T04:51:44.960 INFO:teuthology.orchestra.run.smithi150.stdout: mon reweight min pgs per osd = 4 2024-02-05T04:51:44.960 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd reporter subtree level = osd 2024-02-05T04:51:44.960 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd prime pg temp = true 2024-02-05T04:51:44.960 INFO:teuthology.orchestra.run.smithi150.stdout: mon reweight min bytes per osd = 10 2024-02-05T04:51:44.960 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:51:44.960 INFO:teuthology.orchestra.run.smithi150.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-02-05T04:51:44.960 INFO:teuthology.orchestra.run.smithi150.stdout: auth mon ticket ttl = 660 # 11m 2024-02-05T04:51:44.960 INFO:teuthology.orchestra.run.smithi150.stdout: auth service ticket ttl = 240 # 4m 2024-02-05T04:51:44.960 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:51:44.960 INFO:teuthology.orchestra.run.smithi150.stdout: # don't complain about insecure global_id in the test suite 2024-02-05T04:51:44.960 INFO:teuthology.orchestra.run.smithi150.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-02-05T04:51:44.961 INFO:teuthology.orchestra.run.smithi150.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-02-05T04:51:44.961 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:51:44.961 INFO:teuthology.orchestra.run.smithi150.stdout: # 1m isn't quite enough 2024-02-05T04:51:44.961 INFO:teuthology.orchestra.run.smithi150.stdout: mon_down_mkfs_grace = 2m 2024-02-05T04:51:44.961 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:51:44.961 INFO:teuthology.orchestra.run.smithi150.stdout: mon_warn_on_filestore_osds = false 2024-02-05T04:51:44.961 INFO:teuthology.orchestra.run.smithi150.stdout: mon scrub interval = 300 2024-02-05T04:51:44.961 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-05T04:51:44.961 INFO:teuthology.orchestra.run.smithi150.stdout:[client] 2024-02-05T04:51:44.961 INFO:teuthology.orchestra.run.smithi150.stdout: rgw cache enabled = true 2024-02-05T04:51:44.961 INFO:teuthology.orchestra.run.smithi150.stdout: rgw enable ops log = true 2024-02-05T04:51:44.961 INFO:teuthology.orchestra.run.smithi150.stdout: rgw enable usage log = true 2024-02-05T04:51:44.961 INFO:teuthology.orchestra.run.smithi150.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-02-05T04:51:44.961 INFO:teuthology.orchestra.run.smithi150.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-02-05T04:51:44.961 INFO:teuthology.orchestra.run.smithi150.stdout:[mon.a] 2024-02-05T04:51:44.961 INFO:teuthology.orchestra.run.smithi150.stdout:[mon.b] 2024-02-05T04:51:44.962 INFO:teuthology.orchestra.run.smithi150.stdout:[mon.c] 2024-02-05T04:51:44.962 INFO:tasks.ceph:Creating admin key on mon.a... 2024-02-05T04:51:44.962 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-02-05T04:51:45.044 INFO:tasks.ceph:Copying monmap to all nodes... 2024-02-05T04:51:45.044 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-02-05T04:51:45.044 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-02-05T04:51:45.059 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-02-05T04:51:45.059 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-02-05T04:51:45.113 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi062.front.sepia.ceph.com 2024-02-05T04:51:45.113 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-02-05T04:51:45.113 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/etc/ceph/ceph.keyring 2024-02-05T04:51:45.113 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-02-05T04:51:45.191 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-02-05T04:51:45.191 DEBUG:teuthology.orchestra.run.smithi062:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-02-05T04:51:45.246 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi091.front.sepia.ceph.com 2024-02-05T04:51:45.246 DEBUG:teuthology.orchestra.run.smithi091:> set -ex 2024-02-05T04:51:45.246 DEBUG:teuthology.orchestra.run.smithi091:> sudo dd of=/etc/ceph/ceph.keyring 2024-02-05T04:51:45.246 DEBUG:teuthology.orchestra.run.smithi091:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-02-05T04:51:45.290 DEBUG:teuthology.orchestra.run.smithi091:> set -ex 2024-02-05T04:51:45.291 DEBUG:teuthology.orchestra.run.smithi091:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-02-05T04:51:45.346 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi150.front.sepia.ceph.com 2024-02-05T04:51:45.347 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-02-05T04:51:45.347 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/ceph/ceph.keyring 2024-02-05T04:51:45.347 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-02-05T04:51:45.391 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-02-05T04:51:45.392 DEBUG:teuthology.orchestra.run.smithi150:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-02-05T04:51:45.449 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi196.front.sepia.ceph.com 2024-02-05T04:51:45.449 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-02-05T04:51:45.449 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/etc/ceph/ceph.keyring 2024-02-05T04:51:45.449 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-02-05T04:51:45.490 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-02-05T04:51:45.490 DEBUG:teuthology.orchestra.run.smithi196:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-02-05T04:51:45.545 INFO:tasks.ceph:Setting up mon nodes... 2024-02-05T04:51:45.545 INFO:tasks.ceph:Setting up mgr nodes... 2024-02-05T04:51:45.545 DEBUG:teuthology.orchestra.run.smithi062:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-02-05T04:51:45.599 INFO:teuthology.orchestra.run.smithi062.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-02-05T04:51:45.602 DEBUG:teuthology.orchestra.run.smithi196:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-02-05T04:51:45.672 INFO:teuthology.orchestra.run.smithi196.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-02-05T04:51:45.673 INFO:tasks.ceph:Setting up mds nodes... 2024-02-05T04:51:45.673 INFO:tasks.ceph_client:Setting up client nodes... 2024-02-05T04:51:45.673 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-02-05T04:51:45.714 INFO:teuthology.orchestra.run.smithi196.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-02-05T04:51:45.729 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-02-05T04:51:45.730 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi062.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi091.front.sepia.ceph.com'): {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi150.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi196.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'}} 2024-02-05T04:51:45.730 DEBUG:teuthology.orchestra.run.smithi062:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-02-05T04:51:45.756 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-02-05T04:51:45.756 INFO:tasks.ceph:role: osd.0 2024-02-05T04:51:45.756 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi062.front.sepia.ceph.com 2024-02-05T04:51:45.757 DEBUG:teuthology.orchestra.run.smithi062:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-02-05T04:51:45.830 INFO:teuthology.orchestra.run.smithi062.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-02-05T04:51:45.830 INFO:teuthology.orchestra.run.smithi062.stdout: = sectsz=512 attr=2, projid32bit=1 2024-02-05T04:51:45.830 INFO:teuthology.orchestra.run.smithi062.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-02-05T04:51:45.830 INFO:teuthology.orchestra.run.smithi062.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-02-05T04:51:45.831 INFO:teuthology.orchestra.run.smithi062.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-02-05T04:51:45.831 INFO:teuthology.orchestra.run.smithi062.stdout: = sunit=0 swidth=0 blks 2024-02-05T04:51:45.831 INFO:teuthology.orchestra.run.smithi062.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-02-05T04:51:45.831 INFO:teuthology.orchestra.run.smithi062.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-02-05T04:51:45.831 INFO:teuthology.orchestra.run.smithi062.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-02-05T04:51:45.831 INFO:teuthology.orchestra.run.smithi062.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-02-05T04:51:46.339 INFO:teuthology.orchestra.run.smithi062.stdout:Discarding blocks...Done. 2024-02-05T04:51:46.341 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi062.front.sepia.ceph.com -o noatime 2024-02-05T04:51:46.341 DEBUG:teuthology.orchestra.run.smithi062:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-02-05T04:51:46.474 DEBUG:teuthology.orchestra.run.smithi062:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-02-05T04:51:46.512 DEBUG:teuthology.orchestra.run.smithi062:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-02-05T04:51:46.579 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-02-05T04:51:46.579 INFO:tasks.ceph:role: osd.4 2024-02-05T04:51:46.579 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi062.front.sepia.ceph.com 2024-02-05T04:51:46.579 DEBUG:teuthology.orchestra.run.smithi062:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-02-05T04:51:46.653 INFO:teuthology.orchestra.run.smithi062.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-02-05T04:51:46.653 INFO:teuthology.orchestra.run.smithi062.stdout: = sectsz=512 attr=2, projid32bit=1 2024-02-05T04:51:46.653 INFO:teuthology.orchestra.run.smithi062.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-02-05T04:51:46.653 INFO:teuthology.orchestra.run.smithi062.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-02-05T04:51:46.653 INFO:teuthology.orchestra.run.smithi062.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-02-05T04:51:46.653 INFO:teuthology.orchestra.run.smithi062.stdout: = sunit=0 swidth=0 blks 2024-02-05T04:51:46.654 INFO:teuthology.orchestra.run.smithi062.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-02-05T04:51:46.654 INFO:teuthology.orchestra.run.smithi062.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-02-05T04:51:46.654 INFO:teuthology.orchestra.run.smithi062.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-02-05T04:51:46.654 INFO:teuthology.orchestra.run.smithi062.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-02-05T04:51:47.105 INFO:teuthology.orchestra.run.smithi062.stdout:Discarding blocks...Done. 2024-02-05T04:51:47.107 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi062.front.sepia.ceph.com -o noatime 2024-02-05T04:51:47.107 DEBUG:teuthology.orchestra.run.smithi062:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2024-02-05T04:51:47.194 DEBUG:teuthology.orchestra.run.smithi062:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-02-05T04:51:47.224 DEBUG:teuthology.orchestra.run.smithi062:> sudo mkdir -p /var/lib/ceph/osd/ceph-8 2024-02-05T04:51:47.290 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-02-05T04:51:47.290 INFO:tasks.ceph:role: osd.8 2024-02-05T04:51:47.290 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi062.front.sepia.ceph.com 2024-02-05T04:51:47.291 DEBUG:teuthology.orchestra.run.smithi062:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-02-05T04:51:47.364 INFO:teuthology.orchestra.run.smithi062.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-02-05T04:51:47.365 INFO:teuthology.orchestra.run.smithi062.stdout: = sectsz=512 attr=2, projid32bit=1 2024-02-05T04:51:47.365 INFO:teuthology.orchestra.run.smithi062.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-02-05T04:51:47.365 INFO:teuthology.orchestra.run.smithi062.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-02-05T04:51:47.365 INFO:teuthology.orchestra.run.smithi062.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-02-05T04:51:47.365 INFO:teuthology.orchestra.run.smithi062.stdout: = sunit=0 swidth=0 blks 2024-02-05T04:51:47.365 INFO:teuthology.orchestra.run.smithi062.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-02-05T04:51:47.365 INFO:teuthology.orchestra.run.smithi062.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-02-05T04:51:47.365 INFO:teuthology.orchestra.run.smithi062.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-02-05T04:51:47.365 INFO:teuthology.orchestra.run.smithi062.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-02-05T04:51:47.849 INFO:teuthology.orchestra.run.smithi062.stdout:Discarding blocks...Done. 2024-02-05T04:51:47.851 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi062.front.sepia.ceph.com -o noatime 2024-02-05T04:51:47.851 DEBUG:teuthology.orchestra.run.smithi062:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-8 2024-02-05T04:51:47.989 DEBUG:teuthology.orchestra.run.smithi062:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-8 2024-02-05T04:51:48.020 DEBUG:teuthology.orchestra.run.smithi062:> sudo mkdir -p /var/lib/ceph/osd/ceph-12 2024-02-05T04:51:48.086 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-02-05T04:51:48.086 INFO:tasks.ceph:role: osd.12 2024-02-05T04:51:48.086 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi062.front.sepia.ceph.com 2024-02-05T04:51:48.086 DEBUG:teuthology.orchestra.run.smithi062:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-02-05T04:51:48.161 INFO:teuthology.orchestra.run.smithi062.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-02-05T04:51:48.161 INFO:teuthology.orchestra.run.smithi062.stdout: = sectsz=512 attr=2, projid32bit=1 2024-02-05T04:51:48.161 INFO:teuthology.orchestra.run.smithi062.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-02-05T04:51:48.161 INFO:teuthology.orchestra.run.smithi062.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-02-05T04:51:48.161 INFO:teuthology.orchestra.run.smithi062.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-02-05T04:51:48.161 INFO:teuthology.orchestra.run.smithi062.stdout: = sunit=0 swidth=0 blks 2024-02-05T04:51:48.161 INFO:teuthology.orchestra.run.smithi062.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-02-05T04:51:48.161 INFO:teuthology.orchestra.run.smithi062.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-02-05T04:51:48.161 INFO:teuthology.orchestra.run.smithi062.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-02-05T04:51:48.161 INFO:teuthology.orchestra.run.smithi062.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-02-05T04:51:48.670 INFO:teuthology.orchestra.run.smithi062.stdout:Discarding blocks...Done. 2024-02-05T04:51:48.672 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi062.front.sepia.ceph.com -o noatime 2024-02-05T04:51:48.672 DEBUG:teuthology.orchestra.run.smithi062:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-12 2024-02-05T04:51:48.811 DEBUG:teuthology.orchestra.run.smithi062:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-12 2024-02-05T04:51:48.845 DEBUG:teuthology.orchestra.run.smithi062:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-02-05T04:51:48.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-02-05T04:51:48.963+0000 7f587a2164c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-02-05T04:51:48.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-02-05T04:51:48.963+0000 7f587a2164c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-02-05T04:51:48.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-02-05T04:51:48.963+0000 7f587a2164c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-02-05T04:51:48.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-02-05T04:51:48.963+0000 7f587a2164c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-02-05T04:51:48.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-02-05T04:51:48.963+0000 7f587a2164c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-02-05T04:51:48.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-02-05T04:51:48.964+0000 7f587a2164c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-02-05T04:51:51.824 DEBUG:teuthology.orchestra.run.smithi062:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-02-05T04:51:51.856 DEBUG:teuthology.orchestra.run.smithi062:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-02-05T04:51:51.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-02-05T04:51:51.946+0000 7f4a98b704c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-02-05T04:51:51.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-02-05T04:51:51.946+0000 7f4a98b704c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-02-05T04:51:51.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-02-05T04:51:51.946+0000 7f4a98b704c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-02-05T04:51:51.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-02-05T04:51:51.946+0000 7f4a98b704c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-02-05T04:51:51.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-02-05T04:51:51.946+0000 7f4a98b704c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-02-05T04:51:51.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-02-05T04:51:51.947+0000 7f4a98b704c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-02-05T04:51:54.798 DEBUG:teuthology.orchestra.run.smithi062:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-02-05T04:51:54.830 DEBUG:teuthology.orchestra.run.smithi062:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 8 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-02-05T04:51:54.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-02-05T04:51:54.921+0000 7f8df30204c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-8/keyring: can't open /var/lib/ceph/osd/ceph-8/keyring: (2) No such file or directory 2024-02-05T04:51:54.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-02-05T04:51:54.921+0000 7f8df30204c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-8/keyring 2024-02-05T04:51:54.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-02-05T04:51:54.921+0000 7f8df30204c0 -1 bluestore(/var/lib/ceph/osd/ceph-8/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-8/block: (2) No such file or directory 2024-02-05T04:51:54.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-02-05T04:51:54.922+0000 7f8df30204c0 -1 bluestore(/var/lib/ceph/osd/ceph-8/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-8/block: (2) No such file or directory 2024-02-05T04:51:54.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-02-05T04:51:54.922+0000 7f8df30204c0 -1 bluestore(/var/lib/ceph/osd/ceph-8/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-8/block: (2) No such file or directory 2024-02-05T04:51:54.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-02-05T04:51:54.922+0000 7f8df30204c0 -1 bluestore(/var/lib/ceph/osd/ceph-8) _read_fsid unparsable uuid 2024-02-05T04:51:57.775 DEBUG:teuthology.orchestra.run.smithi062:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-8 2024-02-05T04:51:57.806 DEBUG:teuthology.orchestra.run.smithi062:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 12 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-02-05T04:51:57.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-02-05T04:51:57.896+0000 7fea1f8204c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-12/keyring: can't open /var/lib/ceph/osd/ceph-12/keyring: (2) No such file or directory 2024-02-05T04:51:57.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-02-05T04:51:57.896+0000 7fea1f8204c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-12/keyring 2024-02-05T04:51:57.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-02-05T04:51:57.896+0000 7fea1f8204c0 -1 bluestore(/var/lib/ceph/osd/ceph-12/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-12/block: (2) No such file or directory 2024-02-05T04:51:57.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-02-05T04:51:57.896+0000 7fea1f8204c0 -1 bluestore(/var/lib/ceph/osd/ceph-12/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-12/block: (2) No such file or directory 2024-02-05T04:51:57.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-02-05T04:51:57.896+0000 7fea1f8204c0 -1 bluestore(/var/lib/ceph/osd/ceph-12/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-12/block: (2) No such file or directory 2024-02-05T04:51:57.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-02-05T04:51:57.897+0000 7fea1f8204c0 -1 bluestore(/var/lib/ceph/osd/ceph-12) _read_fsid unparsable uuid 2024-02-05T04:52:00.747 DEBUG:teuthology.orchestra.run.smithi062:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-12 2024-02-05T04:52:00.778 DEBUG:teuthology.orchestra.run.smithi091:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-02-05T04:52:00.810 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-02-05T04:52:00.811 INFO:tasks.ceph:role: osd.1 2024-02-05T04:52:00.811 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi091.front.sepia.ceph.com 2024-02-05T04:52:00.811 DEBUG:teuthology.orchestra.run.smithi091:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-02-05T04:52:00.888 INFO:teuthology.orchestra.run.smithi091.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-02-05T04:52:00.889 INFO:teuthology.orchestra.run.smithi091.stdout: = sectsz=512 attr=2, projid32bit=1 2024-02-05T04:52:00.889 INFO:teuthology.orchestra.run.smithi091.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-02-05T04:52:00.889 INFO:teuthology.orchestra.run.smithi091.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-02-05T04:52:00.889 INFO:teuthology.orchestra.run.smithi091.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-02-05T04:52:00.889 INFO:teuthology.orchestra.run.smithi091.stdout: = sunit=0 swidth=0 blks 2024-02-05T04:52:00.889 INFO:teuthology.orchestra.run.smithi091.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-02-05T04:52:00.889 INFO:teuthology.orchestra.run.smithi091.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-02-05T04:52:00.889 INFO:teuthology.orchestra.run.smithi091.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-02-05T04:52:00.889 INFO:teuthology.orchestra.run.smithi091.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-02-05T04:52:01.398 INFO:teuthology.orchestra.run.smithi091.stdout:Discarding blocks...Done. 2024-02-05T04:52:01.400 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi091.front.sepia.ceph.com -o noatime 2024-02-05T04:52:01.400 DEBUG:teuthology.orchestra.run.smithi091:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-1 2024-02-05T04:52:01.518 DEBUG:teuthology.orchestra.run.smithi091:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-02-05T04:52:01.555 DEBUG:teuthology.orchestra.run.smithi091:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-02-05T04:52:01.624 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-02-05T04:52:01.624 INFO:tasks.ceph:role: osd.5 2024-02-05T04:52:01.624 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi091.front.sepia.ceph.com 2024-02-05T04:52:01.624 DEBUG:teuthology.orchestra.run.smithi091:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-02-05T04:52:01.700 INFO:teuthology.orchestra.run.smithi091.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-02-05T04:52:01.700 INFO:teuthology.orchestra.run.smithi091.stdout: = sectsz=512 attr=2, projid32bit=1 2024-02-05T04:52:01.701 INFO:teuthology.orchestra.run.smithi091.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-02-05T04:52:01.701 INFO:teuthology.orchestra.run.smithi091.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-02-05T04:52:01.701 INFO:teuthology.orchestra.run.smithi091.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-02-05T04:52:01.701 INFO:teuthology.orchestra.run.smithi091.stdout: = sunit=0 swidth=0 blks 2024-02-05T04:52:01.701 INFO:teuthology.orchestra.run.smithi091.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-02-05T04:52:01.701 INFO:teuthology.orchestra.run.smithi091.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-02-05T04:52:01.701 INFO:teuthology.orchestra.run.smithi091.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-02-05T04:52:01.701 INFO:teuthology.orchestra.run.smithi091.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-02-05T04:52:02.163 INFO:teuthology.orchestra.run.smithi091.stdout:Discarding blocks...Done. 2024-02-05T04:52:02.165 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi091.front.sepia.ceph.com -o noatime 2024-02-05T04:52:02.165 DEBUG:teuthology.orchestra.run.smithi091:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-02-05T04:52:02.262 DEBUG:teuthology.orchestra.run.smithi091:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-02-05T04:52:02.293 DEBUG:teuthology.orchestra.run.smithi091:> sudo mkdir -p /var/lib/ceph/osd/ceph-9 2024-02-05T04:52:02.361 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-02-05T04:52:02.361 INFO:tasks.ceph:role: osd.9 2024-02-05T04:52:02.361 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi091.front.sepia.ceph.com 2024-02-05T04:52:02.361 DEBUG:teuthology.orchestra.run.smithi091:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-02-05T04:52:02.435 INFO:teuthology.orchestra.run.smithi091.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-02-05T04:52:02.436 INFO:teuthology.orchestra.run.smithi091.stdout: = sectsz=512 attr=2, projid32bit=1 2024-02-05T04:52:02.436 INFO:teuthology.orchestra.run.smithi091.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-02-05T04:52:02.436 INFO:teuthology.orchestra.run.smithi091.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-02-05T04:52:02.436 INFO:teuthology.orchestra.run.smithi091.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-02-05T04:52:02.436 INFO:teuthology.orchestra.run.smithi091.stdout: = sunit=0 swidth=0 blks 2024-02-05T04:52:02.436 INFO:teuthology.orchestra.run.smithi091.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-02-05T04:52:02.436 INFO:teuthology.orchestra.run.smithi091.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-02-05T04:52:02.436 INFO:teuthology.orchestra.run.smithi091.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-02-05T04:52:02.436 INFO:teuthology.orchestra.run.smithi091.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-02-05T04:52:02.929 INFO:teuthology.orchestra.run.smithi091.stdout:Discarding blocks...Done. 2024-02-05T04:52:02.930 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi091.front.sepia.ceph.com -o noatime 2024-02-05T04:52:02.931 DEBUG:teuthology.orchestra.run.smithi091:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-9 2024-02-05T04:52:03.079 DEBUG:teuthology.orchestra.run.smithi091:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-9 2024-02-05T04:52:03.111 DEBUG:teuthology.orchestra.run.smithi091:> sudo mkdir -p /var/lib/ceph/osd/ceph-13 2024-02-05T04:52:03.178 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-02-05T04:52:03.178 INFO:tasks.ceph:role: osd.13 2024-02-05T04:52:03.178 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi091.front.sepia.ceph.com 2024-02-05T04:52:03.178 DEBUG:teuthology.orchestra.run.smithi091:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-02-05T04:52:03.252 INFO:teuthology.orchestra.run.smithi091.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-02-05T04:52:03.252 INFO:teuthology.orchestra.run.smithi091.stdout: = sectsz=512 attr=2, projid32bit=1 2024-02-05T04:52:03.252 INFO:teuthology.orchestra.run.smithi091.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-02-05T04:52:03.253 INFO:teuthology.orchestra.run.smithi091.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-02-05T04:52:03.253 INFO:teuthology.orchestra.run.smithi091.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-02-05T04:52:03.253 INFO:teuthology.orchestra.run.smithi091.stdout: = sunit=0 swidth=0 blks 2024-02-05T04:52:03.253 INFO:teuthology.orchestra.run.smithi091.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-02-05T04:52:03.253 INFO:teuthology.orchestra.run.smithi091.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-02-05T04:52:03.253 INFO:teuthology.orchestra.run.smithi091.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-02-05T04:52:03.253 INFO:teuthology.orchestra.run.smithi091.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-02-05T04:52:03.774 INFO:teuthology.orchestra.run.smithi091.stdout:Discarding blocks...Done. 2024-02-05T04:52:03.776 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi091.front.sepia.ceph.com -o noatime 2024-02-05T04:52:03.776 DEBUG:teuthology.orchestra.run.smithi091:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-13 2024-02-05T04:52:03.922 DEBUG:teuthology.orchestra.run.smithi091:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-13 2024-02-05T04:52:03.954 DEBUG:teuthology.orchestra.run.smithi091:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-02-05T04:52:04.055 INFO:teuthology.orchestra.run.smithi091.stderr:2024-02-05T04:52:04.054+0000 7f5b171744c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-02-05T04:52:04.056 INFO:teuthology.orchestra.run.smithi091.stderr:2024-02-05T04:52:04.055+0000 7f5b171744c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-02-05T04:52:04.056 INFO:teuthology.orchestra.run.smithi091.stderr:2024-02-05T04:52:04.055+0000 7f5b171744c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-02-05T04:52:04.056 INFO:teuthology.orchestra.run.smithi091.stderr:2024-02-05T04:52:04.055+0000 7f5b171744c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-02-05T04:52:04.056 INFO:teuthology.orchestra.run.smithi091.stderr:2024-02-05T04:52:04.055+0000 7f5b171744c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-02-05T04:52:04.056 INFO:teuthology.orchestra.run.smithi091.stderr:2024-02-05T04:52:04.055+0000 7f5b171744c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-02-05T04:52:06.905 DEBUG:teuthology.orchestra.run.smithi091:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-02-05T04:52:06.936 DEBUG:teuthology.orchestra.run.smithi091:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-02-05T04:52:07.030 INFO:teuthology.orchestra.run.smithi091.stderr:2024-02-05T04:52:07.029+0000 7f06f28d64c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-02-05T04:52:07.031 INFO:teuthology.orchestra.run.smithi091.stderr:2024-02-05T04:52:07.030+0000 7f06f28d64c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-02-05T04:52:07.031 INFO:teuthology.orchestra.run.smithi091.stderr:2024-02-05T04:52:07.030+0000 7f06f28d64c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-02-05T04:52:07.031 INFO:teuthology.orchestra.run.smithi091.stderr:2024-02-05T04:52:07.030+0000 7f06f28d64c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-02-05T04:52:07.031 INFO:teuthology.orchestra.run.smithi091.stderr:2024-02-05T04:52:07.030+0000 7f06f28d64c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-02-05T04:52:07.031 INFO:teuthology.orchestra.run.smithi091.stderr:2024-02-05T04:52:07.030+0000 7f06f28d64c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-02-05T04:52:09.879 DEBUG:teuthology.orchestra.run.smithi091:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-02-05T04:52:09.910 DEBUG:teuthology.orchestra.run.smithi091:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 9 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-02-05T04:52:10.001 INFO:teuthology.orchestra.run.smithi091.stderr:2024-02-05T04:52:10.000+0000 7fb39708a4c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-9/keyring: can't open /var/lib/ceph/osd/ceph-9/keyring: (2) No such file or directory 2024-02-05T04:52:10.001 INFO:teuthology.orchestra.run.smithi091.stderr:2024-02-05T04:52:10.000+0000 7fb39708a4c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-9/keyring 2024-02-05T04:52:10.001 INFO:teuthology.orchestra.run.smithi091.stderr:2024-02-05T04:52:10.000+0000 7fb39708a4c0 -1 bluestore(/var/lib/ceph/osd/ceph-9/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-9/block: (2) No such file or directory 2024-02-05T04:52:10.001 INFO:teuthology.orchestra.run.smithi091.stderr:2024-02-05T04:52:10.000+0000 7fb39708a4c0 -1 bluestore(/var/lib/ceph/osd/ceph-9/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-9/block: (2) No such file or directory 2024-02-05T04:52:10.001 INFO:teuthology.orchestra.run.smithi091.stderr:2024-02-05T04:52:10.000+0000 7fb39708a4c0 -1 bluestore(/var/lib/ceph/osd/ceph-9/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-9/block: (2) No such file or directory 2024-02-05T04:52:10.001 INFO:teuthology.orchestra.run.smithi091.stderr:2024-02-05T04:52:10.001+0000 7fb39708a4c0 -1 bluestore(/var/lib/ceph/osd/ceph-9) _read_fsid unparsable uuid 2024-02-05T04:52:12.852 DEBUG:teuthology.orchestra.run.smithi091:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-9 2024-02-05T04:52:12.883 DEBUG:teuthology.orchestra.run.smithi091:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 13 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-02-05T04:52:12.973 INFO:teuthology.orchestra.run.smithi091.stderr:2024-02-05T04:52:12.971+0000 7fbf139fe4c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-13/keyring: can't open /var/lib/ceph/osd/ceph-13/keyring: (2) No such file or directory 2024-02-05T04:52:12.973 INFO:teuthology.orchestra.run.smithi091.stderr:2024-02-05T04:52:12.972+0000 7fbf139fe4c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-13/keyring 2024-02-05T04:52:12.973 INFO:teuthology.orchestra.run.smithi091.stderr:2024-02-05T04:52:12.972+0000 7fbf139fe4c0 -1 bluestore(/var/lib/ceph/osd/ceph-13/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-13/block: (2) No such file or directory 2024-02-05T04:52:12.973 INFO:teuthology.orchestra.run.smithi091.stderr:2024-02-05T04:52:12.972+0000 7fbf139fe4c0 -1 bluestore(/var/lib/ceph/osd/ceph-13/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-13/block: (2) No such file or directory 2024-02-05T04:52:12.973 INFO:teuthology.orchestra.run.smithi091.stderr:2024-02-05T04:52:12.972+0000 7fbf139fe4c0 -1 bluestore(/var/lib/ceph/osd/ceph-13/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-13/block: (2) No such file or directory 2024-02-05T04:52:12.973 INFO:teuthology.orchestra.run.smithi091.stderr:2024-02-05T04:52:12.972+0000 7fbf139fe4c0 -1 bluestore(/var/lib/ceph/osd/ceph-13) _read_fsid unparsable uuid 2024-02-05T04:52:15.836 DEBUG:teuthology.orchestra.run.smithi091:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-13 2024-02-05T04:52:15.888 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-02-05T04:52:15.923 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-02-05T04:52:15.923 INFO:tasks.ceph:role: osd.2 2024-02-05T04:52:15.923 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi150.front.sepia.ceph.com 2024-02-05T04:52:15.923 DEBUG:teuthology.orchestra.run.smithi150:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-02-05T04:52:16.001 INFO:teuthology.orchestra.run.smithi150.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-02-05T04:52:16.001 INFO:teuthology.orchestra.run.smithi150.stdout: = sectsz=512 attr=2, projid32bit=1 2024-02-05T04:52:16.001 INFO:teuthology.orchestra.run.smithi150.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-02-05T04:52:16.001 INFO:teuthology.orchestra.run.smithi150.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-02-05T04:52:16.001 INFO:teuthology.orchestra.run.smithi150.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-02-05T04:52:16.001 INFO:teuthology.orchestra.run.smithi150.stdout: = sunit=0 swidth=0 blks 2024-02-05T04:52:16.001 INFO:teuthology.orchestra.run.smithi150.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-02-05T04:52:16.001 INFO:teuthology.orchestra.run.smithi150.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-02-05T04:52:16.001 INFO:teuthology.orchestra.run.smithi150.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-02-05T04:52:16.001 INFO:teuthology.orchestra.run.smithi150.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-02-05T04:52:16.511 INFO:teuthology.orchestra.run.smithi150.stdout:Discarding blocks...Done. 2024-02-05T04:52:16.512 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi150.front.sepia.ceph.com -o noatime 2024-02-05T04:52:16.513 DEBUG:teuthology.orchestra.run.smithi150:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-2 2024-02-05T04:52:16.650 DEBUG:teuthology.orchestra.run.smithi150:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-02-05T04:52:16.690 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-02-05T04:52:16.760 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-02-05T04:52:16.760 INFO:tasks.ceph:role: osd.6 2024-02-05T04:52:16.760 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi150.front.sepia.ceph.com 2024-02-05T04:52:16.760 DEBUG:teuthology.orchestra.run.smithi150:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-02-05T04:52:16.837 INFO:teuthology.orchestra.run.smithi150.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-02-05T04:52:16.837 INFO:teuthology.orchestra.run.smithi150.stdout: = sectsz=512 attr=2, projid32bit=1 2024-02-05T04:52:16.837 INFO:teuthology.orchestra.run.smithi150.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-02-05T04:52:16.837 INFO:teuthology.orchestra.run.smithi150.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-02-05T04:52:16.837 INFO:teuthology.orchestra.run.smithi150.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-02-05T04:52:16.837 INFO:teuthology.orchestra.run.smithi150.stdout: = sunit=0 swidth=0 blks 2024-02-05T04:52:16.838 INFO:teuthology.orchestra.run.smithi150.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-02-05T04:52:16.838 INFO:teuthology.orchestra.run.smithi150.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-02-05T04:52:16.838 INFO:teuthology.orchestra.run.smithi150.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-02-05T04:52:16.838 INFO:teuthology.orchestra.run.smithi150.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-02-05T04:52:17.279 INFO:teuthology.orchestra.run.smithi150.stdout:Discarding blocks...Done. 2024-02-05T04:52:17.280 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi150.front.sepia.ceph.com -o noatime 2024-02-05T04:52:17.281 DEBUG:teuthology.orchestra.run.smithi150:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-6 2024-02-05T04:52:17.373 DEBUG:teuthology.orchestra.run.smithi150:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-02-05T04:52:17.403 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /var/lib/ceph/osd/ceph-10 2024-02-05T04:52:17.471 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-02-05T04:52:17.471 INFO:tasks.ceph:role: osd.10 2024-02-05T04:52:17.471 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi150.front.sepia.ceph.com 2024-02-05T04:52:17.472 DEBUG:teuthology.orchestra.run.smithi150:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-02-05T04:52:17.545 INFO:teuthology.orchestra.run.smithi150.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-02-05T04:52:17.545 INFO:teuthology.orchestra.run.smithi150.stdout: = sectsz=512 attr=2, projid32bit=1 2024-02-05T04:52:17.545 INFO:teuthology.orchestra.run.smithi150.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-02-05T04:52:17.546 INFO:teuthology.orchestra.run.smithi150.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-02-05T04:52:17.546 INFO:teuthology.orchestra.run.smithi150.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-02-05T04:52:17.546 INFO:teuthology.orchestra.run.smithi150.stdout: = sunit=0 swidth=0 blks 2024-02-05T04:52:17.546 INFO:teuthology.orchestra.run.smithi150.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-02-05T04:52:17.546 INFO:teuthology.orchestra.run.smithi150.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-02-05T04:52:17.546 INFO:teuthology.orchestra.run.smithi150.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-02-05T04:52:17.546 INFO:teuthology.orchestra.run.smithi150.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-02-05T04:52:18.041 INFO:teuthology.orchestra.run.smithi150.stdout:Discarding blocks...Done. 2024-02-05T04:52:18.044 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi150.front.sepia.ceph.com -o noatime 2024-02-05T04:52:18.044 DEBUG:teuthology.orchestra.run.smithi150:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-10 2024-02-05T04:52:18.165 DEBUG:teuthology.orchestra.run.smithi150:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-10 2024-02-05T04:52:18.199 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /var/lib/ceph/osd/ceph-14 2024-02-05T04:52:18.267 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-02-05T04:52:18.268 INFO:tasks.ceph:role: osd.14 2024-02-05T04:52:18.268 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi150.front.sepia.ceph.com 2024-02-05T04:52:18.268 DEBUG:teuthology.orchestra.run.smithi150:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-02-05T04:52:18.342 INFO:teuthology.orchestra.run.smithi150.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-02-05T04:52:18.343 INFO:teuthology.orchestra.run.smithi150.stdout: = sectsz=512 attr=2, projid32bit=1 2024-02-05T04:52:18.343 INFO:teuthology.orchestra.run.smithi150.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-02-05T04:52:18.343 INFO:teuthology.orchestra.run.smithi150.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-02-05T04:52:18.343 INFO:teuthology.orchestra.run.smithi150.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-02-05T04:52:18.343 INFO:teuthology.orchestra.run.smithi150.stdout: = sunit=0 swidth=0 blks 2024-02-05T04:52:18.343 INFO:teuthology.orchestra.run.smithi150.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-02-05T04:52:18.343 INFO:teuthology.orchestra.run.smithi150.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-02-05T04:52:18.343 INFO:teuthology.orchestra.run.smithi150.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-02-05T04:52:18.343 INFO:teuthology.orchestra.run.smithi150.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-02-05T04:52:18.845 INFO:teuthology.orchestra.run.smithi150.stdout:Discarding blocks...Done. 2024-02-05T04:52:18.847 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi150.front.sepia.ceph.com -o noatime 2024-02-05T04:52:18.847 DEBUG:teuthology.orchestra.run.smithi150:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-14 2024-02-05T04:52:18.973 DEBUG:teuthology.orchestra.run.smithi150:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-14 2024-02-05T04:52:19.011 DEBUG:teuthology.orchestra.run.smithi150:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-02-05T04:52:19.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-05T04:52:19.113+0000 7fa6d7a0d4c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-02-05T04:52:19.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-05T04:52:19.113+0000 7fa6d7a0d4c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-02-05T04:52:19.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-05T04:52:19.113+0000 7fa6d7a0d4c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-02-05T04:52:19.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-05T04:52:19.113+0000 7fa6d7a0d4c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-02-05T04:52:19.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-05T04:52:19.113+0000 7fa6d7a0d4c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-02-05T04:52:19.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-05T04:52:19.114+0000 7fa6d7a0d4c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-02-05T04:52:21.972 DEBUG:teuthology.orchestra.run.smithi150:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-02-05T04:52:22.005 DEBUG:teuthology.orchestra.run.smithi150:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-02-05T04:52:22.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-05T04:52:22.094+0000 7feda5e1f4c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-02-05T04:52:22.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-05T04:52:22.094+0000 7feda5e1f4c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-02-05T04:52:22.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-05T04:52:22.094+0000 7feda5e1f4c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-02-05T04:52:22.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-05T04:52:22.094+0000 7feda5e1f4c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-02-05T04:52:22.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-05T04:52:22.094+0000 7feda5e1f4c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-02-05T04:52:22.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-05T04:52:22.095+0000 7feda5e1f4c0 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-02-05T04:52:24.949 DEBUG:teuthology.orchestra.run.smithi150:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-02-05T04:52:24.980 DEBUG:teuthology.orchestra.run.smithi150:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 10 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-02-05T04:52:25.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-05T04:52:25.069+0000 7f1fc83374c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-10/keyring: can't open /var/lib/ceph/osd/ceph-10/keyring: (2) No such file or directory 2024-02-05T04:52:25.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-05T04:52:25.069+0000 7f1fc83374c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-10/keyring 2024-02-05T04:52:25.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-05T04:52:25.069+0000 7f1fc83374c0 -1 bluestore(/var/lib/ceph/osd/ceph-10/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-10/block: (2) No such file or directory 2024-02-05T04:52:25.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-05T04:52:25.069+0000 7f1fc83374c0 -1 bluestore(/var/lib/ceph/osd/ceph-10/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-10/block: (2) No such file or directory 2024-02-05T04:52:25.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-05T04:52:25.069+0000 7f1fc83374c0 -1 bluestore(/var/lib/ceph/osd/ceph-10/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-10/block: (2) No such file or directory 2024-02-05T04:52:25.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-05T04:52:25.070+0000 7f1fc83374c0 -1 bluestore(/var/lib/ceph/osd/ceph-10) _read_fsid unparsable uuid 2024-02-05T04:52:27.921 DEBUG:teuthology.orchestra.run.smithi150:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-10 2024-02-05T04:52:27.953 DEBUG:teuthology.orchestra.run.smithi150:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 14 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-02-05T04:52:28.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-05T04:52:28.045+0000 7f7f9ac124c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-14/keyring: can't open /var/lib/ceph/osd/ceph-14/keyring: (2) No such file or directory 2024-02-05T04:52:28.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-05T04:52:28.045+0000 7f7f9ac124c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-14/keyring 2024-02-05T04:52:28.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-05T04:52:28.045+0000 7f7f9ac124c0 -1 bluestore(/var/lib/ceph/osd/ceph-14/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-14/block: (2) No such file or directory 2024-02-05T04:52:28.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-05T04:52:28.045+0000 7f7f9ac124c0 -1 bluestore(/var/lib/ceph/osd/ceph-14/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-14/block: (2) No such file or directory 2024-02-05T04:52:28.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-05T04:52:28.045+0000 7f7f9ac124c0 -1 bluestore(/var/lib/ceph/osd/ceph-14/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-14/block: (2) No such file or directory 2024-02-05T04:52:28.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-05T04:52:28.046+0000 7f7f9ac124c0 -1 bluestore(/var/lib/ceph/osd/ceph-14) _read_fsid unparsable uuid 2024-02-05T04:52:30.906 DEBUG:teuthology.orchestra.run.smithi150:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-14 2024-02-05T04:52:30.937 DEBUG:teuthology.orchestra.run.smithi196:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-02-05T04:52:30.970 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-02-05T04:52:30.970 INFO:tasks.ceph:role: osd.3 2024-02-05T04:52:30.970 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi196.front.sepia.ceph.com 2024-02-05T04:52:30.970 DEBUG:teuthology.orchestra.run.smithi196:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-02-05T04:52:31.047 INFO:teuthology.orchestra.run.smithi196.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-02-05T04:52:31.047 INFO:teuthology.orchestra.run.smithi196.stdout: = sectsz=512 attr=2, projid32bit=1 2024-02-05T04:52:31.047 INFO:teuthology.orchestra.run.smithi196.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-02-05T04:52:31.047 INFO:teuthology.orchestra.run.smithi196.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-02-05T04:52:31.047 INFO:teuthology.orchestra.run.smithi196.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-02-05T04:52:31.048 INFO:teuthology.orchestra.run.smithi196.stdout: = sunit=0 swidth=0 blks 2024-02-05T04:52:31.048 INFO:teuthology.orchestra.run.smithi196.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-02-05T04:52:31.048 INFO:teuthology.orchestra.run.smithi196.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-02-05T04:52:31.048 INFO:teuthology.orchestra.run.smithi196.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-02-05T04:52:31.048 INFO:teuthology.orchestra.run.smithi196.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-02-05T04:52:31.552 INFO:teuthology.orchestra.run.smithi196.stdout:Discarding blocks...Done. 2024-02-05T04:52:31.555 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi196.front.sepia.ceph.com -o noatime 2024-02-05T04:52:31.555 DEBUG:teuthology.orchestra.run.smithi196:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2024-02-05T04:52:31.692 DEBUG:teuthology.orchestra.run.smithi196:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-02-05T04:52:31.728 DEBUG:teuthology.orchestra.run.smithi196:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-02-05T04:52:31.798 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-02-05T04:52:31.798 INFO:tasks.ceph:role: osd.7 2024-02-05T04:52:31.798 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi196.front.sepia.ceph.com 2024-02-05T04:52:31.798 DEBUG:teuthology.orchestra.run.smithi196:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-02-05T04:52:31.874 INFO:teuthology.orchestra.run.smithi196.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-02-05T04:52:31.874 INFO:teuthology.orchestra.run.smithi196.stdout: = sectsz=512 attr=2, projid32bit=1 2024-02-05T04:52:31.874 INFO:teuthology.orchestra.run.smithi196.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-02-05T04:52:31.874 INFO:teuthology.orchestra.run.smithi196.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-02-05T04:52:31.874 INFO:teuthology.orchestra.run.smithi196.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-02-05T04:52:31.874 INFO:teuthology.orchestra.run.smithi196.stdout: = sunit=0 swidth=0 blks 2024-02-05T04:52:31.874 INFO:teuthology.orchestra.run.smithi196.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-02-05T04:52:31.874 INFO:teuthology.orchestra.run.smithi196.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-02-05T04:52:31.874 INFO:teuthology.orchestra.run.smithi196.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-02-05T04:52:31.874 INFO:teuthology.orchestra.run.smithi196.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-02-05T04:52:32.327 INFO:teuthology.orchestra.run.smithi196.stdout:Discarding blocks...Done. 2024-02-05T04:52:32.329 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi196.front.sepia.ceph.com -o noatime 2024-02-05T04:52:32.329 DEBUG:teuthology.orchestra.run.smithi196:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-7 2024-02-05T04:52:32.426 DEBUG:teuthology.orchestra.run.smithi196:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-02-05T04:52:32.458 DEBUG:teuthology.orchestra.run.smithi196:> sudo mkdir -p /var/lib/ceph/osd/ceph-11 2024-02-05T04:52:32.525 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-02-05T04:52:32.526 INFO:tasks.ceph:role: osd.11 2024-02-05T04:52:32.526 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi196.front.sepia.ceph.com 2024-02-05T04:52:32.526 DEBUG:teuthology.orchestra.run.smithi196:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-02-05T04:52:32.600 INFO:teuthology.orchestra.run.smithi196.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-02-05T04:52:32.600 INFO:teuthology.orchestra.run.smithi196.stdout: = sectsz=512 attr=2, projid32bit=1 2024-02-05T04:52:32.600 INFO:teuthology.orchestra.run.smithi196.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-02-05T04:52:32.600 INFO:teuthology.orchestra.run.smithi196.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-02-05T04:52:32.601 INFO:teuthology.orchestra.run.smithi196.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-02-05T04:52:32.601 INFO:teuthology.orchestra.run.smithi196.stdout: = sunit=0 swidth=0 blks 2024-02-05T04:52:32.601 INFO:teuthology.orchestra.run.smithi196.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-02-05T04:52:32.601 INFO:teuthology.orchestra.run.smithi196.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-02-05T04:52:32.601 INFO:teuthology.orchestra.run.smithi196.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-02-05T04:52:32.601 INFO:teuthology.orchestra.run.smithi196.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-02-05T04:52:33.092 INFO:teuthology.orchestra.run.smithi196.stdout:Discarding blocks...Done. 2024-02-05T04:52:33.093 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi196.front.sepia.ceph.com -o noatime 2024-02-05T04:52:33.094 DEBUG:teuthology.orchestra.run.smithi196:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-11 2024-02-05T04:52:33.239 DEBUG:teuthology.orchestra.run.smithi196:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-11 2024-02-05T04:52:33.271 DEBUG:teuthology.orchestra.run.smithi196:> sudo mkdir -p /var/lib/ceph/osd/ceph-15 2024-02-05T04:52:33.341 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-02-05T04:52:33.341 INFO:tasks.ceph:role: osd.15 2024-02-05T04:52:33.341 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi196.front.sepia.ceph.com 2024-02-05T04:52:33.341 DEBUG:teuthology.orchestra.run.smithi196:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-02-05T04:52:33.416 INFO:teuthology.orchestra.run.smithi196.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-02-05T04:52:33.416 INFO:teuthology.orchestra.run.smithi196.stdout: = sectsz=512 attr=2, projid32bit=1 2024-02-05T04:52:33.416 INFO:teuthology.orchestra.run.smithi196.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-02-05T04:52:33.416 INFO:teuthology.orchestra.run.smithi196.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-02-05T04:52:33.416 INFO:teuthology.orchestra.run.smithi196.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-02-05T04:52:33.416 INFO:teuthology.orchestra.run.smithi196.stdout: = sunit=0 swidth=0 blks 2024-02-05T04:52:33.416 INFO:teuthology.orchestra.run.smithi196.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-02-05T04:52:33.416 INFO:teuthology.orchestra.run.smithi196.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-02-05T04:52:33.416 INFO:teuthology.orchestra.run.smithi196.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-02-05T04:52:33.416 INFO:teuthology.orchestra.run.smithi196.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-02-05T04:52:33.926 INFO:teuthology.orchestra.run.smithi196.stdout:Discarding blocks...Done. 2024-02-05T04:52:33.929 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi196.front.sepia.ceph.com -o noatime 2024-02-05T04:52:33.929 DEBUG:teuthology.orchestra.run.smithi196:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-15 2024-02-05T04:52:34.062 DEBUG:teuthology.orchestra.run.smithi196:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-15 2024-02-05T04:52:34.097 DEBUG:teuthology.orchestra.run.smithi196:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-02-05T04:52:34.200 INFO:teuthology.orchestra.run.smithi196.stderr:2024-02-05T04:52:34.197+0000 7f8baecaf4c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-02-05T04:52:34.200 INFO:teuthology.orchestra.run.smithi196.stderr:2024-02-05T04:52:34.197+0000 7f8baecaf4c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-02-05T04:52:34.201 INFO:teuthology.orchestra.run.smithi196.stderr:2024-02-05T04:52:34.197+0000 7f8baecaf4c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-02-05T04:52:34.201 INFO:teuthology.orchestra.run.smithi196.stderr:2024-02-05T04:52:34.197+0000 7f8baecaf4c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-02-05T04:52:34.201 INFO:teuthology.orchestra.run.smithi196.stderr:2024-02-05T04:52:34.197+0000 7f8baecaf4c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-02-05T04:52:34.201 INFO:teuthology.orchestra.run.smithi196.stderr:2024-02-05T04:52:34.198+0000 7f8baecaf4c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-02-05T04:52:37.052 DEBUG:teuthology.orchestra.run.smithi196:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-02-05T04:52:37.084 DEBUG:teuthology.orchestra.run.smithi196:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-02-05T04:52:37.175 INFO:teuthology.orchestra.run.smithi196.stderr:2024-02-05T04:52:37.174+0000 7f40216c04c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-02-05T04:52:37.176 INFO:teuthology.orchestra.run.smithi196.stderr:2024-02-05T04:52:37.174+0000 7f40216c04c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-02-05T04:52:37.176 INFO:teuthology.orchestra.run.smithi196.stderr:2024-02-05T04:52:37.174+0000 7f40216c04c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-02-05T04:52:37.176 INFO:teuthology.orchestra.run.smithi196.stderr:2024-02-05T04:52:37.174+0000 7f40216c04c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-02-05T04:52:37.176 INFO:teuthology.orchestra.run.smithi196.stderr:2024-02-05T04:52:37.174+0000 7f40216c04c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-02-05T04:52:37.176 INFO:teuthology.orchestra.run.smithi196.stderr:2024-02-05T04:52:37.175+0000 7f40216c04c0 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-02-05T04:52:40.023 DEBUG:teuthology.orchestra.run.smithi196:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-02-05T04:52:40.054 DEBUG:teuthology.orchestra.run.smithi196:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 11 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-02-05T04:52:40.147 INFO:teuthology.orchestra.run.smithi196.stderr:2024-02-05T04:52:40.145+0000 7f600b9ad4c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-11/keyring: can't open /var/lib/ceph/osd/ceph-11/keyring: (2) No such file or directory 2024-02-05T04:52:40.147 INFO:teuthology.orchestra.run.smithi196.stderr:2024-02-05T04:52:40.146+0000 7f600b9ad4c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-11/keyring 2024-02-05T04:52:40.147 INFO:teuthology.orchestra.run.smithi196.stderr:2024-02-05T04:52:40.146+0000 7f600b9ad4c0 -1 bluestore(/var/lib/ceph/osd/ceph-11/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-11/block: (2) No such file or directory 2024-02-05T04:52:40.147 INFO:teuthology.orchestra.run.smithi196.stderr:2024-02-05T04:52:40.146+0000 7f600b9ad4c0 -1 bluestore(/var/lib/ceph/osd/ceph-11/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-11/block: (2) No such file or directory 2024-02-05T04:52:40.147 INFO:teuthology.orchestra.run.smithi196.stderr:2024-02-05T04:52:40.146+0000 7f600b9ad4c0 -1 bluestore(/var/lib/ceph/osd/ceph-11/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-11/block: (2) No such file or directory 2024-02-05T04:52:40.148 INFO:teuthology.orchestra.run.smithi196.stderr:2024-02-05T04:52:40.146+0000 7f600b9ad4c0 -1 bluestore(/var/lib/ceph/osd/ceph-11) _read_fsid unparsable uuid 2024-02-05T04:52:43.002 DEBUG:teuthology.orchestra.run.smithi196:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-11 2024-02-05T04:52:43.032 DEBUG:teuthology.orchestra.run.smithi196:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 15 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-02-05T04:52:43.122 INFO:teuthology.orchestra.run.smithi196.stderr:2024-02-05T04:52:43.120+0000 7f138d1de4c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-15/keyring: can't open /var/lib/ceph/osd/ceph-15/keyring: (2) No such file or directory 2024-02-05T04:52:43.122 INFO:teuthology.orchestra.run.smithi196.stderr:2024-02-05T04:52:43.120+0000 7f138d1de4c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-15/keyring 2024-02-05T04:52:43.122 INFO:teuthology.orchestra.run.smithi196.stderr:2024-02-05T04:52:43.120+0000 7f138d1de4c0 -1 bluestore(/var/lib/ceph/osd/ceph-15/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-15/block: (2) No such file or directory 2024-02-05T04:52:43.122 INFO:teuthology.orchestra.run.smithi196.stderr:2024-02-05T04:52:43.120+0000 7f138d1de4c0 -1 bluestore(/var/lib/ceph/osd/ceph-15/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-15/block: (2) No such file or directory 2024-02-05T04:52:43.122 INFO:teuthology.orchestra.run.smithi196.stderr:2024-02-05T04:52:43.120+0000 7f138d1de4c0 -1 bluestore(/var/lib/ceph/osd/ceph-15/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-15/block: (2) No such file or directory 2024-02-05T04:52:43.122 INFO:teuthology.orchestra.run.smithi196.stderr:2024-02-05T04:52:43.121+0000 7f138d1de4c0 -1 bluestore(/var/lib/ceph/osd/ceph-15) _read_fsid unparsable uuid 2024-02-05T04:52:45.975 DEBUG:teuthology.orchestra.run.smithi196:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-15 2024-02-05T04:52:46.007 INFO:tasks.ceph:Reading keys from all nodes... 2024-02-05T04:52:46.007 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-02-05T04:52:46.007 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-02-05T04:52:46.039 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-02-05T04:52:46.039 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-02-05T04:52:46.110 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-02-05T04:52:46.110 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-02-05T04:52:46.179 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-02-05T04:52:46.179 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/var/lib/ceph/osd/ceph-8/keyring of=/dev/stdout 2024-02-05T04:52:46.248 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-02-05T04:52:46.249 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/var/lib/ceph/osd/ceph-12/keyring of=/dev/stdout 2024-02-05T04:52:46.318 DEBUG:teuthology.orchestra.run.smithi091:> set -ex 2024-02-05T04:52:46.318 DEBUG:teuthology.orchestra.run.smithi091:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-02-05T04:52:46.351 DEBUG:teuthology.orchestra.run.smithi091:> set -ex 2024-02-05T04:52:46.351 DEBUG:teuthology.orchestra.run.smithi091:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-02-05T04:52:46.422 DEBUG:teuthology.orchestra.run.smithi091:> set -ex 2024-02-05T04:52:46.422 DEBUG:teuthology.orchestra.run.smithi091:> sudo dd if=/var/lib/ceph/osd/ceph-9/keyring of=/dev/stdout 2024-02-05T04:52:46.496 DEBUG:teuthology.orchestra.run.smithi091:> set -ex 2024-02-05T04:52:46.496 DEBUG:teuthology.orchestra.run.smithi091:> sudo dd if=/var/lib/ceph/osd/ceph-13/keyring of=/dev/stdout 2024-02-05T04:52:46.564 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-02-05T04:52:46.564 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-02-05T04:52:46.596 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-02-05T04:52:46.596 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-02-05T04:52:46.667 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-02-05T04:52:46.667 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/osd/ceph-10/keyring of=/dev/stdout 2024-02-05T04:52:46.736 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-02-05T04:52:46.736 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/osd/ceph-14/keyring of=/dev/stdout 2024-02-05T04:52:46.804 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-02-05T04:52:46.804 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-02-05T04:52:46.837 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-02-05T04:52:46.837 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-02-05T04:52:46.908 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-02-05T04:52:46.908 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-02-05T04:52:46.977 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-02-05T04:52:46.978 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/var/lib/ceph/osd/ceph-11/keyring of=/dev/stdout 2024-02-05T04:52:47.046 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-02-05T04:52:47.047 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/var/lib/ceph/osd/ceph-15/keyring of=/dev/stdout 2024-02-05T04:52:47.115 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-02-05T04:52:47.115 DEBUG:teuthology.orchestra.run.smithi196:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-02-05T04:52:47.170 INFO:tasks.ceph:Adding keys to all mons... 2024-02-05T04:52:47.171 DEBUG:teuthology.orchestra.run.smithi062:> sudo tee -a /etc/ceph/ceph.keyring 2024-02-05T04:52:47.173 DEBUG:teuthology.orchestra.run.smithi091:> sudo tee -a /etc/ceph/ceph.keyring 2024-02-05T04:52:47.176 DEBUG:teuthology.orchestra.run.smithi150:> sudo tee -a /etc/ceph/ceph.keyring 2024-02-05T04:52:47.200 INFO:teuthology.orchestra.run.smithi062.stdout:[mgr.y] 2024-02-05T04:52:47.200 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQBhacBl61jAIxAA9/YwwwZnkL9ls7D58v8ipg== 2024-02-05T04:52:47.200 INFO:teuthology.orchestra.run.smithi062.stdout:[osd.0] 2024-02-05T04:52:47.200 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQBkacBlbN54ORAAsOR8DeVEvUInRAsCiUZV1A== 2024-02-05T04:52:47.200 INFO:teuthology.orchestra.run.smithi062.stdout:[osd.4] 2024-02-05T04:52:47.200 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQBnacBlJjZzOBAAWDiaoASbxdIrF4rGNHkCMg== 2024-02-05T04:52:47.200 INFO:teuthology.orchestra.run.smithi062.stdout:[osd.8] 2024-02-05T04:52:47.201 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQBqacBl+Cf9NhAA1bfxqubcyb++PGxATgsePQ== 2024-02-05T04:52:47.201 INFO:teuthology.orchestra.run.smithi062.stdout:[osd.12] 2024-02-05T04:52:47.201 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQBtacBl23F7NRAAbEapXeth/IbXdrhx4uDyjw== 2024-02-05T04:52:47.201 INFO:teuthology.orchestra.run.smithi062.stdout:[osd.1] 2024-02-05T04:52:47.201 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQB0acBlTjJMAxAA/iEOtEDb+EQppHd8riSNPg== 2024-02-05T04:52:47.201 INFO:teuthology.orchestra.run.smithi062.stdout:[osd.5] 2024-02-05T04:52:47.201 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQB3acBlnaPQARAAuzOzpf0fcfEiSFmlGBNWiA== 2024-02-05T04:52:47.201 INFO:teuthology.orchestra.run.smithi062.stdout:[osd.9] 2024-02-05T04:52:47.201 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQB6acBl1MgKABAAcwNMkPwOcestyjXrOFW63w== 2024-02-05T04:52:47.201 INFO:teuthology.orchestra.run.smithi062.stdout:[osd.13] 2024-02-05T04:52:47.201 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQB8acBlSXz1ORAAxGaRAEwytomzzc3RK5R0YA== 2024-02-05T04:52:47.201 INFO:teuthology.orchestra.run.smithi062.stdout:[osd.2] 2024-02-05T04:52:47.201 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQCDacBlK1HUBhAAy0siVgjpogJG3eqA/A71yg== 2024-02-05T04:52:47.202 INFO:teuthology.orchestra.run.smithi062.stdout:[osd.6] 2024-02-05T04:52:47.202 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQCGacBlzE6wBRAAEaJ4TM9IihWprkoTPkKtqA== 2024-02-05T04:52:47.202 INFO:teuthology.orchestra.run.smithi062.stdout:[osd.10] 2024-02-05T04:52:47.202 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQCJacBlzLU1BBAAC3Gym0uk8gk/vGVTwnxttQ== 2024-02-05T04:52:47.202 INFO:teuthology.orchestra.run.smithi062.stdout:[osd.14] 2024-02-05T04:52:47.202 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQCMacBlagjEAhAAUgFhVR8Lv2NGl+BDzSwTww== 2024-02-05T04:52:47.202 INFO:teuthology.orchestra.run.smithi062.stdout:[mgr.x] 2024-02-05T04:52:47.202 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQBhacBlZOP6JxAAfHZ/RRNVGfG37Bx/zxsWlg== 2024-02-05T04:52:47.202 INFO:teuthology.orchestra.run.smithi062.stdout:[osd.3] 2024-02-05T04:52:47.202 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQCSacBlRxjWCxAAHPdq2PErpoUqA3YRGN+RlA== 2024-02-05T04:52:47.202 INFO:teuthology.orchestra.run.smithi062.stdout:[osd.7] 2024-02-05T04:52:47.202 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQCVacBlJp1yChAAfu1xBHzcytTTDcJVWHuPbg== 2024-02-05T04:52:47.202 INFO:teuthology.orchestra.run.smithi062.stdout:[osd.11] 2024-02-05T04:52:47.202 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQCYacBlK/u/CBAAPGYp/s/2kYpLbqyPQNCCkQ== 2024-02-05T04:52:47.202 INFO:teuthology.orchestra.run.smithi062.stdout:[osd.15] 2024-02-05T04:52:47.202 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQCbacBl1yw9BxAAzWqxZJhMEB2v5jRFJs8a+Q== 2024-02-05T04:52:47.203 INFO:teuthology.orchestra.run.smithi062.stdout:[client.0] 2024-02-05T04:52:47.203 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQBhacBlNuScKhAAOmenVR7wR85Fq1aZR0+2Bg== 2024-02-05T04:52:47.203 INFO:teuthology.orchestra.run.smithi091.stdout:[mgr.y] 2024-02-05T04:52:47.203 INFO:teuthology.orchestra.run.smithi091.stdout: key = AQBhacBl61jAIxAA9/YwwwZnkL9ls7D58v8ipg== 2024-02-05T04:52:47.203 INFO:teuthology.orchestra.run.smithi091.stdout:[osd.0] 2024-02-05T04:52:47.203 INFO:teuthology.orchestra.run.smithi091.stdout: key = AQBkacBlbN54ORAAsOR8DeVEvUInRAsCiUZV1A== 2024-02-05T04:52:47.203 INFO:teuthology.orchestra.run.smithi091.stdout:[osd.4] 2024-02-05T04:52:47.204 INFO:teuthology.orchestra.run.smithi091.stdout: key = AQBnacBlJjZzOBAAWDiaoASbxdIrF4rGNHkCMg== 2024-02-05T04:52:47.204 INFO:teuthology.orchestra.run.smithi091.stdout:[osd.8] 2024-02-05T04:52:47.204 INFO:teuthology.orchestra.run.smithi091.stdout: key = AQBqacBl+Cf9NhAA1bfxqubcyb++PGxATgsePQ== 2024-02-05T04:52:47.204 INFO:teuthology.orchestra.run.smithi091.stdout:[osd.12] 2024-02-05T04:52:47.204 INFO:teuthology.orchestra.run.smithi091.stdout: key = AQBtacBl23F7NRAAbEapXeth/IbXdrhx4uDyjw== 2024-02-05T04:52:47.204 INFO:teuthology.orchestra.run.smithi091.stdout:[osd.1] 2024-02-05T04:52:47.204 INFO:teuthology.orchestra.run.smithi091.stdout: key = AQB0acBlTjJMAxAA/iEOtEDb+EQppHd8riSNPg== 2024-02-05T04:52:47.204 INFO:teuthology.orchestra.run.smithi091.stdout:[osd.5] 2024-02-05T04:52:47.204 INFO:teuthology.orchestra.run.smithi091.stdout: key = AQB3acBlnaPQARAAuzOzpf0fcfEiSFmlGBNWiA== 2024-02-05T04:52:47.204 INFO:teuthology.orchestra.run.smithi091.stdout:[osd.9] 2024-02-05T04:52:47.204 INFO:teuthology.orchestra.run.smithi091.stdout: key = AQB6acBl1MgKABAAcwNMkPwOcestyjXrOFW63w== 2024-02-05T04:52:47.204 INFO:teuthology.orchestra.run.smithi091.stdout:[osd.13] 2024-02-05T04:52:47.204 INFO:teuthology.orchestra.run.smithi091.stdout: key = AQB8acBlSXz1ORAAxGaRAEwytomzzc3RK5R0YA== 2024-02-05T04:52:47.204 INFO:teuthology.orchestra.run.smithi091.stdout:[osd.2] 2024-02-05T04:52:47.205 INFO:teuthology.orchestra.run.smithi091.stdout: key = AQCDacBlK1HUBhAAy0siVgjpogJG3eqA/A71yg== 2024-02-05T04:52:47.205 INFO:teuthology.orchestra.run.smithi091.stdout:[osd.6] 2024-02-05T04:52:47.205 INFO:teuthology.orchestra.run.smithi091.stdout: key = AQCGacBlzE6wBRAAEaJ4TM9IihWprkoTPkKtqA== 2024-02-05T04:52:47.205 INFO:teuthology.orchestra.run.smithi091.stdout:[osd.10] 2024-02-05T04:52:47.205 INFO:teuthology.orchestra.run.smithi091.stdout: key = AQCJacBlzLU1BBAAC3Gym0uk8gk/vGVTwnxttQ== 2024-02-05T04:52:47.205 INFO:teuthology.orchestra.run.smithi091.stdout:[osd.14] 2024-02-05T04:52:47.205 INFO:teuthology.orchestra.run.smithi091.stdout: key = AQCMacBlagjEAhAAUgFhVR8Lv2NGl+BDzSwTww== 2024-02-05T04:52:47.205 INFO:teuthology.orchestra.run.smithi091.stdout:[mgr.x] 2024-02-05T04:52:47.205 INFO:teuthology.orchestra.run.smithi091.stdout: key = AQBhacBlZOP6JxAAfHZ/RRNVGfG37Bx/zxsWlg== 2024-02-05T04:52:47.205 INFO:teuthology.orchestra.run.smithi091.stdout:[osd.3] 2024-02-05T04:52:47.205 INFO:teuthology.orchestra.run.smithi091.stdout: key = AQCSacBlRxjWCxAAHPdq2PErpoUqA3YRGN+RlA== 2024-02-05T04:52:47.205 INFO:teuthology.orchestra.run.smithi091.stdout:[osd.7] 2024-02-05T04:52:47.205 INFO:teuthology.orchestra.run.smithi091.stdout: key = AQCVacBlJp1yChAAfu1xBHzcytTTDcJVWHuPbg== 2024-02-05T04:52:47.205 INFO:teuthology.orchestra.run.smithi091.stdout:[osd.11] 2024-02-05T04:52:47.205 INFO:teuthology.orchestra.run.smithi091.stdout: key = AQCYacBlK/u/CBAAPGYp/s/2kYpLbqyPQNCCkQ== 2024-02-05T04:52:47.205 INFO:teuthology.orchestra.run.smithi091.stdout:[osd.15] 2024-02-05T04:52:47.205 INFO:teuthology.orchestra.run.smithi091.stdout: key = AQCbacBl1yw9BxAAzWqxZJhMEB2v5jRFJs8a+Q== 2024-02-05T04:52:47.206 INFO:teuthology.orchestra.run.smithi091.stdout:[client.0] 2024-02-05T04:52:47.206 INFO:teuthology.orchestra.run.smithi091.stdout: key = AQBhacBlNuScKhAAOmenVR7wR85Fq1aZR0+2Bg== 2024-02-05T04:52:47.208 INFO:teuthology.orchestra.run.smithi150.stdout:[mgr.y] 2024-02-05T04:52:47.208 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQBhacBl61jAIxAA9/YwwwZnkL9ls7D58v8ipg== 2024-02-05T04:52:47.208 INFO:teuthology.orchestra.run.smithi150.stdout:[osd.0] 2024-02-05T04:52:47.208 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQBkacBlbN54ORAAsOR8DeVEvUInRAsCiUZV1A== 2024-02-05T04:52:47.208 INFO:teuthology.orchestra.run.smithi150.stdout:[osd.4] 2024-02-05T04:52:47.208 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQBnacBlJjZzOBAAWDiaoASbxdIrF4rGNHkCMg== 2024-02-05T04:52:47.208 INFO:teuthology.orchestra.run.smithi150.stdout:[osd.8] 2024-02-05T04:52:47.208 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQBqacBl+Cf9NhAA1bfxqubcyb++PGxATgsePQ== 2024-02-05T04:52:47.208 INFO:teuthology.orchestra.run.smithi150.stdout:[osd.12] 2024-02-05T04:52:47.208 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQBtacBl23F7NRAAbEapXeth/IbXdrhx4uDyjw== 2024-02-05T04:52:47.208 INFO:teuthology.orchestra.run.smithi150.stdout:[osd.1] 2024-02-05T04:52:47.209 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQB0acBlTjJMAxAA/iEOtEDb+EQppHd8riSNPg== 2024-02-05T04:52:47.209 INFO:teuthology.orchestra.run.smithi150.stdout:[osd.5] 2024-02-05T04:52:47.209 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQB3acBlnaPQARAAuzOzpf0fcfEiSFmlGBNWiA== 2024-02-05T04:52:47.209 INFO:teuthology.orchestra.run.smithi150.stdout:[osd.9] 2024-02-05T04:52:47.209 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQB6acBl1MgKABAAcwNMkPwOcestyjXrOFW63w== 2024-02-05T04:52:47.209 INFO:teuthology.orchestra.run.smithi150.stdout:[osd.13] 2024-02-05T04:52:47.209 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQB8acBlSXz1ORAAxGaRAEwytomzzc3RK5R0YA== 2024-02-05T04:52:47.209 INFO:teuthology.orchestra.run.smithi150.stdout:[osd.2] 2024-02-05T04:52:47.209 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQCDacBlK1HUBhAAy0siVgjpogJG3eqA/A71yg== 2024-02-05T04:52:47.209 INFO:teuthology.orchestra.run.smithi150.stdout:[osd.6] 2024-02-05T04:52:47.209 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQCGacBlzE6wBRAAEaJ4TM9IihWprkoTPkKtqA== 2024-02-05T04:52:47.209 INFO:teuthology.orchestra.run.smithi150.stdout:[osd.10] 2024-02-05T04:52:47.209 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQCJacBlzLU1BBAAC3Gym0uk8gk/vGVTwnxttQ== 2024-02-05T04:52:47.209 INFO:teuthology.orchestra.run.smithi150.stdout:[osd.14] 2024-02-05T04:52:47.209 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQCMacBlagjEAhAAUgFhVR8Lv2NGl+BDzSwTww== 2024-02-05T04:52:47.210 INFO:teuthology.orchestra.run.smithi150.stdout:[mgr.x] 2024-02-05T04:52:47.210 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQBhacBlZOP6JxAAfHZ/RRNVGfG37Bx/zxsWlg== 2024-02-05T04:52:47.210 INFO:teuthology.orchestra.run.smithi150.stdout:[osd.3] 2024-02-05T04:52:47.210 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQCSacBlRxjWCxAAHPdq2PErpoUqA3YRGN+RlA== 2024-02-05T04:52:47.210 INFO:teuthology.orchestra.run.smithi150.stdout:[osd.7] 2024-02-05T04:52:47.210 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQCVacBlJp1yChAAfu1xBHzcytTTDcJVWHuPbg== 2024-02-05T04:52:47.210 INFO:teuthology.orchestra.run.smithi150.stdout:[osd.11] 2024-02-05T04:52:47.210 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQCYacBlK/u/CBAAPGYp/s/2kYpLbqyPQNCCkQ== 2024-02-05T04:52:47.210 INFO:teuthology.orchestra.run.smithi150.stdout:[osd.15] 2024-02-05T04:52:47.210 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQCbacBl1yw9BxAAzWqxZJhMEB2v5jRFJs8a+Q== 2024-02-05T04:52:47.210 INFO:teuthology.orchestra.run.smithi150.stdout:[client.0] 2024-02-05T04:52:47.210 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQBhacBlNuScKhAAOmenVR7wR85Fq1aZR0+2Bg== 2024-02-05T04:52:47.211 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-02-05T04:52:47.249 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-02-05T04:52:47.252 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-02-05T04:52:47.307 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-05T04:52:47.312 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-05T04:52:47.314 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-05T04:52:47.367 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-05T04:52:47.370 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-05T04:52:47.372 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-05T04:52:47.424 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-05T04:52:47.426 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-05T04:52:47.428 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-05T04:52:47.480 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.12 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-05T04:52:47.483 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.12 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-05T04:52:47.485 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.12 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-05T04:52:47.537 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-05T04:52:47.539 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-05T04:52:47.541 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-05T04:52:47.594 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-05T04:52:47.596 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-05T04:52:47.598 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-05T04:52:47.650 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-05T04:52:47.652 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-05T04:52:47.654 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-05T04:52:47.706 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.13 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-05T04:52:47.708 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.13 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-05T04:52:47.710 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.13 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-05T04:52:47.762 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-05T04:52:47.764 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-05T04:52:47.766 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-05T04:52:47.818 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-05T04:52:47.820 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-05T04:52:47.822 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-05T04:52:47.874 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-05T04:52:47.876 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-05T04:52:47.879 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-05T04:52:47.931 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.14 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-05T04:52:47.933 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.14 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-05T04:52:47.936 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.14 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-05T04:52:47.988 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-02-05T04:52:47.992 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-02-05T04:52:47.995 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-02-05T04:52:48.046 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-05T04:52:48.049 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-05T04:52:48.051 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-05T04:52:48.103 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-05T04:52:48.105 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-05T04:52:48.107 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-05T04:52:48.159 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-05T04:52:48.162 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-05T04:52:48.165 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-05T04:52:48.218 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.15 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-05T04:52:48.220 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.15 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-05T04:52:48.222 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.15 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-05T04:52:48.274 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-02-05T04:52:48.276 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-02-05T04:52:48.278 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-02-05T04:52:48.336 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-02-05T04:52:48.337 DEBUG:teuthology.orchestra.run.smithi062:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-02-05T04:52:48.363 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-02-05T04:52:48.460 DEBUG:teuthology.orchestra.run.smithi062:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-02-05T04:52:48.486 DEBUG:teuthology.orchestra.run.smithi091:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-02-05T04:52:48.513 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-02-05T04:52:48.609 DEBUG:teuthology.orchestra.run.smithi091:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-02-05T04:52:48.636 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-02-05T04:52:48.664 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-02-05T04:52:48.761 DEBUG:teuthology.orchestra.run.smithi150:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-02-05T04:52:48.788 DEBUG:teuthology.orchestra.run.smithi062:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-02-05T04:52:48.790 DEBUG:teuthology.orchestra.run.smithi091:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-02-05T04:52:48.792 DEBUG:teuthology.orchestra.run.smithi150:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-02-05T04:52:48.843 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-02-05T04:52:48.843 INFO:tasks.ceph.mon.a:Restarting daemon 2024-02-05T04:52:48.843 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-02-05T04:52:48.846 INFO:tasks.ceph.mon.a:Started 2024-02-05T04:52:48.846 INFO:tasks.ceph.mon.b:Restarting daemon 2024-02-05T04:52:48.846 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-02-05T04:52:48.849 INFO:tasks.ceph.mon.b:Started 2024-02-05T04:52:48.849 INFO:tasks.ceph.mon.c:Restarting daemon 2024-02-05T04:52:48.849 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-02-05T04:52:48.885 INFO:tasks.ceph.mon.c:Started 2024-02-05T04:52:48.886 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-02-05T04:52:48.886 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-02-05T04:52:48.886 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-02-05T04:52:48.888 INFO:tasks.ceph.mgr.y:Started 2024-02-05T04:52:48.888 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-02-05T04:52:48.889 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-02-05T04:52:48.891 INFO:tasks.ceph.mgr.x:Started 2024-02-05T04:52:48.891 INFO:tasks.ceph:Setting crush tunables to default 2024-02-05T04:52:48.891 DEBUG:teuthology.orchestra.run.smithi062:> sudo ceph --cluster ceph osd crush tunables default 2024-02-05T04:52:55.490 INFO:teuthology.orchestra.run.smithi062.stderr:adjusted tunables profile to default 2024-02-05T04:52:55.502 INFO:tasks.ceph:check_enable_crimson: False 2024-02-05T04:52:55.502 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-02-05T04:52:55.502 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-02-05T04:52:55.502 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-02-05T04:52:55.530 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-02-05T04:52:55.530 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-02-05T04:52:55.596 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-02-05T04:52:55.596 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/var/lib/ceph/osd/ceph-8/fsid of=/dev/stdout 2024-02-05T04:52:55.663 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-02-05T04:52:55.663 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/var/lib/ceph/osd/ceph-12/fsid of=/dev/stdout 2024-02-05T04:52:55.730 DEBUG:teuthology.orchestra.run.smithi091:> set -ex 2024-02-05T04:52:55.730 DEBUG:teuthology.orchestra.run.smithi091:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-02-05T04:52:55.763 DEBUG:teuthology.orchestra.run.smithi091:> set -ex 2024-02-05T04:52:55.763 DEBUG:teuthology.orchestra.run.smithi091:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-02-05T04:52:55.833 DEBUG:teuthology.orchestra.run.smithi091:> set -ex 2024-02-05T04:52:55.833 DEBUG:teuthology.orchestra.run.smithi091:> sudo dd if=/var/lib/ceph/osd/ceph-9/fsid of=/dev/stdout 2024-02-05T04:52:55.862 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-02-05T04:52:55.861+0000 7fa605a42040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-05T04:52:55.887 INFO:tasks.ceph.mgr.y.smithi062.stderr:2024-02-05T04:52:55.886+0000 7f20b282c040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-05T04:52:55.905 DEBUG:teuthology.orchestra.run.smithi091:> set -ex 2024-02-05T04:52:55.905 DEBUG:teuthology.orchestra.run.smithi091:> sudo dd if=/var/lib/ceph/osd/ceph-13/fsid of=/dev/stdout 2024-02-05T04:52:55.915 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-02-05T04:52:55.914+0000 7fa605a42040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-05T04:52:55.942 INFO:tasks.ceph.mgr.y.smithi062.stderr:2024-02-05T04:52:55.941+0000 7f20b282c040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-05T04:52:55.973 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-02-05T04:52:55.973 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-02-05T04:52:56.005 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-02-05T04:52:56.005 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-02-05T04:52:56.075 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-02-05T04:52:56.075 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/osd/ceph-10/fsid of=/dev/stdout 2024-02-05T04:52:56.144 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-02-05T04:52:56.145 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/osd/ceph-14/fsid of=/dev/stdout 2024-02-05T04:52:56.214 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-02-05T04:52:56.215 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-02-05T04:52:56.242 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-02-05T04:52:56.242 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-02-05T04:52:56.309 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-02-05T04:52:56.309 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/var/lib/ceph/osd/ceph-11/fsid of=/dev/stdout 2024-02-05T04:52:56.377 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-02-05T04:52:56.377 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/var/lib/ceph/osd/ceph-15/fsid of=/dev/stdout 2024-02-05T04:52:56.437 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-02-05T04:52:56.436+0000 7fa605a42040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-05T04:52:56.444 DEBUG:teuthology.orchestra.run.smithi196:> sudo ceph --cluster ceph osd new cdb503c5-a32c-4a9a-b1d9-194ab5ee48aa 0 2024-02-05T04:52:56.474 INFO:tasks.ceph.mgr.y.smithi062.stderr:2024-02-05T04:52:56.474+0000 7f20b282c040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-05T04:52:56.492 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-02-05T04:52:56.491+0000 7fa605a42040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-05T04:52:56.529 INFO:tasks.ceph.mgr.y.smithi062.stderr:2024-02-05T04:52:56.528+0000 7f20b282c040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-05T04:52:56.539 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-02-05T04:52:56.538+0000 7fa605a42040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-05T04:52:56.575 INFO:tasks.ceph.mgr.y.smithi062.stderr:2024-02-05T04:52:56.574+0000 7f20b282c040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-05T04:52:56.597 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-02-05T04:52:56.596+0000 7fa605a42040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-05T04:52:56.632 INFO:tasks.ceph.mgr.y.smithi062.stderr:2024-02-05T04:52:56.630+0000 7f20b282c040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-05T04:52:56.640 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-02-05T04:52:56.640+0000 7fa605a42040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-05T04:52:56.668 INFO:teuthology.orchestra.run.smithi196.stdout:0 2024-02-05T04:52:56.674 INFO:tasks.ceph.mgr.y.smithi062.stderr:2024-02-05T04:52:56.674+0000 7f20b282c040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-05T04:52:56.679 DEBUG:teuthology.orchestra.run.smithi196:> sudo ceph --cluster ceph osd new d7a3d394-74b2-4cb1-a754-4fe77e605ea9 1 2024-02-05T04:52:56.747 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-02-05T04:52:56.745+0000 7fa605a42040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-05T04:52:56.780 INFO:tasks.ceph.mgr.y.smithi062.stderr:2024-02-05T04:52:56.779+0000 7f20b282c040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-05T04:52:56.820 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-02-05T04:52:56.819+0000 7fa605a42040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-05T04:52:56.852 INFO:tasks.ceph.mgr.y.smithi062.stderr:2024-02-05T04:52:56.852+0000 7f20b282c040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-05T04:52:56.868 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-02-05T04:52:56.867+0000 7fa605a42040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-05T04:52:56.900 INFO:tasks.ceph.mgr.y.smithi062.stderr:2024-02-05T04:52:56.899+0000 7f20b282c040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-05T04:52:56.902 INFO:teuthology.orchestra.run.smithi196.stdout:1 2024-02-05T04:52:56.914 DEBUG:teuthology.orchestra.run.smithi196:> sudo ceph --cluster ceph osd new d6e99efe-475f-41c1-88a3-87ce6b26ce75 2 2024-02-05T04:52:56.923 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-02-05T04:52:56.921+0000 7fa605a42040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-05T04:52:56.955 INFO:tasks.ceph.mgr.y.smithi062.stderr:2024-02-05T04:52:56.953+0000 7f20b282c040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-05T04:52:56.972 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-02-05T04:52:56.971+0000 7fa605a42040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-05T04:52:57.002 INFO:tasks.ceph.mgr.y.smithi062.stderr:2024-02-05T04:52:57.002+0000 7f20b282c040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-05T04:52:57.104 INFO:teuthology.orchestra.run.smithi196.stdout:2 2024-02-05T04:52:57.116 DEBUG:teuthology.orchestra.run.smithi196:> sudo ceph --cluster ceph osd new 3c964090-5e07-40bd-ac7c-b66e29eb25b4 3 2024-02-05T04:52:57.296 INFO:teuthology.orchestra.run.smithi196.stdout:3 2024-02-05T04:52:57.308 DEBUG:teuthology.orchestra.run.smithi196:> sudo ceph --cluster ceph osd new 22bd25f9-3d93-4bcb-bcfc-71feae1097f0 4 2024-02-05T04:52:57.368 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-02-05T04:52:57.367+0000 7fa605a42040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-05T04:52:57.399 INFO:tasks.ceph.mgr.y.smithi062.stderr:2024-02-05T04:52:57.398+0000 7f20b282c040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-05T04:52:57.494 INFO:teuthology.orchestra.run.smithi196.stdout:4 2024-02-05T04:52:57.505 DEBUG:teuthology.orchestra.run.smithi196:> sudo ceph --cluster ceph osd new c6ae2bd4-4a4f-4f23-8991-743ffa42ed3c 5 2024-02-05T04:52:57.509 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-02-05T04:52:57.508+0000 7fa605a42040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-05T04:52:57.540 INFO:tasks.ceph.mgr.y.smithi062.stderr:2024-02-05T04:52:57.539+0000 7f20b282c040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-05T04:52:57.629 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-02-05T04:52:57.628+0000 7fa605a42040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-05T04:52:57.655 INFO:tasks.ceph.mgr.y.smithi062.stderr:2024-02-05T04:52:57.654+0000 7f20b282c040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-05T04:52:57.676 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-02-05T04:52:57.675+0000 7fa605a42040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-05T04:52:57.687 INFO:teuthology.orchestra.run.smithi196.stdout:5 2024-02-05T04:52:57.698 DEBUG:teuthology.orchestra.run.smithi196:> sudo ceph --cluster ceph osd new 49da8b05-ffa3-4172-983e-f081d3eb9ae6 6 2024-02-05T04:52:57.702 INFO:tasks.ceph.mgr.y.smithi062.stderr:2024-02-05T04:52:57.701+0000 7f20b282c040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-05T04:52:57.779 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-02-05T04:52:57.778+0000 7fa605a42040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-05T04:52:57.803 INFO:tasks.ceph.mgr.y.smithi062.stderr:2024-02-05T04:52:57.802+0000 7f20b282c040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-05T04:52:57.879 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-02-05T04:52:57.878+0000 7fa605a42040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-05T04:52:57.901 INFO:tasks.ceph.mgr.y.smithi062.stderr:2024-02-05T04:52:57.900+0000 7f20b282c040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-05T04:52:57.920 INFO:teuthology.orchestra.run.smithi196.stdout:6 2024-02-05T04:52:57.931 DEBUG:teuthology.orchestra.run.smithi196:> sudo ceph --cluster ceph osd new 4ece26e1-2c63-4223-baaf-882cd8261ade 7 2024-02-05T04:52:58.027 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-02-05T04:52:58.024+0000 7fa605a42040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-05T04:52:58.047 INFO:tasks.ceph.mgr.y.smithi062.stderr:2024-02-05T04:52:58.046+0000 7f20b282c040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-05T04:52:58.070 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-02-05T04:52:58.069+0000 7fa605a42040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-05T04:52:58.091 INFO:tasks.ceph.mgr.y.smithi062.stderr:2024-02-05T04:52:58.090+0000 7f20b282c040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-05T04:52:58.152 INFO:teuthology.orchestra.run.smithi196.stdout:7 2024-02-05T04:52:58.165 DEBUG:teuthology.orchestra.run.smithi196:> sudo ceph --cluster ceph osd new 9ac666e9-291e-4696-a6bb-e10c9deef3f5 8 2024-02-05T04:52:58.217 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-02-05T04:52:58.216+0000 7fa605a42040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-05T04:52:58.237 INFO:tasks.ceph.mgr.y.smithi062.stderr:2024-02-05T04:52:58.236+0000 7f20b282c040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-05T04:52:58.348 INFO:teuthology.orchestra.run.smithi196.stdout:8 2024-02-05T04:52:58.360 DEBUG:teuthology.orchestra.run.smithi196:> sudo ceph --cluster ceph osd new 7fe04cb8-ac67-46ac-b2de-1f8f60a50373 9 2024-02-05T04:52:58.542 INFO:teuthology.orchestra.run.smithi196.stdout:9 2024-02-05T04:52:58.554 DEBUG:teuthology.orchestra.run.smithi196:> sudo ceph --cluster ceph osd new a6b0ac25-da07-4fee-b646-a4530ce5b343 10 2024-02-05T04:52:58.737 INFO:teuthology.orchestra.run.smithi196.stdout:10 2024-02-05T04:52:58.748 DEBUG:teuthology.orchestra.run.smithi196:> sudo ceph --cluster ceph osd new 31b22258-77da-495c-b518-9ce0297b9165 11 2024-02-05T04:52:58.932 INFO:teuthology.orchestra.run.smithi196.stdout:11 2024-02-05T04:52:58.943 DEBUG:teuthology.orchestra.run.smithi196:> sudo ceph --cluster ceph osd new 322917eb-26ba-4439-8300-4d723bada623 12 2024-02-05T04:52:59.039 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-02-05T04:52:59.037+0000 7fa605a42040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-05T04:52:59.068 INFO:tasks.ceph.mgr.y.smithi062.stderr:2024-02-05T04:52:59.067+0000 7f20b282c040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-05T04:52:59.125 INFO:teuthology.orchestra.run.smithi196.stdout:12 2024-02-05T04:52:59.136 DEBUG:teuthology.orchestra.run.smithi196:> sudo ceph --cluster ceph osd new 9c708044-4726-42b6-9c13-03963f956d2d 13 2024-02-05T04:52:59.318 INFO:teuthology.orchestra.run.smithi196.stdout:13 2024-02-05T04:52:59.322 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-02-05T04:52:59.321+0000 7fa605a42040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-05T04:52:59.329 DEBUG:teuthology.orchestra.run.smithi196:> sudo ceph --cluster ceph osd new 0208144a-db9a-404c-b31e-13bc2e276b09 14 2024-02-05T04:52:59.351 INFO:tasks.ceph.mgr.y.smithi062.stderr:2024-02-05T04:52:59.350+0000 7f20b282c040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-05T04:52:59.377 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-02-05T04:52:59.376+0000 7fa605a42040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-05T04:52:59.403 INFO:tasks.ceph.mgr.y.smithi062.stderr:2024-02-05T04:52:59.403+0000 7f20b282c040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-05T04:52:59.483 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-02-05T04:52:59.482+0000 7fa605a42040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-05T04:52:59.510 INFO:tasks.ceph.mgr.y.smithi062.stderr:2024-02-05T04:52:59.509+0000 7f20b282c040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-05T04:52:59.510 INFO:teuthology.orchestra.run.smithi196.stdout:14 2024-02-05T04:52:59.522 DEBUG:teuthology.orchestra.run.smithi196:> sudo ceph --cluster ceph osd new 0591e7e1-dc07-4d80-b574-9b8348635764 15 2024-02-05T04:52:59.697 INFO:teuthology.orchestra.run.smithi196.stdout:15 2024-02-05T04:52:59.712 INFO:tasks.ceph.osd.0:Restarting daemon 2024-02-05T04:52:59.712 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-02-05T04:52:59.715 INFO:tasks.ceph.osd.0:Started 2024-02-05T04:52:59.715 INFO:tasks.ceph.osd.4:Restarting daemon 2024-02-05T04:52:59.715 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-02-05T04:52:59.717 INFO:tasks.ceph.osd.4:Started 2024-02-05T04:52:59.717 INFO:tasks.ceph.osd.8:Restarting daemon 2024-02-05T04:52:59.718 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 8 2024-02-05T04:52:59.720 INFO:tasks.ceph.osd.8:Started 2024-02-05T04:52:59.720 INFO:tasks.ceph.osd.12:Restarting daemon 2024-02-05T04:52:59.720 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 12 2024-02-05T04:52:59.722 INFO:tasks.ceph.osd.12:Started 2024-02-05T04:52:59.722 INFO:tasks.ceph.osd.1:Restarting daemon 2024-02-05T04:52:59.722 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-02-05T04:52:59.725 INFO:tasks.ceph.osd.1:Started 2024-02-05T04:52:59.726 INFO:tasks.ceph.osd.5:Restarting daemon 2024-02-05T04:52:59.726 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-02-05T04:52:59.728 INFO:tasks.ceph.osd.5:Started 2024-02-05T04:52:59.728 INFO:tasks.ceph.osd.9:Restarting daemon 2024-02-05T04:52:59.728 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 9 2024-02-05T04:52:59.730 INFO:tasks.ceph.osd.9:Started 2024-02-05T04:52:59.730 INFO:tasks.ceph.osd.13:Restarting daemon 2024-02-05T04:52:59.730 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 13 2024-02-05T04:52:59.733 INFO:tasks.ceph.osd.13:Started 2024-02-05T04:52:59.733 INFO:tasks.ceph.osd.2:Restarting daemon 2024-02-05T04:52:59.733 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-02-05T04:52:59.736 INFO:tasks.ceph.osd.2:Started 2024-02-05T04:52:59.736 INFO:tasks.ceph.osd.6:Restarting daemon 2024-02-05T04:52:59.736 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-02-05T04:52:59.738 INFO:tasks.ceph.osd.6:Started 2024-02-05T04:52:59.738 INFO:tasks.ceph.osd.10:Restarting daemon 2024-02-05T04:52:59.738 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 10 2024-02-05T04:52:59.740 INFO:tasks.ceph.osd.10:Started 2024-02-05T04:52:59.740 INFO:tasks.ceph.osd.14:Restarting daemon 2024-02-05T04:52:59.740 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 14 2024-02-05T04:52:59.743 INFO:tasks.ceph.osd.14:Started 2024-02-05T04:52:59.743 INFO:tasks.ceph.osd.3:Restarting daemon 2024-02-05T04:52:59.743 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-02-05T04:52:59.745 INFO:tasks.ceph.osd.3:Started 2024-02-05T04:52:59.745 INFO:tasks.ceph.osd.7:Restarting daemon 2024-02-05T04:52:59.745 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-02-05T04:52:59.747 INFO:tasks.ceph.osd.7:Started 2024-02-05T04:52:59.747 INFO:tasks.ceph.osd.11:Restarting daemon 2024-02-05T04:52:59.748 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 11 2024-02-05T04:52:59.750 INFO:tasks.ceph.osd.11:Started 2024-02-05T04:52:59.750 INFO:tasks.ceph.osd.15:Restarting daemon 2024-02-05T04:52:59.750 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 15 2024-02-05T04:52:59.752 INFO:tasks.ceph.osd.15:Started 2024-02-05T04:52:59.752 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-05T04:52:59.935 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:52:59.935 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":18,"fsid":"88c11a2d-382e-4f21-b8af-45ab9c504b93","created":"2024-02-05T04:52:55.182961+0000","modified":"2024-02-05T04:52:59.695913+0000","last_up_change":"0.000000","last_in_change":"2024-02-05T04:52:59.695913+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cdb503c5-a32c-4a9a-b1d9-194ab5ee48aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d7a3d394-74b2-4cb1-a754-4fe77e605ea9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d6e99efe-475f-41c1-88a3-87ce6b26ce75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3c964090-5e07-40bd-ac7c-b66e29eb25b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"22bd25f9-3d93-4bcb-bcfc-71feae1097f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c6ae2bd4-4a4f-4f23-8991-743ffa42ed3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"49da8b05-ffa3-4172-983e-f081d3eb9ae6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4ece26e1-2c63-4223-baaf-882cd8261ade","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"9ac666e9-291e-4696-a6bb-e10c9deef3f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"7fe04cb8-ac67-46ac-b2de-1f8f60a50373","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"a6b0ac25-da07-4fee-b646-a4530ce5b343","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"31b22258-77da-495c-b518-9ce0297b9165","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"322917eb-26ba-4439-8300-4d723bada623","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"9c708044-4726-42b6-9c13-03963f956d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"0208144a-db9a-404c-b31e-13bc2e276b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"0591e7e1-dc07-4d80-b574-9b8348635764","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-05T04:52:59.945 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-02-05T04:52:59.945 INFO:tasks.ceph:Waiting for OSDs to come up 2024-02-05T04:53:00.321 INFO:tasks.ceph.osd.4.smithi062.stderr:2024-02-05T04:53:00.320+0000 7f70902234c0 -1 Falling back to public interface 2024-02-05T04:53:00.322 INFO:tasks.ceph.osd.8.smithi062.stderr:2024-02-05T04:53:00.321+0000 7f9389a134c0 -1 Falling back to public interface 2024-02-05T04:53:00.323 INFO:tasks.ceph.osd.0.smithi062.stderr:2024-02-05T04:53:00.322+0000 7fecec6614c0 -1 Falling back to public interface 2024-02-05T04:53:00.327 INFO:tasks.ceph.osd.12.smithi062.stderr:2024-02-05T04:53:00.326+0000 7f5e5bcda4c0 -1 Falling back to public interface 2024-02-05T04:53:00.335 INFO:tasks.ceph.osd.5.smithi091.stderr:2024-02-05T04:53:00.333+0000 7f639bc144c0 -1 Falling back to public interface 2024-02-05T04:53:00.338 INFO:tasks.ceph.osd.3.smithi196.stderr:2024-02-05T04:53:00.337+0000 7fc8856164c0 -1 Falling back to public interface 2024-02-05T04:53:00.341 INFO:tasks.ceph.osd.7.smithi196.stderr:2024-02-05T04:53:00.340+0000 7fb28521b4c0 -1 Falling back to public interface 2024-02-05T04:53:00.342 INFO:tasks.ceph.osd.11.smithi196.stderr:2024-02-05T04:53:00.341+0000 7f85ac5e24c0 -1 Falling back to public interface 2024-02-05T04:53:00.343 INFO:tasks.ceph.osd.1.smithi091.stderr:2024-02-05T04:53:00.342+0000 7ff9e00144c0 -1 Falling back to public interface 2024-02-05T04:53:00.344 INFO:tasks.ceph.osd.9.smithi091.stderr:2024-02-05T04:53:00.342+0000 7efc1c6234c0 -1 Falling back to public interface 2024-02-05T04:53:00.344 INFO:tasks.ceph.osd.13.smithi091.stderr:2024-02-05T04:53:00.342+0000 7f52735ab4c0 -1 Falling back to public interface 2024-02-05T04:53:00.345 INFO:tasks.ceph.osd.9.smithi091.stderr:2024-02-05T04:53:00.342+0000 7efc1b02c640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6809/0: (98) Address already in use 2024-02-05T04:53:00.345 INFO:tasks.ceph.osd.1.smithi091.stderr:2024-02-05T04:53:00.343+0000 7ff9dea88640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6817/0: (98) Address already in use 2024-02-05T04:53:00.346 INFO:tasks.ceph.osd.9.smithi091.stderr:2024-02-05T04:53:00.343+0000 7efc1b02c640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6819/0: (98) Address already in use 2024-02-05T04:53:00.347 INFO:tasks.ceph.osd.15.smithi196.stderr:2024-02-05T04:53:00.345+0000 7f1fb04a34c0 -1 Falling back to public interface 2024-02-05T04:53:00.347 INFO:tasks.ceph.osd.6.smithi150.stderr:2024-02-05T04:53:00.344+0000 7f6bee80d4c0 -1 Falling back to public interface 2024-02-05T04:53:00.349 INFO:tasks.ceph.osd.10.smithi150.stderr:2024-02-05T04:53:00.347+0000 7f30792ab4c0 -1 Falling back to public interface 2024-02-05T04:53:00.349 INFO:tasks.ceph.osd.14.smithi150.stderr:2024-02-05T04:53:00.348+0000 7f5f84e124c0 -1 Falling back to public interface 2024-02-05T04:53:00.352 INFO:tasks.ceph.osd.2.smithi150.stderr:2024-02-05T04:53:00.351+0000 7f733ba154c0 -1 Falling back to public interface 2024-02-05T04:53:00.353 INFO:tasks.ceph.osd.14.smithi150.stderr:2024-02-05T04:53:00.352+0000 7f5f83888640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6823/0: (98) Address already in use 2024-02-05T04:53:00.521 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-02-05T04:53:00.520+0000 7fa5a6e95640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-02-05T04:53:00.521 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-02-05T04:53:00.520+0000 7fa5a6e95640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-02-05T04:53:01.557 DEBUG:teuthology.orchestra.run.smithi062:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-02-05T04:53:01.853 INFO:teuthology.misc.health.smithi062.stdout: 2024-02-05T04:53:01.853 INFO:teuthology.misc.health.smithi062.stdout:{"epoch":18,"fsid":"88c11a2d-382e-4f21-b8af-45ab9c504b93","created":"2024-02-05T04:52:55.182961+0000","modified":"2024-02-05T04:52:59.695913+0000","last_up_change":"0.000000","last_in_change":"2024-02-05T04:52:59.695913+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cdb503c5-a32c-4a9a-b1d9-194ab5ee48aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d7a3d394-74b2-4cb1-a754-4fe77e605ea9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d6e99efe-475f-41c1-88a3-87ce6b26ce75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3c964090-5e07-40bd-ac7c-b66e29eb25b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"22bd25f9-3d93-4bcb-bcfc-71feae1097f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c6ae2bd4-4a4f-4f23-8991-743ffa42ed3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"49da8b05-ffa3-4172-983e-f081d3eb9ae6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4ece26e1-2c63-4223-baaf-882cd8261ade","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"9ac666e9-291e-4696-a6bb-e10c9deef3f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"7fe04cb8-ac67-46ac-b2de-1f8f60a50373","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"a6b0ac25-da07-4fee-b646-a4530ce5b343","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"31b22258-77da-495c-b518-9ce0297b9165","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"322917eb-26ba-4439-8300-4d723bada623","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"9c708044-4726-42b6-9c13-03963f956d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"0208144a-db9a-404c-b31e-13bc2e276b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"0591e7e1-dc07-4d80-b574-9b8348635764","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-05T04:53:01.864 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-02-05T04:53:03.755 INFO:tasks.ceph.osd.4.smithi062.stderr:2024-02-05T04:53:03.754+0000 7f70902234c0 -1 osd.4 0 log_to_monitors true 2024-02-05T04:53:03.758 INFO:tasks.ceph.osd.5.smithi091.stderr:2024-02-05T04:53:03.756+0000 7f639bc144c0 -1 osd.5 0 log_to_monitors true 2024-02-05T04:53:03.772 INFO:tasks.ceph.osd.3.smithi196.stderr:2024-02-05T04:53:03.771+0000 7fc8856164c0 -1 osd.3 0 log_to_monitors true 2024-02-05T04:53:03.797 INFO:tasks.ceph.osd.6.smithi150.stderr:2024-02-05T04:53:03.795+0000 7f6bee80d4c0 -1 osd.6 0 log_to_monitors true 2024-02-05T04:53:03.811 INFO:tasks.ceph.osd.13.smithi091.stderr:2024-02-05T04:53:03.810+0000 7f52735ab4c0 -1 osd.13 0 log_to_monitors true 2024-02-05T04:53:03.812 INFO:tasks.ceph.osd.0.smithi062.stderr:2024-02-05T04:53:03.811+0000 7fecec6614c0 -1 osd.0 0 log_to_monitors true 2024-02-05T04:53:03.819 INFO:tasks.ceph.osd.8.smithi062.stderr:2024-02-05T04:53:03.817+0000 7f9389a134c0 -1 osd.8 0 log_to_monitors true 2024-02-05T04:53:03.834 INFO:tasks.ceph.osd.1.smithi091.stderr:2024-02-05T04:53:03.833+0000 7ff9e00144c0 -1 osd.1 0 log_to_monitors true 2024-02-05T04:53:03.840 INFO:tasks.ceph.osd.12.smithi062.stderr:2024-02-05T04:53:03.839+0000 7f5e5bcda4c0 -1 osd.12 0 log_to_monitors true 2024-02-05T04:53:03.853 INFO:tasks.ceph.osd.11.smithi196.stderr:2024-02-05T04:53:03.852+0000 7f85ac5e24c0 -1 osd.11 0 log_to_monitors true 2024-02-05T04:53:03.854 INFO:tasks.ceph.osd.15.smithi196.stderr:2024-02-05T04:53:03.853+0000 7f1fb04a34c0 -1 osd.15 0 log_to_monitors true 2024-02-05T04:53:03.859 INFO:tasks.ceph.osd.7.smithi196.stderr:2024-02-05T04:53:03.858+0000 7fb28521b4c0 -1 osd.7 0 log_to_monitors true 2024-02-05T04:53:03.860 INFO:tasks.ceph.osd.14.smithi150.stderr:2024-02-05T04:53:03.858+0000 7f5f84e124c0 -1 osd.14 0 log_to_monitors true 2024-02-05T04:53:03.860 INFO:tasks.ceph.osd.10.smithi150.stderr:2024-02-05T04:53:03.859+0000 7f30792ab4c0 -1 osd.10 0 log_to_monitors true 2024-02-05T04:53:03.870 INFO:tasks.ceph.osd.9.smithi091.stderr:2024-02-05T04:53:03.869+0000 7efc1c6234c0 -1 osd.9 0 log_to_monitors true 2024-02-05T04:53:03.874 INFO:tasks.ceph.osd.2.smithi150.stderr:2024-02-05T04:53:03.873+0000 7f733ba154c0 -1 osd.2 0 log_to_monitors true 2024-02-05T04:53:04.823 INFO:tasks.ceph.osd.4.smithi062.stderr:2024-02-05T04:53:04.822+0000 7f708b2d1640 -1 osd.4 0 waiting for initial osdmap 2024-02-05T04:53:04.834 INFO:tasks.ceph.osd.4.smithi062.stderr:2024-02-05T04:53:04.833+0000 7f7087912640 -1 osd.4 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-05T04:53:05.603 INFO:tasks.ceph.osd.5.smithi091.stderr:2024-02-05T04:53:05.601+0000 7f639737f640 -1 osd.5 0 waiting for initial osdmap 2024-02-05T04:53:05.612 INFO:tasks.ceph.osd.6.smithi150.stderr:2024-02-05T04:53:05.611+0000 7f6be99b9640 -1 osd.6 0 waiting for initial osdmap 2024-02-05T04:53:05.616 INFO:tasks.ceph.osd.5.smithi091.stderr:2024-02-05T04:53:05.615+0000 7f638f9b8640 -1 osd.5 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-05T04:53:05.627 INFO:tasks.ceph.osd.6.smithi150.stderr:2024-02-05T04:53:05.626+0000 7f6be1ff2640 -1 osd.6 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-05T04:53:05.627 INFO:tasks.ceph.osd.3.smithi196.stderr:2024-02-05T04:53:05.626+0000 7fc88078d640 -1 osd.3 0 waiting for initial osdmap 2024-02-05T04:53:05.637 INFO:tasks.ceph.osd.3.smithi196.stderr:2024-02-05T04:53:05.636+0000 7fc878dc6640 -1 osd.3 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-05T04:53:05.639 INFO:tasks.ceph.osd.3.smithi196.stderr:2024-02-05T04:53:05.638+0000 7fc8739f4640 -1 osd.3 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-05T04:53:05.751 INFO:tasks.ceph.osd.6.smithi150.stderr:2024-02-05T04:53:05.750+0000 7f6bdcc20640 -1 osd.6 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-05T04:53:05.798 INFO:tasks.ceph.osd.5.smithi091.stderr:2024-02-05T04:53:05.797+0000 7f638a5e6640 -1 osd.5 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-05T04:53:05.835 INFO:tasks.ceph.osd.15.smithi196.stderr:2024-02-05T04:53:05.834+0000 7f1fabc0e640 -1 osd.15 0 waiting for initial osdmap 2024-02-05T04:53:05.836 INFO:tasks.ceph.osd.14.smithi150.stderr:2024-02-05T04:53:05.835+0000 7f5f8004c640 -1 osd.14 0 waiting for initial osdmap 2024-02-05T04:53:05.838 INFO:tasks.ceph.osd.9.smithi091.stderr:2024-02-05T04:53:05.837+0000 7efc177f0640 -1 osd.9 0 waiting for initial osdmap 2024-02-05T04:53:05.847 INFO:tasks.ceph.osd.1.smithi091.stderr:2024-02-05T04:53:05.846+0000 7ff9db24c640 -1 osd.1 0 waiting for initial osdmap 2024-02-05T04:53:05.863 INFO:tasks.ceph.osd.1.smithi091.stderr:2024-02-05T04:53:05.862+0000 7ff9d788d640 -1 osd.1 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-05T04:53:05.864 INFO:tasks.ceph.osd.9.smithi091.stderr:2024-02-05T04:53:05.862+0000 7efc13e31640 -1 osd.9 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-05T04:53:05.866 INFO:tasks.ceph.osd.15.smithi196.stderr:2024-02-05T04:53:05.865+0000 7f1fa824f640 -1 osd.15 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-05T04:53:05.930 INFO:tasks.ceph.osd.14.smithi150.stderr:2024-02-05T04:53:05.929+0000 7f5f7c68d640 -1 osd.14 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-05T04:53:06.668 INFO:tasks.ceph.osd.13.smithi091.stderr:2024-02-05T04:53:06.666+0000 7f526ed16640 -1 osd.13 0 waiting for initial osdmap 2024-02-05T04:53:06.679 INFO:tasks.ceph.osd.13.smithi091.stderr:2024-02-05T04:53:06.678+0000 7f526734f640 -1 osd.13 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-05T04:53:07.039 INFO:tasks.ceph.osd.2.smithi150.stderr:2024-02-05T04:53:07.038+0000 7f7336bc2640 -1 osd.2 0 waiting for initial osdmap 2024-02-05T04:53:07.042 INFO:tasks.ceph.osd.11.smithi196.stderr:2024-02-05T04:53:07.041+0000 7f85a7d4f640 -1 osd.11 0 waiting for initial osdmap 2024-02-05T04:53:07.047 INFO:tasks.ceph.osd.10.smithi150.stderr:2024-02-05T04:53:07.046+0000 7f3074a16640 -1 osd.10 0 waiting for initial osdmap 2024-02-05T04:53:07.054 INFO:tasks.ceph.osd.2.smithi150.stderr:2024-02-05T04:53:07.053+0000 7f732f1fb640 -1 osd.2 22 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-05T04:53:07.056 INFO:tasks.ceph.osd.11.smithi196.stderr:2024-02-05T04:53:07.055+0000 7f85a0388640 -1 osd.11 22 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-05T04:53:07.061 INFO:tasks.ceph.osd.10.smithi150.stderr:2024-02-05T04:53:07.060+0000 7f306d04f640 -1 osd.10 22 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-05T04:53:07.062 INFO:tasks.ceph.osd.7.smithi196.stderr:2024-02-05T04:53:07.061+0000 7fb2803e2640 -1 osd.7 0 waiting for initial osdmap 2024-02-05T04:53:07.078 INFO:tasks.ceph.osd.7.smithi196.stderr:2024-02-05T04:53:07.077+0000 7fb278a1b640 -1 osd.7 22 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-05T04:53:07.551 INFO:tasks.ceph.osd.12.smithi062.stderr:2024-02-05T04:53:07.549+0000 7f5e57447640 -1 osd.12 0 waiting for initial osdmap 2024-02-05T04:53:07.552 INFO:tasks.ceph.osd.8.smithi062.stderr:2024-02-05T04:53:07.552+0000 7f9384bbb640 -1 osd.8 0 waiting for initial osdmap 2024-02-05T04:53:07.560 INFO:tasks.ceph.osd.0.smithi062.stderr:2024-02-05T04:53:07.560+0000 7fece7dcc640 -1 osd.0 0 waiting for initial osdmap 2024-02-05T04:53:07.565 INFO:tasks.ceph.osd.12.smithi062.stderr:2024-02-05T04:53:07.564+0000 7f5e4fa80640 -1 osd.12 22 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-05T04:53:07.573 INFO:tasks.ceph.osd.0.smithi062.stderr:2024-02-05T04:53:07.572+0000 7fece0405640 -1 osd.0 22 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-05T04:53:07.573 INFO:tasks.ceph.osd.8.smithi062.stderr:2024-02-05T04:53:07.572+0000 7f937d1f4640 -1 osd.8 22 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-05T04:53:09.473 DEBUG:teuthology.orchestra.run.smithi062:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-02-05T04:53:09.792 INFO:teuthology.misc.health.smithi062.stdout: 2024-02-05T04:53:09.792 INFO:teuthology.misc.health.smithi062.stdout:{"epoch":24,"fsid":"88c11a2d-382e-4f21-b8af-45ab9c504b93","created":"2024-02-05T04:52:55.182961+0000","modified":"2024-02-05T04:53:08.909926+0000","last_up_change":"2024-02-05T04:53:07.900741+0000","last_in_change":"2024-02-05T04:52:59.695913+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-05T04:53:07.549923+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cdb503c5-a32c-4a9a-b1d9-194ab5ee48aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":1189094240},{"type":"v1","addr":"172.21.15.62:6816","nonce":1189094240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":1189094240},{"type":"v1","addr":"172.21.15.62:6819","nonce":1189094240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":1189094240},{"type":"v1","addr":"172.21.15.62:6823","nonce":1189094240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":1189094240},{"type":"v1","addr":"172.21.15.62:6821","nonce":1189094240}]},"public_addr":"172.21.15.62:6816/1189094240","cluster_addr":"172.21.15.62:6819/1189094240","heartbeat_back_addr":"172.21.15.62:6823/1189094240","heartbeat_front_addr":"172.21.15.62:6821/1189094240","state":["exists","up"]},{"osd":1,"uuid":"d7a3d394-74b2-4cb1-a754-4fe77e605ea9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6808","nonce":2878185190},{"type":"v1","addr":"172.21.15.91:6811","nonce":2878185190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6814","nonce":2878185190},{"type":"v1","addr":"172.21.15.91:6818","nonce":2878185190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6828","nonce":2878185190},{"type":"v1","addr":"172.21.15.91:6830","nonce":2878185190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6821","nonce":2878185190},{"type":"v1","addr":"172.21.15.91:6824","nonce":2878185190}]},"public_addr":"172.21.15.91:6811/2878185190","cluster_addr":"172.21.15.91:6818/2878185190","heartbeat_back_addr":"172.21.15.91:6830/2878185190","heartbeat_front_addr":"172.21.15.91:6824/2878185190","state":["exists","up"]},{"osd":2,"uuid":"d6e99efe-475f-41c1-88a3-87ce6b26ce75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":40232889},{"type":"v1","addr":"172.21.15.150:6825","nonce":40232889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":40232889},{"type":"v1","addr":"172.21.15.150:6827","nonce":40232889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":40232889},{"type":"v1","addr":"172.21.15.150:6831","nonce":40232889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":40232889},{"type":"v1","addr":"172.21.15.150:6829","nonce":40232889}]},"public_addr":"172.21.15.150:6825/40232889","cluster_addr":"172.21.15.150:6827/40232889","heartbeat_back_addr":"172.21.15.150:6831/40232889","heartbeat_front_addr":"172.21.15.150:6829/40232889","state":["exists","up"]},{"osd":3,"uuid":"3c964090-5e07-40bd-ac7c-b66e29eb25b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":3834523862},{"type":"v1","addr":"172.21.15.196:6803","nonce":3834523862}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":3834523862},{"type":"v1","addr":"172.21.15.196:6805","nonce":3834523862}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":3834523862},{"type":"v1","addr":"172.21.15.196:6809","nonce":3834523862}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":3834523862},{"type":"v1","addr":"172.21.15.196:6807","nonce":3834523862}]},"public_addr":"172.21.15.196:6803/3834523862","cluster_addr":"172.21.15.196:6805/3834523862","heartbeat_back_addr":"172.21.15.196:6809/3834523862","heartbeat_front_addr":"172.21.15.196:6807/3834523862","state":["exists","up"]},{"osd":4,"uuid":"22bd25f9-3d93-4bcb-bcfc-71feae1097f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":2244900897},{"type":"v1","addr":"172.21.15.62:6801","nonce":2244900897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":2244900897},{"type":"v1","addr":"172.21.15.62:6803","nonce":2244900897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":2244900897},{"type":"v1","addr":"172.21.15.62:6807","nonce":2244900897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":2244900897},{"type":"v1","addr":"172.21.15.62:6805","nonce":2244900897}]},"public_addr":"172.21.15.62:6801/2244900897","cluster_addr":"172.21.15.62:6803/2244900897","heartbeat_back_addr":"172.21.15.62:6807/2244900897","heartbeat_front_addr":"172.21.15.62:6805/2244900897","state":["exists","up"]},{"osd":5,"uuid":"c6ae2bd4-4a4f-4f23-8991-743ffa42ed3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6800","nonce":473596866},{"type":"v1","addr":"172.21.15.91:6801","nonce":473596866}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6802","nonce":473596866},{"type":"v1","addr":"172.21.15.91:6803","nonce":473596866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6806","nonce":473596866},{"type":"v1","addr":"172.21.15.91:6807","nonce":473596866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6804","nonce":473596866},{"type":"v1","addr":"172.21.15.91:6805","nonce":473596866}]},"public_addr":"172.21.15.91:6801/473596866","cluster_addr":"172.21.15.91:6803/473596866","heartbeat_back_addr":"172.21.15.91:6807/473596866","heartbeat_front_addr":"172.21.15.91:6805/473596866","state":["exists","up"]},{"osd":6,"uuid":"49da8b05-ffa3-4172-983e-f081d3eb9ae6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":2199648645},{"type":"v1","addr":"172.21.15.150:6801","nonce":2199648645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":2199648645},{"type":"v1","addr":"172.21.15.150:6803","nonce":2199648645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":2199648645},{"type":"v1","addr":"172.21.15.150:6807","nonce":2199648645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":2199648645},{"type":"v1","addr":"172.21.15.150:6805","nonce":2199648645}]},"public_addr":"172.21.15.150:6801/2199648645","cluster_addr":"172.21.15.150:6803/2199648645","heartbeat_back_addr":"172.21.15.150:6807/2199648645","heartbeat_front_addr":"172.21.15.150:6805/2199648645","state":["exists","up"]},{"osd":7,"uuid":"4ece26e1-2c63-4223-baaf-882cd8261ade","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":254726044},{"type":"v1","addr":"172.21.15.196:6811","nonce":254726044}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":254726044},{"type":"v1","addr":"172.21.15.196:6813","nonce":254726044}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":254726044},{"type":"v1","addr":"172.21.15.196:6820","nonce":254726044}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":254726044},{"type":"v1","addr":"172.21.15.196:6816","nonce":254726044}]},"public_addr":"172.21.15.196:6811/254726044","cluster_addr":"172.21.15.196:6813/254726044","heartbeat_back_addr":"172.21.15.196:6820/254726044","heartbeat_front_addr":"172.21.15.196:6816/254726044","state":["exists","up"]},{"osd":8,"uuid":"9ac666e9-291e-4696-a6bb-e10c9deef3f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":3461147157},{"type":"v1","addr":"172.21.15.62:6809","nonce":3461147157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":3461147157},{"type":"v1","addr":"172.21.15.62:6811","nonce":3461147157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":3461147157},{"type":"v1","addr":"172.21.15.62:6817","nonce":3461147157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":3461147157},{"type":"v1","addr":"172.21.15.62:6813","nonce":3461147157}]},"public_addr":"172.21.15.62:6809/3461147157","cluster_addr":"172.21.15.62:6811/3461147157","heartbeat_back_addr":"172.21.15.62:6817/3461147157","heartbeat_front_addr":"172.21.15.62:6813/3461147157","state":["exists","up"]},{"osd":9,"uuid":"7fe04cb8-ac67-46ac-b2de-1f8f60a50373","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6810","nonce":1859221562},{"type":"v1","addr":"172.21.15.91:6813","nonce":1859221562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6816","nonce":1859221562},{"type":"v1","addr":"172.21.15.91:6820","nonce":1859221562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6829","nonce":1859221562},{"type":"v1","addr":"172.21.15.91:6831","nonce":1859221562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6823","nonce":1859221562},{"type":"v1","addr":"172.21.15.91:6826","nonce":1859221562}]},"public_addr":"172.21.15.91:6813/1859221562","cluster_addr":"172.21.15.91:6820/1859221562","heartbeat_back_addr":"172.21.15.91:6831/1859221562","heartbeat_front_addr":"172.21.15.91:6826/1859221562","state":["exists","up"]},{"osd":10,"uuid":"a6b0ac25-da07-4fee-b646-a4530ce5b343","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2448697550},{"type":"v1","addr":"172.21.15.150:6809","nonce":2448697550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2448697550},{"type":"v1","addr":"172.21.15.150:6811","nonce":2448697550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":2448697550},{"type":"v1","addr":"172.21.15.150:6819","nonce":2448697550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6813","nonce":2448697550},{"type":"v1","addr":"172.21.15.150:6815","nonce":2448697550}]},"public_addr":"172.21.15.150:6809/2448697550","cluster_addr":"172.21.15.150:6811/2448697550","heartbeat_back_addr":"172.21.15.150:6819/2448697550","heartbeat_front_addr":"172.21.15.150:6815/2448697550","state":["exists","up"]},{"osd":11,"uuid":"31b22258-77da-495c-b518-9ce0297b9165","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6815","nonce":1170435852},{"type":"v1","addr":"172.21.15.196:6817","nonce":1170435852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6819","nonce":1170435852},{"type":"v1","addr":"172.21.15.196:6821","nonce":1170435852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":1170435852},{"type":"v1","addr":"172.21.15.196:6825","nonce":1170435852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":1170435852},{"type":"v1","addr":"172.21.15.196:6823","nonce":1170435852}]},"public_addr":"172.21.15.196:6817/1170435852","cluster_addr":"172.21.15.196:6821/1170435852","heartbeat_back_addr":"172.21.15.196:6825/1170435852","heartbeat_front_addr":"172.21.15.196:6823/1170435852","state":["exists","up"]},{"osd":12,"uuid":"322917eb-26ba-4439-8300-4d723bada623","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":1173398345},{"type":"v1","addr":"172.21.15.62:6825","nonce":1173398345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":1173398345},{"type":"v1","addr":"172.21.15.62:6827","nonce":1173398345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":1173398345},{"type":"v1","addr":"172.21.15.62:6831","nonce":1173398345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":1173398345},{"type":"v1","addr":"172.21.15.62:6829","nonce":1173398345}]},"public_addr":"172.21.15.62:6825/1173398345","cluster_addr":"172.21.15.62:6827/1173398345","heartbeat_back_addr":"172.21.15.62:6831/1173398345","heartbeat_front_addr":"172.21.15.62:6829/1173398345","state":["exists","up"]},{"osd":13,"uuid":"9c708044-4726-42b6-9c13-03963f956d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6809","nonce":3656255795},{"type":"v1","addr":"172.21.15.91:6812","nonce":3656255795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6815","nonce":3656255795},{"type":"v1","addr":"172.21.15.91:6817","nonce":3656255795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6825","nonce":3656255795},{"type":"v1","addr":"172.21.15.91:6827","nonce":3656255795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6819","nonce":3656255795},{"type":"v1","addr":"172.21.15.91:6822","nonce":3656255795}]},"public_addr":"172.21.15.91:6812/3656255795","cluster_addr":"172.21.15.91:6817/3656255795","heartbeat_back_addr":"172.21.15.91:6827/3656255795","heartbeat_front_addr":"172.21.15.91:6822/3656255795","state":["exists","up"]},{"osd":14,"uuid":"0208144a-db9a-404c-b31e-13bc2e276b09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2102158830},{"type":"v1","addr":"172.21.15.150:6814","nonce":2102158830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":2102158830},{"type":"v1","addr":"172.21.15.150:6818","nonce":2102158830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":2102158830},{"type":"v1","addr":"172.21.15.150:6824","nonce":2102158830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":2102158830},{"type":"v1","addr":"172.21.15.150:6821","nonce":2102158830}]},"public_addr":"172.21.15.150:6814/2102158830","cluster_addr":"172.21.15.150:6818/2102158830","heartbeat_back_addr":"172.21.15.150:6824/2102158830","heartbeat_front_addr":"172.21.15.150:6821/2102158830","state":["exists","up"]},{"osd":15,"uuid":"0591e7e1-dc07-4d80-b574-9b8348635764","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":3187639683},{"type":"v1","addr":"172.21.15.196:6827","nonce":3187639683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":3187639683},{"type":"v1","addr":"172.21.15.196:6829","nonce":3187639683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":3187639683},{"type":"v1","addr":"172.21.15.196:6833","nonce":3187639683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":3187639683},{"type":"v1","addr":"172.21.15.196:6831","nonce":3187639683}]},"public_addr":"172.21.15.196:6827/3187639683","cluster_addr":"172.21.15.196:6829/3187639683","heartbeat_back_addr":"172.21.15.196:6833/3187639683","heartbeat_front_addr":"172.21.15.196:6831/3187639683","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T04:53:04.797667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T04:53:04.810592+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T04:53:04.897353+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T04:53:04.736168+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T04:53:04.778014+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T04:53:04.777687+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T04:53:04.774966+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T04:53:04.890390+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T04:53:04.851604+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T04:53:04.907238+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T04:53:04.901590+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T04:53:04.880851+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T04:53:04.803538+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T04:53:04.815624+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T04:53:04.896409+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T04:53:04.844885+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-05T04:53:09.804 DEBUG:teuthology.misc:16 of 16 OSDs are up 2024-02-05T04:53:09.804 INFO:tasks.ceph:Creating RBD pool 2024-02-05T04:53:09.804 DEBUG:teuthology.orchestra.run.smithi062:> sudo ceph --cluster ceph osd pool create rbd 8 2024-02-05T04:53:11.018 INFO:teuthology.orchestra.run.smithi062.stderr:pool 'rbd' created 2024-02-05T04:53:11.032 DEBUG:teuthology.orchestra.run.smithi062:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-02-05T04:53:12.023 INFO:teuthology.orchestra.run.smithi062.stderr:enabled application 'rbd' on pool 'rbd' 2024-02-05T04:53:12.041 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-02-05T04:53:12.041 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-02-05T04:53:12.041 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-02-05T04:53:12.041 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-02-05T04:53:12.042 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-02-05T04:53:12.387 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:53:12.397 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":5,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":2623653333},{"type":"v1","addr":"172.21.15.196:6801","nonce":2623653333}]},"active_addr":"172.21.15.196:6801/2623653333","active_change":"2024-02-05T04:52:59.497892+0000","active_mgr_features":4540701272860131327,"available":true,"standbys":[{"gid":4102,"name":"y","mgr_features":4540701272860131327,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.196:0","nonce":2182569581}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.196:0","nonce":3672061555}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.196:0","nonce":4177015456}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.196:0","nonce":251397813}]}]} 2024-02-05T04:53:12.401 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-02-05T04:53:12.401 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-02-05T04:53:12.401 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-05T04:53:12.695 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:53:12.696 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":27,"fsid":"88c11a2d-382e-4f21-b8af-45ab9c504b93","created":"2024-02-05T04:52:55.182961+0000","modified":"2024-02-05T04:53:12.019105+0000","last_up_change":"2024-02-05T04:53:07.900741+0000","last_in_change":"2024-02-05T04:52:59.695913+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-05T04:53:07.549923+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-02-05T04:53:10.126961+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.0199999809265137,"score_stable":4.0199999809265137,"optimal_score":0.56000000238418579,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cdb503c5-a32c-4a9a-b1d9-194ab5ee48aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":1189094240},{"type":"v1","addr":"172.21.15.62:6816","nonce":1189094240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":1189094240},{"type":"v1","addr":"172.21.15.62:6819","nonce":1189094240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":1189094240},{"type":"v1","addr":"172.21.15.62:6823","nonce":1189094240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":1189094240},{"type":"v1","addr":"172.21.15.62:6821","nonce":1189094240}]},"public_addr":"172.21.15.62:6816/1189094240","cluster_addr":"172.21.15.62:6819/1189094240","heartbeat_back_addr":"172.21.15.62:6823/1189094240","heartbeat_front_addr":"172.21.15.62:6821/1189094240","state":["exists","up"]},{"osd":1,"uuid":"d7a3d394-74b2-4cb1-a754-4fe77e605ea9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6808","nonce":2878185190},{"type":"v1","addr":"172.21.15.91:6811","nonce":2878185190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6814","nonce":2878185190},{"type":"v1","addr":"172.21.15.91:6818","nonce":2878185190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6828","nonce":2878185190},{"type":"v1","addr":"172.21.15.91:6830","nonce":2878185190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6821","nonce":2878185190},{"type":"v1","addr":"172.21.15.91:6824","nonce":2878185190}]},"public_addr":"172.21.15.91:6811/2878185190","cluster_addr":"172.21.15.91:6818/2878185190","heartbeat_back_addr":"172.21.15.91:6830/2878185190","heartbeat_front_addr":"172.21.15.91:6824/2878185190","state":["exists","up"]},{"osd":2,"uuid":"d6e99efe-475f-41c1-88a3-87ce6b26ce75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":40232889},{"type":"v1","addr":"172.21.15.150:6825","nonce":40232889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":40232889},{"type":"v1","addr":"172.21.15.150:6827","nonce":40232889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":40232889},{"type":"v1","addr":"172.21.15.150:6831","nonce":40232889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":40232889},{"type":"v1","addr":"172.21.15.150:6829","nonce":40232889}]},"public_addr":"172.21.15.150:6825/40232889","cluster_addr":"172.21.15.150:6827/40232889","heartbeat_back_addr":"172.21.15.150:6831/40232889","heartbeat_front_addr":"172.21.15.150:6829/40232889","state":["exists","up"]},{"osd":3,"uuid":"3c964090-5e07-40bd-ac7c-b66e29eb25b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":3834523862},{"type":"v1","addr":"172.21.15.196:6803","nonce":3834523862}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":3834523862},{"type":"v1","addr":"172.21.15.196:6805","nonce":3834523862}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":3834523862},{"type":"v1","addr":"172.21.15.196:6809","nonce":3834523862}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":3834523862},{"type":"v1","addr":"172.21.15.196:6807","nonce":3834523862}]},"public_addr":"172.21.15.196:6803/3834523862","cluster_addr":"172.21.15.196:6805/3834523862","heartbeat_back_addr":"172.21.15.196:6809/3834523862","heartbeat_front_addr":"172.21.15.196:6807/3834523862","state":["exists","up"]},{"osd":4,"uuid":"22bd25f9-3d93-4bcb-bcfc-71feae1097f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":2244900897},{"type":"v1","addr":"172.21.15.62:6801","nonce":2244900897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":2244900897},{"type":"v1","addr":"172.21.15.62:6803","nonce":2244900897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":2244900897},{"type":"v1","addr":"172.21.15.62:6807","nonce":2244900897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":2244900897},{"type":"v1","addr":"172.21.15.62:6805","nonce":2244900897}]},"public_addr":"172.21.15.62:6801/2244900897","cluster_addr":"172.21.15.62:6803/2244900897","heartbeat_back_addr":"172.21.15.62:6807/2244900897","heartbeat_front_addr":"172.21.15.62:6805/2244900897","state":["exists","up"]},{"osd":5,"uuid":"c6ae2bd4-4a4f-4f23-8991-743ffa42ed3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6800","nonce":473596866},{"type":"v1","addr":"172.21.15.91:6801","nonce":473596866}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6802","nonce":473596866},{"type":"v1","addr":"172.21.15.91:6803","nonce":473596866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6806","nonce":473596866},{"type":"v1","addr":"172.21.15.91:6807","nonce":473596866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6804","nonce":473596866},{"type":"v1","addr":"172.21.15.91:6805","nonce":473596866}]},"public_addr":"172.21.15.91:6801/473596866","cluster_addr":"172.21.15.91:6803/473596866","heartbeat_back_addr":"172.21.15.91:6807/473596866","heartbeat_front_addr":"172.21.15.91:6805/473596866","state":["exists","up"]},{"osd":6,"uuid":"49da8b05-ffa3-4172-983e-f081d3eb9ae6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":2199648645},{"type":"v1","addr":"172.21.15.150:6801","nonce":2199648645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":2199648645},{"type":"v1","addr":"172.21.15.150:6803","nonce":2199648645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":2199648645},{"type":"v1","addr":"172.21.15.150:6807","nonce":2199648645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":2199648645},{"type":"v1","addr":"172.21.15.150:6805","nonce":2199648645}]},"public_addr":"172.21.15.150:6801/2199648645","cluster_addr":"172.21.15.150:6803/2199648645","heartbeat_back_addr":"172.21.15.150:6807/2199648645","heartbeat_front_addr":"172.21.15.150:6805/2199648645","state":["exists","up"]},{"osd":7,"uuid":"4ece26e1-2c63-4223-baaf-882cd8261ade","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":254726044},{"type":"v1","addr":"172.21.15.196:6811","nonce":254726044}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":254726044},{"type":"v1","addr":"172.21.15.196:6813","nonce":254726044}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":254726044},{"type":"v1","addr":"172.21.15.196:6820","nonce":254726044}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":254726044},{"type":"v1","addr":"172.21.15.196:6816","nonce":254726044}]},"public_addr":"172.21.15.196:6811/254726044","cluster_addr":"172.21.15.196:6813/254726044","heartbeat_back_addr":"172.21.15.196:6820/254726044","heartbeat_front_addr":"172.21.15.196:6816/254726044","state":["exists","up"]},{"osd":8,"uuid":"9ac666e9-291e-4696-a6bb-e10c9deef3f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":3461147157},{"type":"v1","addr":"172.21.15.62:6809","nonce":3461147157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":3461147157},{"type":"v1","addr":"172.21.15.62:6811","nonce":3461147157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":3461147157},{"type":"v1","addr":"172.21.15.62:6817","nonce":3461147157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":3461147157},{"type":"v1","addr":"172.21.15.62:6813","nonce":3461147157}]},"public_addr":"172.21.15.62:6809/3461147157","cluster_addr":"172.21.15.62:6811/3461147157","heartbeat_back_addr":"172.21.15.62:6817/3461147157","heartbeat_front_addr":"172.21.15.62:6813/3461147157","state":["exists","up"]},{"osd":9,"uuid":"7fe04cb8-ac67-46ac-b2de-1f8f60a50373","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6810","nonce":1859221562},{"type":"v1","addr":"172.21.15.91:6813","nonce":1859221562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6816","nonce":1859221562},{"type":"v1","addr":"172.21.15.91:6820","nonce":1859221562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6829","nonce":1859221562},{"type":"v1","addr":"172.21.15.91:6831","nonce":1859221562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6823","nonce":1859221562},{"type":"v1","addr":"172.21.15.91:6826","nonce":1859221562}]},"public_addr":"172.21.15.91:6813/1859221562","cluster_addr":"172.21.15.91:6820/1859221562","heartbeat_back_addr":"172.21.15.91:6831/1859221562","heartbeat_front_addr":"172.21.15.91:6826/1859221562","state":["exists","up"]},{"osd":10,"uuid":"a6b0ac25-da07-4fee-b646-a4530ce5b343","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2448697550},{"type":"v1","addr":"172.21.15.150:6809","nonce":2448697550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2448697550},{"type":"v1","addr":"172.21.15.150:6811","nonce":2448697550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":2448697550},{"type":"v1","addr":"172.21.15.150:6819","nonce":2448697550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6813","nonce":2448697550},{"type":"v1","addr":"172.21.15.150:6815","nonce":2448697550}]},"public_addr":"172.21.15.150:6809/2448697550","cluster_addr":"172.21.15.150:6811/2448697550","heartbeat_back_addr":"172.21.15.150:6819/2448697550","heartbeat_front_addr":"172.21.15.150:6815/2448697550","state":["exists","up"]},{"osd":11,"uuid":"31b22258-77da-495c-b518-9ce0297b9165","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6815","nonce":1170435852},{"type":"v1","addr":"172.21.15.196:6817","nonce":1170435852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6819","nonce":1170435852},{"type":"v1","addr":"172.21.15.196:6821","nonce":1170435852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":1170435852},{"type":"v1","addr":"172.21.15.196:6825","nonce":1170435852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":1170435852},{"type":"v1","addr":"172.21.15.196:6823","nonce":1170435852}]},"public_addr":"172.21.15.196:6817/1170435852","cluster_addr":"172.21.15.196:6821/1170435852","heartbeat_back_addr":"172.21.15.196:6825/1170435852","heartbeat_front_addr":"172.21.15.196:6823/1170435852","state":["exists","up"]},{"osd":12,"uuid":"322917eb-26ba-4439-8300-4d723bada623","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":1173398345},{"type":"v1","addr":"172.21.15.62:6825","nonce":1173398345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":1173398345},{"type":"v1","addr":"172.21.15.62:6827","nonce":1173398345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":1173398345},{"type":"v1","addr":"172.21.15.62:6831","nonce":1173398345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":1173398345},{"type":"v1","addr":"172.21.15.62:6829","nonce":1173398345}]},"public_addr":"172.21.15.62:6825/1173398345","cluster_addr":"172.21.15.62:6827/1173398345","heartbeat_back_addr":"172.21.15.62:6831/1173398345","heartbeat_front_addr":"172.21.15.62:6829/1173398345","state":["exists","up"]},{"osd":13,"uuid":"9c708044-4726-42b6-9c13-03963f956d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6809","nonce":3656255795},{"type":"v1","addr":"172.21.15.91:6812","nonce":3656255795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6815","nonce":3656255795},{"type":"v1","addr":"172.21.15.91:6817","nonce":3656255795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6825","nonce":3656255795},{"type":"v1","addr":"172.21.15.91:6827","nonce":3656255795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6819","nonce":3656255795},{"type":"v1","addr":"172.21.15.91:6822","nonce":3656255795}]},"public_addr":"172.21.15.91:6812/3656255795","cluster_addr":"172.21.15.91:6817/3656255795","heartbeat_back_addr":"172.21.15.91:6827/3656255795","heartbeat_front_addr":"172.21.15.91:6822/3656255795","state":["exists","up"]},{"osd":14,"uuid":"0208144a-db9a-404c-b31e-13bc2e276b09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2102158830},{"type":"v1","addr":"172.21.15.150:6814","nonce":2102158830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":2102158830},{"type":"v1","addr":"172.21.15.150:6818","nonce":2102158830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":2102158830},{"type":"v1","addr":"172.21.15.150:6824","nonce":2102158830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":2102158830},{"type":"v1","addr":"172.21.15.150:6821","nonce":2102158830}]},"public_addr":"172.21.15.150:6814/2102158830","cluster_addr":"172.21.15.150:6818/2102158830","heartbeat_back_addr":"172.21.15.150:6824/2102158830","heartbeat_front_addr":"172.21.15.150:6821/2102158830","state":["exists","up"]},{"osd":15,"uuid":"0591e7e1-dc07-4d80-b574-9b8348635764","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":3187639683},{"type":"v1","addr":"172.21.15.196:6827","nonce":3187639683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":3187639683},{"type":"v1","addr":"172.21.15.196:6829","nonce":3187639683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":3187639683},{"type":"v1","addr":"172.21.15.196:6833","nonce":3187639683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":3187639683},{"type":"v1","addr":"172.21.15.196:6831","nonce":3187639683}]},"public_addr":"172.21.15.196:6827/3187639683","cluster_addr":"172.21.15.196:6829/3187639683","heartbeat_back_addr":"172.21.15.196:6833/3187639683","heartbeat_front_addr":"172.21.15.196:6831/3187639683","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T04:53:04.797667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T04:53:04.810592+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T04:53:04.897353+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T04:53:04.736168+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T04:53:04.778014+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T04:53:04.777687+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T04:53:04.774966+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T04:53:04.890390+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T04:53:04.851604+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T04:53:04.907238+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T04:53:04.901590+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T04:53:04.880851+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T04:53:04.803538+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T04:53:04.815624+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T04:53:04.896409+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T04:53:04.844885+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-05T04:53:12.708 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-02-05T04:53:12.708 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-05T04:53:13.003 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:53:13.003 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":27,"fsid":"88c11a2d-382e-4f21-b8af-45ab9c504b93","created":"2024-02-05T04:52:55.182961+0000","modified":"2024-02-05T04:53:12.019105+0000","last_up_change":"2024-02-05T04:53:07.900741+0000","last_in_change":"2024-02-05T04:52:59.695913+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-05T04:53:07.549923+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-02-05T04:53:10.126961+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.0199999809265137,"score_stable":4.0199999809265137,"optimal_score":0.56000000238418579,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cdb503c5-a32c-4a9a-b1d9-194ab5ee48aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":1189094240},{"type":"v1","addr":"172.21.15.62:6816","nonce":1189094240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":1189094240},{"type":"v1","addr":"172.21.15.62:6819","nonce":1189094240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":1189094240},{"type":"v1","addr":"172.21.15.62:6823","nonce":1189094240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":1189094240},{"type":"v1","addr":"172.21.15.62:6821","nonce":1189094240}]},"public_addr":"172.21.15.62:6816/1189094240","cluster_addr":"172.21.15.62:6819/1189094240","heartbeat_back_addr":"172.21.15.62:6823/1189094240","heartbeat_front_addr":"172.21.15.62:6821/1189094240","state":["exists","up"]},{"osd":1,"uuid":"d7a3d394-74b2-4cb1-a754-4fe77e605ea9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6808","nonce":2878185190},{"type":"v1","addr":"172.21.15.91:6811","nonce":2878185190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6814","nonce":2878185190},{"type":"v1","addr":"172.21.15.91:6818","nonce":2878185190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6828","nonce":2878185190},{"type":"v1","addr":"172.21.15.91:6830","nonce":2878185190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6821","nonce":2878185190},{"type":"v1","addr":"172.21.15.91:6824","nonce":2878185190}]},"public_addr":"172.21.15.91:6811/2878185190","cluster_addr":"172.21.15.91:6818/2878185190","heartbeat_back_addr":"172.21.15.91:6830/2878185190","heartbeat_front_addr":"172.21.15.91:6824/2878185190","state":["exists","up"]},{"osd":2,"uuid":"d6e99efe-475f-41c1-88a3-87ce6b26ce75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":40232889},{"type":"v1","addr":"172.21.15.150:6825","nonce":40232889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":40232889},{"type":"v1","addr":"172.21.15.150:6827","nonce":40232889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":40232889},{"type":"v1","addr":"172.21.15.150:6831","nonce":40232889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":40232889},{"type":"v1","addr":"172.21.15.150:6829","nonce":40232889}]},"public_addr":"172.21.15.150:6825/40232889","cluster_addr":"172.21.15.150:6827/40232889","heartbeat_back_addr":"172.21.15.150:6831/40232889","heartbeat_front_addr":"172.21.15.150:6829/40232889","state":["exists","up"]},{"osd":3,"uuid":"3c964090-5e07-40bd-ac7c-b66e29eb25b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":3834523862},{"type":"v1","addr":"172.21.15.196:6803","nonce":3834523862}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":3834523862},{"type":"v1","addr":"172.21.15.196:6805","nonce":3834523862}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":3834523862},{"type":"v1","addr":"172.21.15.196:6809","nonce":3834523862}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":3834523862},{"type":"v1","addr":"172.21.15.196:6807","nonce":3834523862}]},"public_addr":"172.21.15.196:6803/3834523862","cluster_addr":"172.21.15.196:6805/3834523862","heartbeat_back_addr":"172.21.15.196:6809/3834523862","heartbeat_front_addr":"172.21.15.196:6807/3834523862","state":["exists","up"]},{"osd":4,"uuid":"22bd25f9-3d93-4bcb-bcfc-71feae1097f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":2244900897},{"type":"v1","addr":"172.21.15.62:6801","nonce":2244900897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":2244900897},{"type":"v1","addr":"172.21.15.62:6803","nonce":2244900897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":2244900897},{"type":"v1","addr":"172.21.15.62:6807","nonce":2244900897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":2244900897},{"type":"v1","addr":"172.21.15.62:6805","nonce":2244900897}]},"public_addr":"172.21.15.62:6801/2244900897","cluster_addr":"172.21.15.62:6803/2244900897","heartbeat_back_addr":"172.21.15.62:6807/2244900897","heartbeat_front_addr":"172.21.15.62:6805/2244900897","state":["exists","up"]},{"osd":5,"uuid":"c6ae2bd4-4a4f-4f23-8991-743ffa42ed3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6800","nonce":473596866},{"type":"v1","addr":"172.21.15.91:6801","nonce":473596866}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6802","nonce":473596866},{"type":"v1","addr":"172.21.15.91:6803","nonce":473596866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6806","nonce":473596866},{"type":"v1","addr":"172.21.15.91:6807","nonce":473596866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6804","nonce":473596866},{"type":"v1","addr":"172.21.15.91:6805","nonce":473596866}]},"public_addr":"172.21.15.91:6801/473596866","cluster_addr":"172.21.15.91:6803/473596866","heartbeat_back_addr":"172.21.15.91:6807/473596866","heartbeat_front_addr":"172.21.15.91:6805/473596866","state":["exists","up"]},{"osd":6,"uuid":"49da8b05-ffa3-4172-983e-f081d3eb9ae6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":2199648645},{"type":"v1","addr":"172.21.15.150:6801","nonce":2199648645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":2199648645},{"type":"v1","addr":"172.21.15.150:6803","nonce":2199648645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":2199648645},{"type":"v1","addr":"172.21.15.150:6807","nonce":2199648645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":2199648645},{"type":"v1","addr":"172.21.15.150:6805","nonce":2199648645}]},"public_addr":"172.21.15.150:6801/2199648645","cluster_addr":"172.21.15.150:6803/2199648645","heartbeat_back_addr":"172.21.15.150:6807/2199648645","heartbeat_front_addr":"172.21.15.150:6805/2199648645","state":["exists","up"]},{"osd":7,"uuid":"4ece26e1-2c63-4223-baaf-882cd8261ade","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":254726044},{"type":"v1","addr":"172.21.15.196:6811","nonce":254726044}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":254726044},{"type":"v1","addr":"172.21.15.196:6813","nonce":254726044}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":254726044},{"type":"v1","addr":"172.21.15.196:6820","nonce":254726044}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":254726044},{"type":"v1","addr":"172.21.15.196:6816","nonce":254726044}]},"public_addr":"172.21.15.196:6811/254726044","cluster_addr":"172.21.15.196:6813/254726044","heartbeat_back_addr":"172.21.15.196:6820/254726044","heartbeat_front_addr":"172.21.15.196:6816/254726044","state":["exists","up"]},{"osd":8,"uuid":"9ac666e9-291e-4696-a6bb-e10c9deef3f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":3461147157},{"type":"v1","addr":"172.21.15.62:6809","nonce":3461147157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":3461147157},{"type":"v1","addr":"172.21.15.62:6811","nonce":3461147157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":3461147157},{"type":"v1","addr":"172.21.15.62:6817","nonce":3461147157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":3461147157},{"type":"v1","addr":"172.21.15.62:6813","nonce":3461147157}]},"public_addr":"172.21.15.62:6809/3461147157","cluster_addr":"172.21.15.62:6811/3461147157","heartbeat_back_addr":"172.21.15.62:6817/3461147157","heartbeat_front_addr":"172.21.15.62:6813/3461147157","state":["exists","up"]},{"osd":9,"uuid":"7fe04cb8-ac67-46ac-b2de-1f8f60a50373","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6810","nonce":1859221562},{"type":"v1","addr":"172.21.15.91:6813","nonce":1859221562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6816","nonce":1859221562},{"type":"v1","addr":"172.21.15.91:6820","nonce":1859221562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6829","nonce":1859221562},{"type":"v1","addr":"172.21.15.91:6831","nonce":1859221562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6823","nonce":1859221562},{"type":"v1","addr":"172.21.15.91:6826","nonce":1859221562}]},"public_addr":"172.21.15.91:6813/1859221562","cluster_addr":"172.21.15.91:6820/1859221562","heartbeat_back_addr":"172.21.15.91:6831/1859221562","heartbeat_front_addr":"172.21.15.91:6826/1859221562","state":["exists","up"]},{"osd":10,"uuid":"a6b0ac25-da07-4fee-b646-a4530ce5b343","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2448697550},{"type":"v1","addr":"172.21.15.150:6809","nonce":2448697550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2448697550},{"type":"v1","addr":"172.21.15.150:6811","nonce":2448697550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":2448697550},{"type":"v1","addr":"172.21.15.150:6819","nonce":2448697550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6813","nonce":2448697550},{"type":"v1","addr":"172.21.15.150:6815","nonce":2448697550}]},"public_addr":"172.21.15.150:6809/2448697550","cluster_addr":"172.21.15.150:6811/2448697550","heartbeat_back_addr":"172.21.15.150:6819/2448697550","heartbeat_front_addr":"172.21.15.150:6815/2448697550","state":["exists","up"]},{"osd":11,"uuid":"31b22258-77da-495c-b518-9ce0297b9165","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6815","nonce":1170435852},{"type":"v1","addr":"172.21.15.196:6817","nonce":1170435852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6819","nonce":1170435852},{"type":"v1","addr":"172.21.15.196:6821","nonce":1170435852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":1170435852},{"type":"v1","addr":"172.21.15.196:6825","nonce":1170435852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":1170435852},{"type":"v1","addr":"172.21.15.196:6823","nonce":1170435852}]},"public_addr":"172.21.15.196:6817/1170435852","cluster_addr":"172.21.15.196:6821/1170435852","heartbeat_back_addr":"172.21.15.196:6825/1170435852","heartbeat_front_addr":"172.21.15.196:6823/1170435852","state":["exists","up"]},{"osd":12,"uuid":"322917eb-26ba-4439-8300-4d723bada623","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":1173398345},{"type":"v1","addr":"172.21.15.62:6825","nonce":1173398345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":1173398345},{"type":"v1","addr":"172.21.15.62:6827","nonce":1173398345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":1173398345},{"type":"v1","addr":"172.21.15.62:6831","nonce":1173398345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":1173398345},{"type":"v1","addr":"172.21.15.62:6829","nonce":1173398345}]},"public_addr":"172.21.15.62:6825/1173398345","cluster_addr":"172.21.15.62:6827/1173398345","heartbeat_back_addr":"172.21.15.62:6831/1173398345","heartbeat_front_addr":"172.21.15.62:6829/1173398345","state":["exists","up"]},{"osd":13,"uuid":"9c708044-4726-42b6-9c13-03963f956d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6809","nonce":3656255795},{"type":"v1","addr":"172.21.15.91:6812","nonce":3656255795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6815","nonce":3656255795},{"type":"v1","addr":"172.21.15.91:6817","nonce":3656255795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6825","nonce":3656255795},{"type":"v1","addr":"172.21.15.91:6827","nonce":3656255795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6819","nonce":3656255795},{"type":"v1","addr":"172.21.15.91:6822","nonce":3656255795}]},"public_addr":"172.21.15.91:6812/3656255795","cluster_addr":"172.21.15.91:6817/3656255795","heartbeat_back_addr":"172.21.15.91:6827/3656255795","heartbeat_front_addr":"172.21.15.91:6822/3656255795","state":["exists","up"]},{"osd":14,"uuid":"0208144a-db9a-404c-b31e-13bc2e276b09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2102158830},{"type":"v1","addr":"172.21.15.150:6814","nonce":2102158830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":2102158830},{"type":"v1","addr":"172.21.15.150:6818","nonce":2102158830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":2102158830},{"type":"v1","addr":"172.21.15.150:6824","nonce":2102158830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":2102158830},{"type":"v1","addr":"172.21.15.150:6821","nonce":2102158830}]},"public_addr":"172.21.15.150:6814/2102158830","cluster_addr":"172.21.15.150:6818/2102158830","heartbeat_back_addr":"172.21.15.150:6824/2102158830","heartbeat_front_addr":"172.21.15.150:6821/2102158830","state":["exists","up"]},{"osd":15,"uuid":"0591e7e1-dc07-4d80-b574-9b8348635764","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":3187639683},{"type":"v1","addr":"172.21.15.196:6827","nonce":3187639683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":3187639683},{"type":"v1","addr":"172.21.15.196:6829","nonce":3187639683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":3187639683},{"type":"v1","addr":"172.21.15.196:6833","nonce":3187639683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":3187639683},{"type":"v1","addr":"172.21.15.196:6831","nonce":3187639683}]},"public_addr":"172.21.15.196:6827/3187639683","cluster_addr":"172.21.15.196:6829/3187639683","heartbeat_back_addr":"172.21.15.196:6833/3187639683","heartbeat_front_addr":"172.21.15.196:6831/3187639683","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T04:53:04.797667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T04:53:04.810592+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T04:53:04.897353+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T04:53:04.736168+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T04:53:04.778014+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T04:53:04.777687+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T04:53:04.774966+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T04:53:04.890390+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T04:53:04.851604+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T04:53:04.907238+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T04:53:04.901590+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T04:53:04.880851+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T04:53:04.803538+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T04:53:04.815624+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T04:53:04.896409+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T04:53:04.844885+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-05T04:53:13.016 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-02-05T04:53:13.016 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-02-05T04:53:13.017 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-02-05T04:53:13.017 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-02-05T04:53:13.018 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-02-05T04:53:13.018 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-02-05T04:53:13.018 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-02-05T04:53:13.019 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-02-05T04:53:13.019 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 flush_pg_stats 2024-02-05T04:53:13.020 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 flush_pg_stats 2024-02-05T04:53:13.020 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 flush_pg_stats 2024-02-05T04:53:13.020 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.11 flush_pg_stats 2024-02-05T04:53:13.021 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.12 flush_pg_stats 2024-02-05T04:53:13.021 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.13 flush_pg_stats 2024-02-05T04:53:13.022 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.14 flush_pg_stats 2024-02-05T04:53:13.022 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.15 flush_pg_stats 2024-02-05T04:53:13.443 INFO:teuthology.orchestra.run.smithi062.stdout:98784247813 2024-02-05T04:53:13.443 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-02-05T04:53:13.454 INFO:teuthology.orchestra.run.smithi062.stdout:98784247813 2024-02-05T04:53:13.455 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-02-05T04:53:13.465 INFO:teuthology.orchestra.run.smithi062.stdout:94489280517 2024-02-05T04:53:13.465 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-02-05T04:53:13.479 INFO:teuthology.orchestra.run.smithi062.stdout:98784247813 2024-02-05T04:53:13.479 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.11 2024-02-05T04:53:13.486 INFO:teuthology.orchestra.run.smithi062.stdout:90194313221 2024-02-05T04:53:13.486 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-02-05T04:53:13.507 INFO:teuthology.orchestra.run.smithi062.stdout:98784247812 2024-02-05T04:53:13.508 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-02-05T04:53:13.513 INFO:teuthology.orchestra.run.smithi062.stdout:98784247811 2024-02-05T04:53:13.514 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.12 2024-02-05T04:53:13.524 INFO:teuthology.orchestra.run.smithi062.stdout:94489280515 2024-02-05T04:53:13.524 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.13 2024-02-05T04:53:13.526 INFO:teuthology.orchestra.run.smithi062.stdout:98784247810 2024-02-05T04:53:13.526 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.8 2024-02-05T04:53:13.528 INFO:teuthology.orchestra.run.smithi062.stdout:90194313220 2024-02-05T04:53:13.528 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-02-05T04:53:13.531 INFO:teuthology.orchestra.run.smithi062.stdout:94489280516 2024-02-05T04:53:13.531 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.14 2024-02-05T04:53:13.542 INFO:teuthology.orchestra.run.smithi062.stdout:90194313220 2024-02-05T04:53:13.542 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-02-05T04:53:13.553 INFO:teuthology.orchestra.run.smithi062.stdout:98784247813 2024-02-05T04:53:13.553 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.10 2024-02-05T04:53:13.561 INFO:teuthology.orchestra.run.smithi062.stdout:90194313221 2024-02-05T04:53:13.561 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-02-05T04:53:13.576 INFO:teuthology.orchestra.run.smithi062.stdout:94489280517 2024-02-05T04:53:13.576 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.15 2024-02-05T04:53:13.583 INFO:teuthology.orchestra.run.smithi062.stdout:94489280516 2024-02-05T04:53:13.584 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.9 2024-02-05T04:53:14.403 INFO:teuthology.orchestra.run.smithi062.stdout:98784247813 2024-02-05T04:53:14.427 INFO:teuthology.orchestra.run.smithi062.stdout:94489280517 2024-02-05T04:53:14.431 INFO:teuthology.orchestra.run.smithi062.stdout:94489280515 2024-02-05T04:53:14.442 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247813 got 98784247813 for osd.7 2024-02-05T04:53:14.443 DEBUG:teuthology.parallel:result is None 2024-02-05T04:53:14.449 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280517 got 94489280517 for osd.1 2024-02-05T04:53:14.449 DEBUG:teuthology.parallel:result is None 2024-02-05T04:53:14.469 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280515 for osd.13 2024-02-05T04:53:14.469 DEBUG:teuthology.parallel:result is None 2024-02-05T04:53:14.611 INFO:teuthology.orchestra.run.smithi062.stdout:98784247810 2024-02-05T04:53:14.634 INFO:teuthology.orchestra.run.smithi062.stdout:94489280516 2024-02-05T04:53:14.658 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247810 got 98784247810 for osd.8 2024-02-05T04:53:14.658 DEBUG:teuthology.parallel:result is None 2024-02-05T04:53:14.665 INFO:teuthology.orchestra.run.smithi062.stdout:90194313220 2024-02-05T04:53:14.674 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280517 got 94489280516 for osd.15 2024-02-05T04:53:14.676 INFO:teuthology.orchestra.run.smithi062.stdout:98784247813 2024-02-05T04:53:14.698 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313221 got 90194313220 for osd.6 2024-02-05T04:53:14.717 INFO:teuthology.orchestra.run.smithi062.stdout:94489280515 2024-02-05T04:53:14.725 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247813 got 98784247813 for osd.2 2024-02-05T04:53:14.725 DEBUG:teuthology.parallel:result is None 2024-02-05T04:53:14.734 INFO:teuthology.orchestra.run.smithi062.stdout:98784247813 2024-02-05T04:53:14.745 INFO:teuthology.orchestra.run.smithi062.stdout:90194313220 2024-02-05T04:53:14.748 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280515 for osd.9 2024-02-05T04:53:14.748 INFO:teuthology.orchestra.run.smithi062.stdout:90194313221 2024-02-05T04:53:14.760 INFO:teuthology.orchestra.run.smithi062.stdout:98784247811 2024-02-05T04:53:14.763 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247813 got 98784247813 for osd.10 2024-02-05T04:53:14.763 DEBUG:teuthology.parallel:result is None 2024-02-05T04:53:14.771 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313221 got 90194313221 for osd.3 2024-02-05T04:53:14.771 DEBUG:teuthology.parallel:result is None 2024-02-05T04:53:14.772 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313220 got 90194313220 for osd.5 2024-02-05T04:53:14.772 DEBUG:teuthology.parallel:result is None 2024-02-05T04:53:14.774 INFO:teuthology.orchestra.run.smithi062.stdout:98784247812 2024-02-05T04:53:14.781 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247811 got 98784247811 for osd.12 2024-02-05T04:53:14.781 DEBUG:teuthology.parallel:result is None 2024-02-05T04:53:14.781 INFO:teuthology.orchestra.run.smithi062.stdout:94489280516 2024-02-05T04:53:14.788 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247812 got 98784247812 for osd.0 2024-02-05T04:53:14.789 DEBUG:teuthology.parallel:result is None 2024-02-05T04:53:14.795 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.14 2024-02-05T04:53:14.795 DEBUG:teuthology.parallel:result is None 2024-02-05T04:53:14.797 INFO:teuthology.orchestra.run.smithi062.stdout:98784247813 2024-02-05T04:53:14.798 INFO:teuthology.orchestra.run.smithi062.stdout:90194313220 2024-02-05T04:53:14.815 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313220 got 90194313220 for osd.4 2024-02-05T04:53:14.816 DEBUG:teuthology.parallel:result is None 2024-02-05T04:53:14.817 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247813 got 98784247813 for osd.11 2024-02-05T04:53:14.817 DEBUG:teuthology.parallel:result is None 2024-02-05T04:53:15.675 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.15 2024-02-05T04:53:15.698 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-02-05T04:53:15.749 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.9 2024-02-05T04:53:15.990 INFO:teuthology.orchestra.run.smithi062.stdout:94489280516 2024-02-05T04:53:16.003 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280517 got 94489280516 for osd.15 2024-02-05T04:53:16.024 INFO:teuthology.orchestra.run.smithi062.stdout:90194313220 2024-02-05T04:53:16.037 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313221 got 90194313220 for osd.6 2024-02-05T04:53:16.067 INFO:teuthology.orchestra.run.smithi062.stdout:94489280516 2024-02-05T04:53:16.080 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.9 2024-02-05T04:53:16.080 DEBUG:teuthology.parallel:result is None 2024-02-05T04:53:17.004 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.15 2024-02-05T04:53:17.038 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-02-05T04:53:17.331 INFO:teuthology.orchestra.run.smithi062.stdout:94489280517 2024-02-05T04:53:17.343 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280517 got 94489280517 for osd.15 2024-02-05T04:53:17.344 DEBUG:teuthology.parallel:result is None 2024-02-05T04:53:17.362 INFO:teuthology.orchestra.run.smithi062.stdout:90194313221 2024-02-05T04:53:17.374 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313221 got 90194313221 for osd.6 2024-02-05T04:53:17.374 DEBUG:teuthology.parallel:result is None 2024-02-05T04:53:17.374 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-02-05T04:53:17.374 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-05T04:53:17.706 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:53:17.706 INFO:teuthology.orchestra.run.smithi062.stderr:dumped all 2024-02-05T04:53:17.718 INFO:teuthology.orchestra.run.smithi062.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-02-05T04:53:17.505464+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":438708,"kb_used_data":5440,"kb_used_omap":24,"kb_used_meta":432103,"kb_avail":1509510732,"statfs":{"total":1546188226560,"available":1545738989568,"internally_reserved":0,"allocated":5570560,"data_stored":2832061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25520,"internal_metadata":442473552},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3],"upper_bound":2},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001052"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-02-05T04:53:12.781774+0000","last_change":"2024-02-05T04:53:12.781269+0000","last_active":"2024-02-05T04:53:12.781774+0000","last_peered":"2024-02-05T04:53:12.781774+0000","last_clean":"2024-02-05T04:53:12.781774+0000","last_became_active":"2024-02-05T04:53:12.780970+0000","last_became_peered":"2024-02-05T04:53:12.780970+0000","last_unstale":"2024-02-05T04:53:12.781774+0000","last_undegraded":"2024-02-05T04:53:12.781774+0000","last_fullsized":"2024-02-05T04:53:12.781774+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_clean_scrub_stamp":"2024-02-05T04:53:11.017155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:28.333088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,0],"acting":[15,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-02-05T04:53:13.037228+0000","last_change":"2024-02-05T04:53:12.037950+0000","last_active":"2024-02-05T04:53:13.037228+0000","last_peered":"2024-02-05T04:53:13.037228+0000","last_clean":"2024-02-05T04:53:13.037228+0000","last_became_active":"2024-02-05T04:53:12.037750+0000","last_became_peered":"2024-02-05T04:53:12.037750+0000","last_unstale":"2024-02-05T04:53:13.037228+0000","last_undegraded":"2024-02-05T04:53:13.037228+0000","last_fullsized":"2024-02-05T04:53:13.037228+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_clean_scrub_stamp":"2024-02-05T04:53:11.017155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:26.129079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-02-05T04:53:13.100746+0000","last_change":"2024-02-05T04:53:12.753233+0000","last_active":"2024-02-05T04:53:13.100746+0000","last_peered":"2024-02-05T04:53:13.100746+0000","last_clean":"2024-02-05T04:53:13.100746+0000","last_became_active":"2024-02-05T04:53:12.752915+0000","last_became_peered":"2024-02-05T04:53:12.752915+0000","last_unstale":"2024-02-05T04:53:13.100746+0000","last_undegraded":"2024-02-05T04:53:13.100746+0000","last_fullsized":"2024-02-05T04:53:13.100746+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_clean_scrub_stamp":"2024-02-05T04:53:11.017155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:18.391150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-02-05T04:53:12.033277+0000","last_change":"2024-02-05T04:53:12.032853+0000","last_active":"2024-02-05T04:53:12.033277+0000","last_peered":"2024-02-05T04:53:12.033277+0000","last_clean":"2024-02-05T04:53:12.033277+0000","last_became_active":"2024-02-05T04:53:12.032664+0000","last_became_peered":"2024-02-05T04:53:12.032664+0000","last_unstale":"2024-02-05T04:53:12.033277+0000","last_undegraded":"2024-02-05T04:53:12.033277+0000","last_fullsized":"2024-02-05T04:53:12.033277+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_clean_scrub_stamp":"2024-02-05T04:53:11.017155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:19.614195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-02-05T04:53:13.175251+0000","last_change":"2024-02-05T04:53:12.035901+0000","last_active":"2024-02-05T04:53:13.175251+0000","last_peered":"2024-02-05T04:53:13.175251+0000","last_clean":"2024-02-05T04:53:13.175251+0000","last_became_active":"2024-02-05T04:53:12.035739+0000","last_became_peered":"2024-02-05T04:53:12.035739+0000","last_unstale":"2024-02-05T04:53:13.175251+0000","last_undegraded":"2024-02-05T04:53:13.175251+0000","last_fullsized":"2024-02-05T04:53:13.175251+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_clean_scrub_stamp":"2024-02-05T04:53:11.017155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:14.062467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9],"acting":[3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-02-05T04:53:13.043677+0000","last_change":"2024-02-05T04:53:12.033593+0000","last_active":"2024-02-05T04:53:13.043677+0000","last_peered":"2024-02-05T04:53:13.043677+0000","last_clean":"2024-02-05T04:53:13.043677+0000","last_became_active":"2024-02-05T04:53:12.033337+0000","last_became_peered":"2024-02-05T04:53:12.033337+0000","last_unstale":"2024-02-05T04:53:13.043677+0000","last_undegraded":"2024-02-05T04:53:13.043677+0000","last_fullsized":"2024-02-05T04:53:13.043677+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_clean_scrub_stamp":"2024-02-05T04:53:11.017155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:14.948579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4],"acting":[8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-02-05T04:53:13.175250+0000","last_change":"2024-02-05T04:53:12.040500+0000","last_active":"2024-02-05T04:53:13.175250+0000","last_peered":"2024-02-05T04:53:13.175250+0000","last_clean":"2024-02-05T04:53:13.175250+0000","last_became_active":"2024-02-05T04:53:12.040297+0000","last_became_peered":"2024-02-05T04:53:12.040297+0000","last_unstale":"2024-02-05T04:53:13.175250+0000","last_undegraded":"2024-02-05T04:53:13.175250+0000","last_fullsized":"2024-02-05T04:53:13.175250+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_clean_scrub_stamp":"2024-02-05T04:53:11.017155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:20.053075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-02-05T04:53:13.161338+0000","last_change":"2024-02-05T04:53:12.028466+0000","last_active":"2024-02-05T04:53:13.161338+0000","last_peered":"2024-02-05T04:53:13.161338+0000","last_clean":"2024-02-05T04:53:13.161338+0000","last_became_active":"2024-02-05T04:53:12.028138+0000","last_became_peered":"2024-02-05T04:53:12.028138+0000","last_unstale":"2024-02-05T04:53:13.161338+0000","last_undegraded":"2024-02-05T04:53:13.161338+0000","last_fullsized":"2024-02-05T04:53:13.161338+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_clean_scrub_stamp":"2024-02-05T04:53:11.017155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:38.427375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,15],"acting":[4,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"26'157","reported_seq":210,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-02-05T04:53:12.031874+0000","last_change":"2024-02-05T04:53:09.625781+0000","last_active":"2024-02-05T04:53:12.031874+0000","last_peered":"2024-02-05T04:53:12.031874+0000","last_clean":"2024-02-05T04:53:12.031874+0000","last_became_active":"2024-02-05T04:53:09.624695+0000","last_became_peered":"2024-02-05T04:53:09.624695+0000","last_unstale":"2024-02-05T04:53:12.031874+0000","last_undegraded":"2024-02-05T04:53:12.031874+0000","last_fullsized":"2024-02-05T04:53:12.031874+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:07.900741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:07.900741+0000","last_clean_scrub_stamp":"2024-02-05T04:53:07.900741+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:23.258506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2],"acting":[8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":12,"up_from":23,"seq":98784247812,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":118587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1659999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7689999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8859999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52700000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4970000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9099999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.292}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9350000000000001}]}]},{"osd":11,"up_from":23,"seq":98784247814,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27292,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":278528,"data_stored":109125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40799999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54000000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61399999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45300000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65000000000000002}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53200000000000003}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]}]},{"osd":10,"up_from":23,"seq":98784247814,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27260,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":245760,"data_stored":90132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.245}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78100000000000003}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71299999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.374}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69799999999999995}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66100000000000003}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.62}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76000000000000001}]}]},{"osd":3,"up_from":21,"seq":90194313222,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":118587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,9,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.43099999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64400000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53600000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40999999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77000000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56499999999999995}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70499999999999996}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81299999999999994}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73399999999999999}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69199999999999995}]}]},{"osd":4,"up_from":21,"seq":90194313221,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":118587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,5,6,7,8,9,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77300000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86799999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.9079999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.04}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90200000000000002}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.1699999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98899999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.948}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.073}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.903}]}]},{"osd":5,"up_from":21,"seq":90194313221,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":118587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,6,7,9,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46600000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69999999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73599999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65500000000000003}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79600000000000004}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75900000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91900000000000004}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.04}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97999999999999998}]}]},{"osd":6,"up_from":21,"seq":90194313222,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":118587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,3,4,5,7,8,9,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70699999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.776}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77700000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86599999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84899999999999998}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79100000000000004}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54600000000000004}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93400000000000005}]}]},{"osd":1,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":278528,"data_stored":109125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,4,5,6,9,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.498}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.442}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.622}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66200000000000003}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55300000000000005}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83399999999999996}]}]},{"osd":14,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27280,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":262144,"data_stored":99607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,3,4,5,6,8,9,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51700000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54000000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41999999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68400000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55500000000000005}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47799999999999998}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72599999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82899999999999996}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79600000000000004}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.872}]}]},{"osd":9,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":278528,"data_stored":109125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,3,4,5,6,8,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69999999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.627}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65900000000000003}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.749}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83399999999999996}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78500000000000003}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86799999999999999}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79600000000000004}]}]},{"osd":13,"up_from":22,"seq":94489280515,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27280,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":262144,"data_stored":99607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,3,4,5,6,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46999999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71499999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.498}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55200000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76100000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.823}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86299999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.996}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2150000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247813,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":118587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4259999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82499999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6759999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57899999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79000000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4929999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7549999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.723000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.622}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7150000000000001}]}]},{"osd":15,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":278528,"data_stored":109125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,6,9,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56599999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71999999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.377}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29199999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54800000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46700000000000003}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68200000000000005}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69999999999999996}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.495}]}]},{"osd":2,"up_from":23,"seq":98784247814,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28260,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343580,"statfs":{"total":96636764160,"available":96607825920,"internally_reserved":0,"allocated":811008,"data_stored":638053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,9,11,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40999999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.31900000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72699999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47899999999999998}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79800000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.026}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89600000000000002}]}]},{"osd":7,"up_from":23,"seq":98784247814,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27296,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":278528,"data_stored":109125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,2,6,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0800000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4630000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1100000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69999999999999996}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.028}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.224}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.137}]}]},{"osd":8,"up_from":23,"seq":98784247811,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28280,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343560,"statfs":{"total":96636764160,"available":96607805440,"internally_reserved":0,"allocated":827392,"data_stored":647515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,4,6,7,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40400000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34599999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.375}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79300000000000004}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40500000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51000000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.625}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77300000000000002}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69799999999999995}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71099999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T04:53:17.721 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-05T04:53:18.011 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:53:18.012 INFO:teuthology.orchestra.run.smithi062.stderr:dumped all 2024-02-05T04:53:18.024 INFO:teuthology.orchestra.run.smithi062.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-02-05T04:53:17.505464+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":438708,"kb_used_data":5440,"kb_used_omap":24,"kb_used_meta":432103,"kb_avail":1509510732,"statfs":{"total":1546188226560,"available":1545738989568,"internally_reserved":0,"allocated":5570560,"data_stored":2832061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25520,"internal_metadata":442473552},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3],"upper_bound":2},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001052"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-02-05T04:53:12.781774+0000","last_change":"2024-02-05T04:53:12.781269+0000","last_active":"2024-02-05T04:53:12.781774+0000","last_peered":"2024-02-05T04:53:12.781774+0000","last_clean":"2024-02-05T04:53:12.781774+0000","last_became_active":"2024-02-05T04:53:12.780970+0000","last_became_peered":"2024-02-05T04:53:12.780970+0000","last_unstale":"2024-02-05T04:53:12.781774+0000","last_undegraded":"2024-02-05T04:53:12.781774+0000","last_fullsized":"2024-02-05T04:53:12.781774+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_clean_scrub_stamp":"2024-02-05T04:53:11.017155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:28.333088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,0],"acting":[15,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-02-05T04:53:13.037228+0000","last_change":"2024-02-05T04:53:12.037950+0000","last_active":"2024-02-05T04:53:13.037228+0000","last_peered":"2024-02-05T04:53:13.037228+0000","last_clean":"2024-02-05T04:53:13.037228+0000","last_became_active":"2024-02-05T04:53:12.037750+0000","last_became_peered":"2024-02-05T04:53:12.037750+0000","last_unstale":"2024-02-05T04:53:13.037228+0000","last_undegraded":"2024-02-05T04:53:13.037228+0000","last_fullsized":"2024-02-05T04:53:13.037228+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_clean_scrub_stamp":"2024-02-05T04:53:11.017155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:26.129079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-02-05T04:53:13.100746+0000","last_change":"2024-02-05T04:53:12.753233+0000","last_active":"2024-02-05T04:53:13.100746+0000","last_peered":"2024-02-05T04:53:13.100746+0000","last_clean":"2024-02-05T04:53:13.100746+0000","last_became_active":"2024-02-05T04:53:12.752915+0000","last_became_peered":"2024-02-05T04:53:12.752915+0000","last_unstale":"2024-02-05T04:53:13.100746+0000","last_undegraded":"2024-02-05T04:53:13.100746+0000","last_fullsized":"2024-02-05T04:53:13.100746+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_clean_scrub_stamp":"2024-02-05T04:53:11.017155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:18.391150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-02-05T04:53:12.033277+0000","last_change":"2024-02-05T04:53:12.032853+0000","last_active":"2024-02-05T04:53:12.033277+0000","last_peered":"2024-02-05T04:53:12.033277+0000","last_clean":"2024-02-05T04:53:12.033277+0000","last_became_active":"2024-02-05T04:53:12.032664+0000","last_became_peered":"2024-02-05T04:53:12.032664+0000","last_unstale":"2024-02-05T04:53:12.033277+0000","last_undegraded":"2024-02-05T04:53:12.033277+0000","last_fullsized":"2024-02-05T04:53:12.033277+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_clean_scrub_stamp":"2024-02-05T04:53:11.017155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:19.614195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-02-05T04:53:13.175251+0000","last_change":"2024-02-05T04:53:12.035901+0000","last_active":"2024-02-05T04:53:13.175251+0000","last_peered":"2024-02-05T04:53:13.175251+0000","last_clean":"2024-02-05T04:53:13.175251+0000","last_became_active":"2024-02-05T04:53:12.035739+0000","last_became_peered":"2024-02-05T04:53:12.035739+0000","last_unstale":"2024-02-05T04:53:13.175251+0000","last_undegraded":"2024-02-05T04:53:13.175251+0000","last_fullsized":"2024-02-05T04:53:13.175251+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_clean_scrub_stamp":"2024-02-05T04:53:11.017155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:14.062467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9],"acting":[3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-02-05T04:53:13.043677+0000","last_change":"2024-02-05T04:53:12.033593+0000","last_active":"2024-02-05T04:53:13.043677+0000","last_peered":"2024-02-05T04:53:13.043677+0000","last_clean":"2024-02-05T04:53:13.043677+0000","last_became_active":"2024-02-05T04:53:12.033337+0000","last_became_peered":"2024-02-05T04:53:12.033337+0000","last_unstale":"2024-02-05T04:53:13.043677+0000","last_undegraded":"2024-02-05T04:53:13.043677+0000","last_fullsized":"2024-02-05T04:53:13.043677+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_clean_scrub_stamp":"2024-02-05T04:53:11.017155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:14.948579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4],"acting":[8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-02-05T04:53:13.175250+0000","last_change":"2024-02-05T04:53:12.040500+0000","last_active":"2024-02-05T04:53:13.175250+0000","last_peered":"2024-02-05T04:53:13.175250+0000","last_clean":"2024-02-05T04:53:13.175250+0000","last_became_active":"2024-02-05T04:53:12.040297+0000","last_became_peered":"2024-02-05T04:53:12.040297+0000","last_unstale":"2024-02-05T04:53:13.175250+0000","last_undegraded":"2024-02-05T04:53:13.175250+0000","last_fullsized":"2024-02-05T04:53:13.175250+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_clean_scrub_stamp":"2024-02-05T04:53:11.017155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:20.053075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-02-05T04:53:13.161338+0000","last_change":"2024-02-05T04:53:12.028466+0000","last_active":"2024-02-05T04:53:13.161338+0000","last_peered":"2024-02-05T04:53:13.161338+0000","last_clean":"2024-02-05T04:53:13.161338+0000","last_became_active":"2024-02-05T04:53:12.028138+0000","last_became_peered":"2024-02-05T04:53:12.028138+0000","last_unstale":"2024-02-05T04:53:13.161338+0000","last_undegraded":"2024-02-05T04:53:13.161338+0000","last_fullsized":"2024-02-05T04:53:13.161338+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_clean_scrub_stamp":"2024-02-05T04:53:11.017155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:38.427375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,15],"acting":[4,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"26'157","reported_seq":210,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-02-05T04:53:12.031874+0000","last_change":"2024-02-05T04:53:09.625781+0000","last_active":"2024-02-05T04:53:12.031874+0000","last_peered":"2024-02-05T04:53:12.031874+0000","last_clean":"2024-02-05T04:53:12.031874+0000","last_became_active":"2024-02-05T04:53:09.624695+0000","last_became_peered":"2024-02-05T04:53:09.624695+0000","last_unstale":"2024-02-05T04:53:12.031874+0000","last_undegraded":"2024-02-05T04:53:12.031874+0000","last_fullsized":"2024-02-05T04:53:12.031874+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:07.900741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:07.900741+0000","last_clean_scrub_stamp":"2024-02-05T04:53:07.900741+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:23.258506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2],"acting":[8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":12,"up_from":23,"seq":98784247812,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":118587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1659999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7689999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8859999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52700000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4970000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9099999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.292}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9350000000000001}]}]},{"osd":11,"up_from":23,"seq":98784247814,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27292,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":278528,"data_stored":109125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40799999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54000000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61399999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45300000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65000000000000002}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53200000000000003}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]}]},{"osd":10,"up_from":23,"seq":98784247814,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27260,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":245760,"data_stored":90132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.245}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78100000000000003}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71299999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.374}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69799999999999995}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66100000000000003}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.62}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76000000000000001}]}]},{"osd":3,"up_from":21,"seq":90194313222,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":118587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,9,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.43099999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64400000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53600000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40999999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77000000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56499999999999995}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70499999999999996}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81299999999999994}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73399999999999999}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69199999999999995}]}]},{"osd":4,"up_from":21,"seq":90194313221,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":118587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,5,6,7,8,9,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77300000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86799999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.9079999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.04}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90200000000000002}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.1699999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98899999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.948}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.073}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.903}]}]},{"osd":5,"up_from":21,"seq":90194313221,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":118587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,6,7,9,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46600000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69999999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73599999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65500000000000003}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79600000000000004}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75900000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91900000000000004}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.04}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97999999999999998}]}]},{"osd":6,"up_from":21,"seq":90194313222,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":118587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,3,4,5,7,8,9,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70699999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.776}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77700000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86599999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84899999999999998}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79100000000000004}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54600000000000004}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93400000000000005}]}]},{"osd":1,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":278528,"data_stored":109125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,4,5,6,9,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.498}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.442}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.622}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66200000000000003}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55300000000000005}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83399999999999996}]}]},{"osd":14,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27280,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":262144,"data_stored":99607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,3,4,5,6,8,9,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51700000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54000000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41999999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68400000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55500000000000005}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47799999999999998}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72599999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82899999999999996}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79600000000000004}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.872}]}]},{"osd":9,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":278528,"data_stored":109125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,3,4,5,6,8,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69999999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.627}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65900000000000003}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.749}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83399999999999996}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78500000000000003}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86799999999999999}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79600000000000004}]}]},{"osd":13,"up_from":22,"seq":94489280515,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27280,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":262144,"data_stored":99607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,3,4,5,6,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46999999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71499999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.498}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55200000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76100000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.823}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86299999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.996}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2150000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247813,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":118587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4259999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82499999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6759999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57899999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79000000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4929999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7549999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.723000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.622}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7150000000000001}]}]},{"osd":15,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":278528,"data_stored":109125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,6,9,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56599999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71999999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.377}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29199999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54800000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46700000000000003}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68200000000000005}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69999999999999996}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.495}]}]},{"osd":2,"up_from":23,"seq":98784247814,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28260,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343580,"statfs":{"total":96636764160,"available":96607825920,"internally_reserved":0,"allocated":811008,"data_stored":638053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,9,11,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40999999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.31900000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72699999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47899999999999998}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79800000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.026}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89600000000000002}]}]},{"osd":7,"up_from":23,"seq":98784247814,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27296,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":278528,"data_stored":109125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,2,6,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0800000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4630000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1100000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69999999999999996}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.028}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.224}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.137}]}]},{"osd":8,"up_from":23,"seq":98784247811,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28280,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343560,"statfs":{"total":96636764160,"available":96607805440,"internally_reserved":0,"allocated":827392,"data_stored":647515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,4,6,7,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40400000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34599999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.375}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79300000000000004}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40500000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51000000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.625}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77300000000000002}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69799999999999995}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71099999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T04:53:18.025 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-02-05T04:53:18.026 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-02-05T04:53:18.026 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-02-05T04:53:18.026 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-02-05T04:53:18.355 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:53:18.355 INFO:teuthology.orchestra.run.smithi062.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-02-05T04:53:18.368 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-02-05T04:53:18.368 INFO:teuthology.run_tasks:Running task thrashosds... 2024-02-05T04:53:18.378 INFO:tasks.thrashosds:config is {'bluestore_new_sharding': 'random', 'chance_bluestore_reshard': 1, 'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'min_in': 8, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-02-05T04:53:18.379 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-nitzan-max-pglog-size'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject socket failures': 2500, 'ms tcp read timeout': 5, 'osd_async_recovery_min_cost': 1, 'osd_object_clean_region_max_num_intervals': 1000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.1, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from'], 'sha1': 'd4ed20cfb238ef2c47d64ca4533eca00b8968b98'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'd4ed20cfb238ef2c47d64ca4533eca00b8968b98'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-nitzan-max-pglog-size', 'sha1': 'd4ed20cfb238ef2c47d64ca4533eca00b8968b98'}} 2024-02-05T04:53:18.379 INFO:tasks.thrashosds:config is {'bluestore_new_sharding': 'random', 'chance_bluestore_reshard': 1, 'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'min_in': 8, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0, 'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5} 2024-02-05T04:53:18.379 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-02-05T04:53:18.690 INFO:tasks.thrashosds:Beginning thrashosds... 2024-02-05T04:53:18.691 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-02-05T04:53:18.691 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-05T04:53:18.983 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:53:18.984 INFO:teuthology.orchestra.run.smithi062.stderr:dumped all 2024-02-05T04:53:18.995 INFO:teuthology.orchestra.run.smithi062.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-02-05T04:53:17.505464+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":438708,"kb_used_data":5440,"kb_used_omap":24,"kb_used_meta":432103,"kb_avail":1509510732,"statfs":{"total":1546188226560,"available":1545738989568,"internally_reserved":0,"allocated":5570560,"data_stored":2832061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25520,"internal_metadata":442473552},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3],"upper_bound":2},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001052"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-02-05T04:53:12.781774+0000","last_change":"2024-02-05T04:53:12.781269+0000","last_active":"2024-02-05T04:53:12.781774+0000","last_peered":"2024-02-05T04:53:12.781774+0000","last_clean":"2024-02-05T04:53:12.781774+0000","last_became_active":"2024-02-05T04:53:12.780970+0000","last_became_peered":"2024-02-05T04:53:12.780970+0000","last_unstale":"2024-02-05T04:53:12.781774+0000","last_undegraded":"2024-02-05T04:53:12.781774+0000","last_fullsized":"2024-02-05T04:53:12.781774+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_clean_scrub_stamp":"2024-02-05T04:53:11.017155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:28.333088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,0],"acting":[15,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-02-05T04:53:13.037228+0000","last_change":"2024-02-05T04:53:12.037950+0000","last_active":"2024-02-05T04:53:13.037228+0000","last_peered":"2024-02-05T04:53:13.037228+0000","last_clean":"2024-02-05T04:53:13.037228+0000","last_became_active":"2024-02-05T04:53:12.037750+0000","last_became_peered":"2024-02-05T04:53:12.037750+0000","last_unstale":"2024-02-05T04:53:13.037228+0000","last_undegraded":"2024-02-05T04:53:13.037228+0000","last_fullsized":"2024-02-05T04:53:13.037228+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_clean_scrub_stamp":"2024-02-05T04:53:11.017155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:26.129079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-02-05T04:53:13.100746+0000","last_change":"2024-02-05T04:53:12.753233+0000","last_active":"2024-02-05T04:53:13.100746+0000","last_peered":"2024-02-05T04:53:13.100746+0000","last_clean":"2024-02-05T04:53:13.100746+0000","last_became_active":"2024-02-05T04:53:12.752915+0000","last_became_peered":"2024-02-05T04:53:12.752915+0000","last_unstale":"2024-02-05T04:53:13.100746+0000","last_undegraded":"2024-02-05T04:53:13.100746+0000","last_fullsized":"2024-02-05T04:53:13.100746+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_clean_scrub_stamp":"2024-02-05T04:53:11.017155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:18.391150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-02-05T04:53:12.033277+0000","last_change":"2024-02-05T04:53:12.032853+0000","last_active":"2024-02-05T04:53:12.033277+0000","last_peered":"2024-02-05T04:53:12.033277+0000","last_clean":"2024-02-05T04:53:12.033277+0000","last_became_active":"2024-02-05T04:53:12.032664+0000","last_became_peered":"2024-02-05T04:53:12.032664+0000","last_unstale":"2024-02-05T04:53:12.033277+0000","last_undegraded":"2024-02-05T04:53:12.033277+0000","last_fullsized":"2024-02-05T04:53:12.033277+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_clean_scrub_stamp":"2024-02-05T04:53:11.017155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:19.614195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-02-05T04:53:13.175251+0000","last_change":"2024-02-05T04:53:12.035901+0000","last_active":"2024-02-05T04:53:13.175251+0000","last_peered":"2024-02-05T04:53:13.175251+0000","last_clean":"2024-02-05T04:53:13.175251+0000","last_became_active":"2024-02-05T04:53:12.035739+0000","last_became_peered":"2024-02-05T04:53:12.035739+0000","last_unstale":"2024-02-05T04:53:13.175251+0000","last_undegraded":"2024-02-05T04:53:13.175251+0000","last_fullsized":"2024-02-05T04:53:13.175251+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_clean_scrub_stamp":"2024-02-05T04:53:11.017155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:14.062467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9],"acting":[3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-02-05T04:53:13.043677+0000","last_change":"2024-02-05T04:53:12.033593+0000","last_active":"2024-02-05T04:53:13.043677+0000","last_peered":"2024-02-05T04:53:13.043677+0000","last_clean":"2024-02-05T04:53:13.043677+0000","last_became_active":"2024-02-05T04:53:12.033337+0000","last_became_peered":"2024-02-05T04:53:12.033337+0000","last_unstale":"2024-02-05T04:53:13.043677+0000","last_undegraded":"2024-02-05T04:53:13.043677+0000","last_fullsized":"2024-02-05T04:53:13.043677+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_clean_scrub_stamp":"2024-02-05T04:53:11.017155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:14.948579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4],"acting":[8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-02-05T04:53:13.175250+0000","last_change":"2024-02-05T04:53:12.040500+0000","last_active":"2024-02-05T04:53:13.175250+0000","last_peered":"2024-02-05T04:53:13.175250+0000","last_clean":"2024-02-05T04:53:13.175250+0000","last_became_active":"2024-02-05T04:53:12.040297+0000","last_became_peered":"2024-02-05T04:53:12.040297+0000","last_unstale":"2024-02-05T04:53:13.175250+0000","last_undegraded":"2024-02-05T04:53:13.175250+0000","last_fullsized":"2024-02-05T04:53:13.175250+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_clean_scrub_stamp":"2024-02-05T04:53:11.017155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:20.053075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-02-05T04:53:13.161338+0000","last_change":"2024-02-05T04:53:12.028466+0000","last_active":"2024-02-05T04:53:13.161338+0000","last_peered":"2024-02-05T04:53:13.161338+0000","last_clean":"2024-02-05T04:53:13.161338+0000","last_became_active":"2024-02-05T04:53:12.028138+0000","last_became_peered":"2024-02-05T04:53:12.028138+0000","last_unstale":"2024-02-05T04:53:13.161338+0000","last_undegraded":"2024-02-05T04:53:13.161338+0000","last_fullsized":"2024-02-05T04:53:13.161338+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_clean_scrub_stamp":"2024-02-05T04:53:11.017155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:38.427375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,15],"acting":[4,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"26'157","reported_seq":210,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-02-05T04:53:12.031874+0000","last_change":"2024-02-05T04:53:09.625781+0000","last_active":"2024-02-05T04:53:12.031874+0000","last_peered":"2024-02-05T04:53:12.031874+0000","last_clean":"2024-02-05T04:53:12.031874+0000","last_became_active":"2024-02-05T04:53:09.624695+0000","last_became_peered":"2024-02-05T04:53:09.624695+0000","last_unstale":"2024-02-05T04:53:12.031874+0000","last_undegraded":"2024-02-05T04:53:12.031874+0000","last_fullsized":"2024-02-05T04:53:12.031874+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:07.900741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:07.900741+0000","last_clean_scrub_stamp":"2024-02-05T04:53:07.900741+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:23.258506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2],"acting":[8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":12,"up_from":23,"seq":98784247812,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":118587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1659999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7689999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8859999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52700000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4970000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9099999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.292}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9350000000000001}]}]},{"osd":11,"up_from":23,"seq":98784247814,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27292,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":278528,"data_stored":109125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40799999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54000000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61399999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45300000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65000000000000002}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53200000000000003}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]}]},{"osd":10,"up_from":23,"seq":98784247814,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27260,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":245760,"data_stored":90132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.245}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78100000000000003}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71299999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.374}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69799999999999995}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66100000000000003}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.62}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76000000000000001}]}]},{"osd":3,"up_from":21,"seq":90194313222,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":118587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,9,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.43099999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64400000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53600000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40999999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77000000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56499999999999995}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70499999999999996}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81299999999999994}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73399999999999999}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69199999999999995}]}]},{"osd":4,"up_from":21,"seq":90194313221,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":118587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,5,6,7,8,9,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77300000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86799999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.9079999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.04}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90200000000000002}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.1699999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98899999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.948}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.073}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.903}]}]},{"osd":5,"up_from":21,"seq":90194313221,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":118587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,6,7,9,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46600000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69999999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73599999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65500000000000003}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79600000000000004}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75900000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91900000000000004}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.04}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97999999999999998}]}]},{"osd":6,"up_from":21,"seq":90194313222,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":118587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,3,4,5,7,8,9,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70699999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.776}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77700000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86599999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84899999999999998}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79100000000000004}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54600000000000004}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93400000000000005}]}]},{"osd":1,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":278528,"data_stored":109125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,4,5,6,9,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.498}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.442}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.622}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66200000000000003}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55300000000000005}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83399999999999996}]}]},{"osd":14,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27280,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":262144,"data_stored":99607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,3,4,5,6,8,9,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51700000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54000000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41999999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68400000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55500000000000005}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47799999999999998}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72599999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82899999999999996}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79600000000000004}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.872}]}]},{"osd":9,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":278528,"data_stored":109125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,3,4,5,6,8,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69999999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.627}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65900000000000003}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.749}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83399999999999996}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78500000000000003}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86799999999999999}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79600000000000004}]}]},{"osd":13,"up_from":22,"seq":94489280515,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27280,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":262144,"data_stored":99607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,3,4,5,6,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46999999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71499999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.498}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55200000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76100000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.823}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86299999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.996}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2150000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247813,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":118587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4259999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82499999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6759999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57899999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79000000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4929999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7549999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.723000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.622}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7150000000000001}]}]},{"osd":15,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":278528,"data_stored":109125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,6,9,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56599999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71999999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.377}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29199999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54800000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46700000000000003}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68200000000000005}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69999999999999996}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.495}]}]},{"osd":2,"up_from":23,"seq":98784247814,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28260,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343580,"statfs":{"total":96636764160,"available":96607825920,"internally_reserved":0,"allocated":811008,"data_stored":638053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,9,11,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40999999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.31900000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72699999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47899999999999998}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79800000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.026}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89600000000000002}]}]},{"osd":7,"up_from":23,"seq":98784247814,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27296,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":278528,"data_stored":109125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,2,6,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0800000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4630000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1100000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69999999999999996}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.028}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.224}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.137}]}]},{"osd":8,"up_from":23,"seq":98784247811,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28280,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343560,"statfs":{"total":96636764160,"available":96607805440,"internally_reserved":0,"allocated":827392,"data_stored":647515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,4,6,7,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40400000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34599999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.375}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79300000000000004}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40500000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51000000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.625}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77300000000000002}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69799999999999995}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71099999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T04:53:18.997 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-05T04:53:19.287 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:53:19.288 INFO:teuthology.orchestra.run.smithi062.stderr:dumped all 2024-02-05T04:53:19.299 INFO:teuthology.orchestra.run.smithi062.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-02-05T04:53:17.505464+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":438708,"kb_used_data":5440,"kb_used_omap":24,"kb_used_meta":432103,"kb_avail":1509510732,"statfs":{"total":1546188226560,"available":1545738989568,"internally_reserved":0,"allocated":5570560,"data_stored":2832061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25520,"internal_metadata":442473552},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3],"upper_bound":2},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001052"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-02-05T04:53:12.781774+0000","last_change":"2024-02-05T04:53:12.781269+0000","last_active":"2024-02-05T04:53:12.781774+0000","last_peered":"2024-02-05T04:53:12.781774+0000","last_clean":"2024-02-05T04:53:12.781774+0000","last_became_active":"2024-02-05T04:53:12.780970+0000","last_became_peered":"2024-02-05T04:53:12.780970+0000","last_unstale":"2024-02-05T04:53:12.781774+0000","last_undegraded":"2024-02-05T04:53:12.781774+0000","last_fullsized":"2024-02-05T04:53:12.781774+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_clean_scrub_stamp":"2024-02-05T04:53:11.017155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:28.333088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,0],"acting":[15,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-02-05T04:53:13.037228+0000","last_change":"2024-02-05T04:53:12.037950+0000","last_active":"2024-02-05T04:53:13.037228+0000","last_peered":"2024-02-05T04:53:13.037228+0000","last_clean":"2024-02-05T04:53:13.037228+0000","last_became_active":"2024-02-05T04:53:12.037750+0000","last_became_peered":"2024-02-05T04:53:12.037750+0000","last_unstale":"2024-02-05T04:53:13.037228+0000","last_undegraded":"2024-02-05T04:53:13.037228+0000","last_fullsized":"2024-02-05T04:53:13.037228+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_clean_scrub_stamp":"2024-02-05T04:53:11.017155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:26.129079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-02-05T04:53:13.100746+0000","last_change":"2024-02-05T04:53:12.753233+0000","last_active":"2024-02-05T04:53:13.100746+0000","last_peered":"2024-02-05T04:53:13.100746+0000","last_clean":"2024-02-05T04:53:13.100746+0000","last_became_active":"2024-02-05T04:53:12.752915+0000","last_became_peered":"2024-02-05T04:53:12.752915+0000","last_unstale":"2024-02-05T04:53:13.100746+0000","last_undegraded":"2024-02-05T04:53:13.100746+0000","last_fullsized":"2024-02-05T04:53:13.100746+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_clean_scrub_stamp":"2024-02-05T04:53:11.017155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:18.391150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-02-05T04:53:12.033277+0000","last_change":"2024-02-05T04:53:12.032853+0000","last_active":"2024-02-05T04:53:12.033277+0000","last_peered":"2024-02-05T04:53:12.033277+0000","last_clean":"2024-02-05T04:53:12.033277+0000","last_became_active":"2024-02-05T04:53:12.032664+0000","last_became_peered":"2024-02-05T04:53:12.032664+0000","last_unstale":"2024-02-05T04:53:12.033277+0000","last_undegraded":"2024-02-05T04:53:12.033277+0000","last_fullsized":"2024-02-05T04:53:12.033277+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_clean_scrub_stamp":"2024-02-05T04:53:11.017155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:19.614195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-02-05T04:53:13.175251+0000","last_change":"2024-02-05T04:53:12.035901+0000","last_active":"2024-02-05T04:53:13.175251+0000","last_peered":"2024-02-05T04:53:13.175251+0000","last_clean":"2024-02-05T04:53:13.175251+0000","last_became_active":"2024-02-05T04:53:12.035739+0000","last_became_peered":"2024-02-05T04:53:12.035739+0000","last_unstale":"2024-02-05T04:53:13.175251+0000","last_undegraded":"2024-02-05T04:53:13.175251+0000","last_fullsized":"2024-02-05T04:53:13.175251+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_clean_scrub_stamp":"2024-02-05T04:53:11.017155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:14.062467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9],"acting":[3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-02-05T04:53:13.043677+0000","last_change":"2024-02-05T04:53:12.033593+0000","last_active":"2024-02-05T04:53:13.043677+0000","last_peered":"2024-02-05T04:53:13.043677+0000","last_clean":"2024-02-05T04:53:13.043677+0000","last_became_active":"2024-02-05T04:53:12.033337+0000","last_became_peered":"2024-02-05T04:53:12.033337+0000","last_unstale":"2024-02-05T04:53:13.043677+0000","last_undegraded":"2024-02-05T04:53:13.043677+0000","last_fullsized":"2024-02-05T04:53:13.043677+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_clean_scrub_stamp":"2024-02-05T04:53:11.017155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:14.948579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4],"acting":[8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-02-05T04:53:13.175250+0000","last_change":"2024-02-05T04:53:12.040500+0000","last_active":"2024-02-05T04:53:13.175250+0000","last_peered":"2024-02-05T04:53:13.175250+0000","last_clean":"2024-02-05T04:53:13.175250+0000","last_became_active":"2024-02-05T04:53:12.040297+0000","last_became_peered":"2024-02-05T04:53:12.040297+0000","last_unstale":"2024-02-05T04:53:13.175250+0000","last_undegraded":"2024-02-05T04:53:13.175250+0000","last_fullsized":"2024-02-05T04:53:13.175250+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_clean_scrub_stamp":"2024-02-05T04:53:11.017155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:20.053075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-02-05T04:53:13.161338+0000","last_change":"2024-02-05T04:53:12.028466+0000","last_active":"2024-02-05T04:53:13.161338+0000","last_peered":"2024-02-05T04:53:13.161338+0000","last_clean":"2024-02-05T04:53:13.161338+0000","last_became_active":"2024-02-05T04:53:12.028138+0000","last_became_peered":"2024-02-05T04:53:12.028138+0000","last_unstale":"2024-02-05T04:53:13.161338+0000","last_undegraded":"2024-02-05T04:53:13.161338+0000","last_fullsized":"2024-02-05T04:53:13.161338+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_clean_scrub_stamp":"2024-02-05T04:53:11.017155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:38.427375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,15],"acting":[4,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"26'157","reported_seq":210,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-02-05T04:53:12.031874+0000","last_change":"2024-02-05T04:53:09.625781+0000","last_active":"2024-02-05T04:53:12.031874+0000","last_peered":"2024-02-05T04:53:12.031874+0000","last_clean":"2024-02-05T04:53:12.031874+0000","last_became_active":"2024-02-05T04:53:09.624695+0000","last_became_peered":"2024-02-05T04:53:09.624695+0000","last_unstale":"2024-02-05T04:53:12.031874+0000","last_undegraded":"2024-02-05T04:53:12.031874+0000","last_fullsized":"2024-02-05T04:53:12.031874+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:07.900741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:07.900741+0000","last_clean_scrub_stamp":"2024-02-05T04:53:07.900741+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:23.258506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2],"acting":[8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":12,"up_from":23,"seq":98784247812,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":118587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1659999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7689999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8859999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52700000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4970000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9099999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.292}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9350000000000001}]}]},{"osd":11,"up_from":23,"seq":98784247814,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27292,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":278528,"data_stored":109125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40799999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54000000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61399999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45300000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65000000000000002}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53200000000000003}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]}]},{"osd":10,"up_from":23,"seq":98784247814,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27260,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":245760,"data_stored":90132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.245}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78100000000000003}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71299999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.374}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69799999999999995}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66100000000000003}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.62}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76000000000000001}]}]},{"osd":3,"up_from":21,"seq":90194313222,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":118587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,9,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.43099999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64400000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53600000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40999999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77000000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56499999999999995}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70499999999999996}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81299999999999994}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73399999999999999}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69199999999999995}]}]},{"osd":4,"up_from":21,"seq":90194313221,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":118587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,5,6,7,8,9,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77300000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86799999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.9079999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.04}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90200000000000002}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.1699999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98899999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.948}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.073}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.903}]}]},{"osd":5,"up_from":21,"seq":90194313221,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":118587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,6,7,9,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46600000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69999999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73599999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65500000000000003}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79600000000000004}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75900000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91900000000000004}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.04}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97999999999999998}]}]},{"osd":6,"up_from":21,"seq":90194313222,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":118587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,3,4,5,7,8,9,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70699999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.776}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77700000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86599999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84899999999999998}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79100000000000004}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54600000000000004}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93400000000000005}]}]},{"osd":1,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":278528,"data_stored":109125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,4,5,6,9,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.498}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.442}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.622}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66200000000000003}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55300000000000005}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83399999999999996}]}]},{"osd":14,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27280,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":262144,"data_stored":99607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,3,4,5,6,8,9,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51700000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54000000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41999999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68400000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55500000000000005}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47799999999999998}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72599999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82899999999999996}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79600000000000004}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.872}]}]},{"osd":9,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":278528,"data_stored":109125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,3,4,5,6,8,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69999999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.627}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65900000000000003}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.749}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83399999999999996}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78500000000000003}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86799999999999999}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79600000000000004}]}]},{"osd":13,"up_from":22,"seq":94489280515,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27280,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":262144,"data_stored":99607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,3,4,5,6,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46999999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71499999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.498}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55200000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76100000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.823}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86299999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.996}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2150000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247813,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":118587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4259999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82499999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6759999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57899999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79000000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4929999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7549999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.723000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.622}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7150000000000001}]}]},{"osd":15,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":278528,"data_stored":109125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,6,9,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56599999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71999999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.377}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29199999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54800000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46700000000000003}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68200000000000005}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69999999999999996}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.495}]}]},{"osd":2,"up_from":23,"seq":98784247814,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28260,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343580,"statfs":{"total":96636764160,"available":96607825920,"internally_reserved":0,"allocated":811008,"data_stored":638053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,9,11,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40999999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.31900000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72699999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47899999999999998}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79800000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.026}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89600000000000002}]}]},{"osd":7,"up_from":23,"seq":98784247814,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27296,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":278528,"data_stored":109125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,2,6,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0800000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4630000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1100000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69999999999999996}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.028}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.224}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.137}]}]},{"osd":8,"up_from":23,"seq":98784247811,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28280,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343560,"statfs":{"total":96636764160,"available":96607805440,"internally_reserved":0,"allocated":827392,"data_stored":647515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,4,6,7,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40400000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34599999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.375}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79300000000000004}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40500000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51000000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.625}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77300000000000002}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69799999999999995}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71099999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T04:53:19.301 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-02-05T04:53:19.302 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2024-02-05T04:53:19.593 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 28 2024-02-05T04:53:19.593 INFO:teuthology.orchestra.run.smithi062.stdout:fsid 88c11a2d-382e-4f21-b8af-45ab9c504b93 2024-02-05T04:53:19.593 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-02-05T04:52:55.182961+0000 2024-02-05T04:53:19.593 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-02-05T04:53:13.024046+0000 2024-02-05T04:53:19.593 INFO:teuthology.orchestra.run.smithi062.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-02-05T04:53:19.594 INFO:teuthology.orchestra.run.smithi062.stdout:crush_version 5 2024-02-05T04:53:19.594 INFO:teuthology.orchestra.run.smithi062.stdout:full_ratio 0.95 2024-02-05T04:53:19.594 INFO:teuthology.orchestra.run.smithi062.stdout:backfillfull_ratio 0.9 2024-02-05T04:53:19.594 INFO:teuthology.orchestra.run.smithi062.stdout:nearfull_ratio 0.85 2024-02-05T04:53:19.594 INFO:teuthology.orchestra.run.smithi062.stdout:require_min_compat_client luminous 2024-02-05T04:53:19.594 INFO:teuthology.orchestra.run.smithi062.stdout:min_compat_client jewel 2024-02-05T04:53:19.594 INFO:teuthology.orchestra.run.smithi062.stdout:require_osd_release squid 2024-02-05T04:53:19.594 INFO:teuthology.orchestra.run.smithi062.stdout:stretch_mode_enabled false 2024-02-05T04:53:19.594 INFO:teuthology.orchestra.run.smithi062.stdout:pool 1 '.mgr' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 25 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 15.38 2024-02-05T04:53:19.594 INFO:teuthology.orchestra.run.smithi062.stdout:pool 2 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 28 flags hashpspool stripe_width 0 application rbd read_balance_score 4.02 2024-02-05T04:53:19.594 INFO:teuthology.orchestra.run.smithi062.stdout:max_osd 16 2024-02-05T04:53:19.594 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.62:6814/1189094240,v1:172.21.15.62:6816/1189094240] [v2:172.21.15.62:6818/1189094240,v1:172.21.15.62:6819/1189094240] exists,up cdb503c5-a32c-4a9a-b1d9-194ab5ee48aa 2024-02-05T04:53:19.594 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 up in weight 1 up_from 22 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:172.21.15.91:6808/2878185190,v1:172.21.15.91:6811/2878185190] [v2:172.21.15.91:6814/2878185190,v1:172.21.15.91:6818/2878185190] exists,up d7a3d394-74b2-4cb1-a754-4fe77e605ea9 2024-02-05T04:53:19.594 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.150:6823/40232889,v1:172.21.15.150:6825/40232889] [v2:172.21.15.150:6826/40232889,v1:172.21.15.150:6827/40232889] exists,up d6e99efe-475f-41c1-88a3-87ce6b26ce75 2024-02-05T04:53:19.595 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 up in weight 1 up_from 21 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:172.21.15.196:6802/3834523862,v1:172.21.15.196:6803/3834523862] [v2:172.21.15.196:6804/3834523862,v1:172.21.15.196:6805/3834523862] exists,up 3c964090-5e07-40bd-ac7c-b66e29eb25b4 2024-02-05T04:53:19.595 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 up in weight 1 up_from 21 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:172.21.15.62:6800/2244900897,v1:172.21.15.62:6801/2244900897] [v2:172.21.15.62:6802/2244900897,v1:172.21.15.62:6803/2244900897] exists,up 22bd25f9-3d93-4bcb-bcfc-71feae1097f0 2024-02-05T04:53:19.595 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 up in weight 1 up_from 21 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.91:6800/473596866,v1:172.21.15.91:6801/473596866] [v2:172.21.15.91:6802/473596866,v1:172.21.15.91:6803/473596866] exists,up c6ae2bd4-4a4f-4f23-8991-743ffa42ed3c 2024-02-05T04:53:19.595 INFO:teuthology.orchestra.run.smithi062.stdout:osd.6 up in weight 1 up_from 21 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:172.21.15.150:6800/2199648645,v1:172.21.15.150:6801/2199648645] [v2:172.21.15.150:6802/2199648645,v1:172.21.15.150:6803/2199648645] exists,up 49da8b05-ffa3-4172-983e-f081d3eb9ae6 2024-02-05T04:53:19.595 INFO:teuthology.orchestra.run.smithi062.stdout:osd.7 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.196:6810/254726044,v1:172.21.15.196:6811/254726044] [v2:172.21.15.196:6812/254726044,v1:172.21.15.196:6813/254726044] exists,up 4ece26e1-2c63-4223-baaf-882cd8261ade 2024-02-05T04:53:19.595 INFO:teuthology.orchestra.run.smithi062.stdout:osd.8 up in weight 1 up_from 23 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:172.21.15.62:6808/3461147157,v1:172.21.15.62:6809/3461147157] [v2:172.21.15.62:6810/3461147157,v1:172.21.15.62:6811/3461147157] exists,up 9ac666e9-291e-4696-a6bb-e10c9deef3f5 2024-02-05T04:53:19.595 INFO:teuthology.orchestra.run.smithi062.stdout:osd.9 up in weight 1 up_from 22 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:172.21.15.91:6810/1859221562,v1:172.21.15.91:6813/1859221562] [v2:172.21.15.91:6816/1859221562,v1:172.21.15.91:6820/1859221562] exists,up 7fe04cb8-ac67-46ac-b2de-1f8f60a50373 2024-02-05T04:53:19.595 INFO:teuthology.orchestra.run.smithi062.stdout:osd.10 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.150:6808/2448697550,v1:172.21.15.150:6809/2448697550] [v2:172.21.15.150:6810/2448697550,v1:172.21.15.150:6811/2448697550] exists,up a6b0ac25-da07-4fee-b646-a4530ce5b343 2024-02-05T04:53:19.595 INFO:teuthology.orchestra.run.smithi062.stdout:osd.11 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.196:6815/1170435852,v1:172.21.15.196:6817/1170435852] [v2:172.21.15.196:6819/1170435852,v1:172.21.15.196:6821/1170435852] exists,up 31b22258-77da-495c-b518-9ce0297b9165 2024-02-05T04:53:19.595 INFO:teuthology.orchestra.run.smithi062.stdout:osd.12 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.62:6824/1173398345,v1:172.21.15.62:6825/1173398345] [v2:172.21.15.62:6826/1173398345,v1:172.21.15.62:6827/1173398345] exists,up 322917eb-26ba-4439-8300-4d723bada623 2024-02-05T04:53:19.595 INFO:teuthology.orchestra.run.smithi062.stdout:osd.13 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.91:6809/3656255795,v1:172.21.15.91:6812/3656255795] [v2:172.21.15.91:6815/3656255795,v1:172.21.15.91:6817/3656255795] exists,up 9c708044-4726-42b6-9c13-03963f956d2d 2024-02-05T04:53:19.596 INFO:teuthology.orchestra.run.smithi062.stdout:osd.14 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.150:6812/2102158830,v1:172.21.15.150:6814/2102158830] [v2:172.21.15.150:6816/2102158830,v1:172.21.15.150:6818/2102158830] exists,up 0208144a-db9a-404c-b31e-13bc2e276b09 2024-02-05T04:53:19.596 INFO:teuthology.orchestra.run.smithi062.stdout:osd.15 up in weight 1 up_from 22 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:172.21.15.196:6826/3187639683,v1:172.21.15.196:6827/3187639683] [v2:172.21.15.196:6828/3187639683,v1:172.21.15.196:6829/3187639683] exists,up 0591e7e1-dc07-4d80-b574-9b8348635764 2024-02-05T04:53:19.605 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.62:6814/1189094240,v1:172.21.15.62:6816/1189094240] [v2:172.21.15.62:6818/1189094240,v1:172.21.15.62:6819/1189094240] exists,up cdb503c5-a32c-4a9a-b1d9-194ab5ee48aa', 'osd.1 up in weight 1 up_from 22 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:172.21.15.91:6808/2878185190,v1:172.21.15.91:6811/2878185190] [v2:172.21.15.91:6814/2878185190,v1:172.21.15.91:6818/2878185190] exists,up d7a3d394-74b2-4cb1-a754-4fe77e605ea9', 'osd.2 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.150:6823/40232889,v1:172.21.15.150:6825/40232889] [v2:172.21.15.150:6826/40232889,v1:172.21.15.150:6827/40232889] exists,up d6e99efe-475f-41c1-88a3-87ce6b26ce75', 'osd.3 up in weight 1 up_from 21 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:172.21.15.196:6802/3834523862,v1:172.21.15.196:6803/3834523862] [v2:172.21.15.196:6804/3834523862,v1:172.21.15.196:6805/3834523862] exists,up 3c964090-5e07-40bd-ac7c-b66e29eb25b4', 'osd.4 up in weight 1 up_from 21 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:172.21.15.62:6800/2244900897,v1:172.21.15.62:6801/2244900897] [v2:172.21.15.62:6802/2244900897,v1:172.21.15.62:6803/2244900897] exists,up 22bd25f9-3d93-4bcb-bcfc-71feae1097f0', 'osd.5 up in weight 1 up_from 21 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.91:6800/473596866,v1:172.21.15.91:6801/473596866] [v2:172.21.15.91:6802/473596866,v1:172.21.15.91:6803/473596866] exists,up c6ae2bd4-4a4f-4f23-8991-743ffa42ed3c', 'osd.6 up in weight 1 up_from 21 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:172.21.15.150:6800/2199648645,v1:172.21.15.150:6801/2199648645] [v2:172.21.15.150:6802/2199648645,v1:172.21.15.150:6803/2199648645] exists,up 49da8b05-ffa3-4172-983e-f081d3eb9ae6', 'osd.7 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.196:6810/254726044,v1:172.21.15.196:6811/254726044] [v2:172.21.15.196:6812/254726044,v1:172.21.15.196:6813/254726044] exists,up 4ece26e1-2c63-4223-baaf-882cd8261ade', 'osd.8 up in weight 1 up_from 23 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:172.21.15.62:6808/3461147157,v1:172.21.15.62:6809/3461147157] [v2:172.21.15.62:6810/3461147157,v1:172.21.15.62:6811/3461147157] exists,up 9ac666e9-291e-4696-a6bb-e10c9deef3f5', 'osd.9 up in weight 1 up_from 22 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:172.21.15.91:6810/1859221562,v1:172.21.15.91:6813/1859221562] [v2:172.21.15.91:6816/1859221562,v1:172.21.15.91:6820/1859221562] exists,up 7fe04cb8-ac67-46ac-b2de-1f8f60a50373', 'osd.10 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.150:6808/2448697550,v1:172.21.15.150:6809/2448697550] [v2:172.21.15.150:6810/2448697550,v1:172.21.15.150:6811/2448697550] exists,up a6b0ac25-da07-4fee-b646-a4530ce5b343', 'osd.11 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.196:6815/1170435852,v1:172.21.15.196:6817/1170435852] [v2:172.21.15.196:6819/1170435852,v1:172.21.15.196:6821/1170435852] exists,up 31b22258-77da-495c-b518-9ce0297b9165', 'osd.12 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.62:6824/1173398345,v1:172.21.15.62:6825/1173398345] [v2:172.21.15.62:6826/1173398345,v1:172.21.15.62:6827/1173398345] exists,up 322917eb-26ba-4439-8300-4d723bada623', 'osd.13 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.91:6809/3656255795,v1:172.21.15.91:6812/3656255795] [v2:172.21.15.91:6815/3656255795,v1:172.21.15.91:6817/3656255795] exists,up 9c708044-4726-42b6-9c13-03963f956d2d', 'osd.14 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.150:6812/2102158830,v1:172.21.15.150:6814/2102158830] [v2:172.21.15.150:6816/2102158830,v1:172.21.15.150:6818/2102158830] exists,up 0208144a-db9a-404c-b31e-13bc2e276b09', 'osd.15 up in weight 1 up_from 22 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:172.21.15.196:6826/3187639683,v1:172.21.15.196:6827/3187639683] [v2:172.21.15.196:6828/3187639683,v1:172.21.15.196:6829/3187639683] exists,up 0591e7e1-dc07-4d80-b574-9b8348635764'] 2024-02-05T04:53:19.606 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2024-02-05T04:53:19.707 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-02-05T04:53:19.708 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "mon.a", 2024-02-05T04:53:19.708 INFO:teuthology.orchestra.run.smithi062.stdout: "cluster": "ceph", 2024-02-05T04:53:19.708 INFO:teuthology.orchestra.run.smithi062.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-02-05T04:53:19.708 INFO:teuthology.orchestra.run.smithi062.stdout: "admin_socket_mode": "", 2024-02-05T04:53:19.708 INFO:teuthology.orchestra.run.smithi062.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-02-05T04:53:19.708 INFO:teuthology.orchestra.run.smithi062.stdout: "auth_client_required": "cephx, none", 2024-02-05T04:53:19.708 INFO:teuthology.orchestra.run.smithi062.stdout: "auth_cluster_required": "cephx", 2024-02-05T04:53:19.708 INFO:teuthology.orchestra.run.smithi062.stdout: "auth_debug": "true", 2024-02-05T04:53:19.708 INFO:teuthology.orchestra.run.smithi062.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-02-05T04:53:19.708 INFO:teuthology.orchestra.run.smithi062.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-02-05T04:53:19.708 INFO:teuthology.orchestra.run.smithi062.stdout: "auth_service_required": "cephx", 2024-02-05T04:53:19.709 INFO:teuthology.orchestra.run.smithi062.stdout: "auth_service_ticket_ttl": "240.000000", 2024-02-05T04:53:19.709 INFO:teuthology.orchestra.run.smithi062.stdout: "auth_supported": "cephx", 2024-02-05T04:53:19.709 INFO:teuthology.orchestra.run.smithi062.stdout: "bdev_aio": "true", 2024-02-05T04:53:19.709 INFO:teuthology.orchestra.run.smithi062.stdout: "bdev_aio_max_queue_depth": "1024", 2024-02-05T04:53:19.709 INFO:teuthology.orchestra.run.smithi062.stdout: "bdev_aio_poll_ms": "250", 2024-02-05T04:53:19.709 INFO:teuthology.orchestra.run.smithi062.stdout: "bdev_aio_reap_max": "16", 2024-02-05T04:53:19.709 INFO:teuthology.orchestra.run.smithi062.stdout: "bdev_async_discard": "false", 2024-02-05T04:53:19.709 INFO:teuthology.orchestra.run.smithi062.stdout: "bdev_block_size": "4096", 2024-02-05T04:53:19.709 INFO:teuthology.orchestra.run.smithi062.stdout: "bdev_debug_aio": "false", 2024-02-05T04:53:19.709 INFO:teuthology.orchestra.run.smithi062.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-02-05T04:53:19.709 INFO:teuthology.orchestra.run.smithi062.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-02-05T04:53:19.709 INFO:teuthology.orchestra.run.smithi062.stdout: "bdev_debug_inflight_ios": "false", 2024-02-05T04:53:19.709 INFO:teuthology.orchestra.run.smithi062.stdout: "bdev_enable_discard": "false", 2024-02-05T04:53:19.709 INFO:teuthology.orchestra.run.smithi062.stdout: "bdev_flock_retry": "3", 2024-02-05T04:53:19.709 INFO:teuthology.orchestra.run.smithi062.stdout: "bdev_flock_retry_interval": "0.100000", 2024-02-05T04:53:19.710 INFO:teuthology.orchestra.run.smithi062.stdout: "bdev_inject_crash": "0", 2024-02-05T04:53:19.710 INFO:teuthology.orchestra.run.smithi062.stdout: "bdev_inject_crash_flush_delay": "2", 2024-02-05T04:53:19.710 INFO:teuthology.orchestra.run.smithi062.stdout: "bdev_ioring": "false", 2024-02-05T04:53:19.710 INFO:teuthology.orchestra.run.smithi062.stdout: "bdev_ioring_hipri": "false", 2024-02-05T04:53:19.710 INFO:teuthology.orchestra.run.smithi062.stdout: "bdev_ioring_sqthread_poll": "false", 2024-02-05T04:53:19.710 INFO:teuthology.orchestra.run.smithi062.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-02-05T04:53:19.710 INFO:teuthology.orchestra.run.smithi062.stdout: "bdev_read_buffer_alignment": "4096", 2024-02-05T04:53:19.710 INFO:teuthology.orchestra.run.smithi062.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-02-05T04:53:19.710 INFO:teuthology.orchestra.run.smithi062.stdout: "bdev_type": "", 2024-02-05T04:53:19.710 INFO:teuthology.orchestra.run.smithi062.stdout: "bluefs_alloc_size": "1048576", 2024-02-05T04:53:19.710 INFO:teuthology.orchestra.run.smithi062.stdout: "bluefs_allocator": "hybrid", 2024-02-05T04:53:19.710 INFO:teuthology.orchestra.run.smithi062.stdout: "bluefs_buffered_io": "true", 2024-02-05T04:53:19.710 INFO:teuthology.orchestra.run.smithi062.stdout: "bluefs_check_for_zeros": "false", 2024-02-05T04:53:19.710 INFO:teuthology.orchestra.run.smithi062.stdout: "bluefs_check_volume_selector_often": "false", 2024-02-05T04:53:19.711 INFO:teuthology.orchestra.run.smithi062.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-02-05T04:53:19.711 INFO:teuthology.orchestra.run.smithi062.stdout: "bluefs_compact_log_sync": "false", 2024-02-05T04:53:19.711 INFO:teuthology.orchestra.run.smithi062.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-02-05T04:53:19.711 INFO:teuthology.orchestra.run.smithi062.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-02-05T04:53:19.711 INFO:teuthology.orchestra.run.smithi062.stdout: "bluefs_log_compact_min_size": "16777216", 2024-02-05T04:53:19.711 INFO:teuthology.orchestra.run.smithi062.stdout: "bluefs_log_replay_check_allocations": "true", 2024-02-05T04:53:19.711 INFO:teuthology.orchestra.run.smithi062.stdout: "bluefs_max_log_runway": "4194304", 2024-02-05T04:53:19.711 INFO:teuthology.orchestra.run.smithi062.stdout: "bluefs_max_prefetch": "1048576", 2024-02-05T04:53:19.711 INFO:teuthology.orchestra.run.smithi062.stdout: "bluefs_min_flush_size": "524288", 2024-02-05T04:53:19.711 INFO:teuthology.orchestra.run.smithi062.stdout: "bluefs_min_log_runway": "1048576", 2024-02-05T04:53:19.711 INFO:teuthology.orchestra.run.smithi062.stdout: "bluefs_replay_recovery": "false", 2024-02-05T04:53:19.711 INFO:teuthology.orchestra.run.smithi062.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-02-05T04:53:19.711 INFO:teuthology.orchestra.run.smithi062.stdout: "bluefs_shared_alloc_size": "65536", 2024-02-05T04:53:19.712 INFO:teuthology.orchestra.run.smithi062.stdout: "bluefs_sync_write": "false", 2024-02-05T04:53:19.712 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-02-05T04:53:19.712 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-02-05T04:53:19.712 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-02-05T04:53:19.712 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_allocation_from_file": "true", 2024-02-05T04:53:19.712 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_allocator": "hybrid", 2024-02-05T04:53:19.712 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-02-05T04:53:19.712 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-02-05T04:53:19.712 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-02-05T04:53:19.712 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-02-05T04:53:19.712 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-02-05T04:53:19.712 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-02-05T04:53:19.712 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_blobid_prealloc": "10240", 2024-02-05T04:53:19.713 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_block_create": "true", 2024-02-05T04:53:19.713 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_block_db_create": "false", 2024-02-05T04:53:19.713 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_block_db_path": "", 2024-02-05T04:53:19.713 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_block_db_size": "0", 2024-02-05T04:53:19.713 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_block_path": "", 2024-02-05T04:53:19.713 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_block_preallocate_file": "false", 2024-02-05T04:53:19.713 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_block_size": "107374182400", 2024-02-05T04:53:19.713 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_block_wal_create": "false", 2024-02-05T04:53:19.713 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_block_wal_path": "", 2024-02-05T04:53:19.713 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_block_wal_size": "100663296", 2024-02-05T04:53:19.713 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_bluefs": "true", 2024-02-05T04:53:19.713 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-02-05T04:53:19.713 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_bluefs_env_mirror": "false", 2024-02-05T04:53:19.713 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-02-05T04:53:19.714 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-02-05T04:53:19.714 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-02-05T04:53:19.714 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-02-05T04:53:19.714 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-02-05T04:53:19.714 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-02-05T04:53:19.714 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_autotune": "true", 2024-02-05T04:53:19.714 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-02-05T04:53:19.714 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-02-05T04:53:19.714 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-02-05T04:53:19.714 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-02-05T04:53:19.714 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_size": "0", 2024-02-05T04:53:19.714 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-02-05T04:53:19.714 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-02-05T04:53:19.714 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-02-05T04:53:19.715 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-02-05T04:53:19.715 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_type": "2q", 2024-02-05T04:53:19.715 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-02-05T04:53:19.715 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_clone_cow": "true", 2024-02-05T04:53:19.715 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_compression_algorithm": "snappy", 2024-02-05T04:53:19.715 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_compression_max_blob_size": "0", 2024-02-05T04:53:19.715 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-02-05T04:53:19.715 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-02-05T04:53:19.715 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_compression_min_blob_size": "0", 2024-02-05T04:53:19.715 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-02-05T04:53:19.715 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-02-05T04:53:19.715 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_compression_mode": "none", 2024-02-05T04:53:19.715 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-02-05T04:53:19.716 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_csum_type": "crc32c", 2024-02-05T04:53:19.716 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_debug_enforce_settings": "default", 2024-02-05T04:53:19.716 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_debug_freelist": "false", 2024-02-05T04:53:19.716 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_debug_fsck_abort": "false", 2024-02-05T04:53:19.716 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-02-05T04:53:19.716 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-02-05T04:53:19.716 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_debug_inject_read_err": "false", 2024-02-05T04:53:19.716 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_debug_legacy_omap": "false", 2024-02-05T04:53:19.716 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-02-05T04:53:19.716 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-02-05T04:53:19.716 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-02-05T04:53:19.716 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-02-05T04:53:19.716 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_debug_prefill": "0.000000", 2024-02-05T04:53:19.716 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-02-05T04:53:19.716 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-02-05T04:53:19.717 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-02-05T04:53:19.717 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_debug_small_allocations": "0", 2024-02-05T04:53:19.717 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-02-05T04:53:19.717 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_default_buffered_read": "true", 2024-02-05T04:53:19.717 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_default_buffered_write": "false", 2024-02-05T04:53:19.717 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_deferred_batch_ops": "0", 2024-02-05T04:53:19.717 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-02-05T04:53:19.717 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-02-05T04:53:19.717 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_elastic_shared_blobs": "true", 2024-02-05T04:53:19.717 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-02-05T04:53:19.717 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-02-05T04:53:19.717 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-02-05T04:53:19.717 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-02-05T04:53:19.717 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-02-05T04:53:19.717 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_fail_eio": "false", 2024-02-05T04:53:19.718 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-02-05T04:53:19.718 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-02-05T04:53:19.718 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-02-05T04:53:19.718 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-02-05T04:53:19.718 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_fsck_on_mkfs": "true", 2024-02-05T04:53:19.718 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-02-05T04:53:19.718 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_fsck_on_mount": "false", 2024-02-05T04:53:19.718 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-02-05T04:53:19.718 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_fsck_on_umount": "false", 2024-02-05T04:53:19.718 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-02-05T04:53:19.718 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-02-05T04:53:19.718 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-02-05T04:53:19.718 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-02-05T04:53:19.718 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-02-05T04:53:19.718 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-02-05T04:53:19.719 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-02-05T04:53:19.719 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-02-05T04:53:19.719 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_ignore_data_csum": "false", 2024-02-05T04:53:19.719 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-02-05T04:53:19.719 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_kvbackend": "rocksdb", 2024-02-05T04:53:19.719 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-02-05T04:53:19.719 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-02-05T04:53:19.719 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_log_op_age": "5.000000", 2024-02-05T04:53:19.719 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_max_alloc_size": "0", 2024-02-05T04:53:19.719 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_max_blob_size": "0", 2024-02-05T04:53:19.719 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-02-05T04:53:19.719 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-02-05T04:53:19.719 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_max_defer_interval": "3.000000", 2024-02-05T04:53:19.719 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_max_deferred_txc": "32", 2024-02-05T04:53:19.720 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_min_alloc_size": "0", 2024-02-05T04:53:19.720 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-02-05T04:53:19.720 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-02-05T04:53:19.720 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_nid_prealloc": "1024", 2024-02-05T04:53:19.720 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_prefer_deferred_size": "0", 2024-02-05T04:53:19.720 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-02-05T04:53:19.720 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-02-05T04:53:19.720 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_qfsck_on_mount": "true", 2024-02-05T04:53:19.720 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_retry_disk_reads": "3", 2024-02-05T04:53:19.720 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_rocksdb_cf": "true", 2024-02-05T04:53:19.720 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-02-05T04:53:19.720 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-02-05T04:53:19.720 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_rocksdb_options_annex": "", 2024-02-05T04:53:19.720 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_spdk_coremask": "0x1", 2024-02-05T04:53:19.721 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_spdk_io_sleep": "5", 2024-02-05T04:53:19.721 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_spdk_max_io_completion": "0", 2024-02-05T04:53:19.721 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_spdk_mem": "512", 2024-02-05T04:53:19.721 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_sync_submit_transaction": "false", 2024-02-05T04:53:19.721 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_throttle_bytes": "67108864", 2024-02-05T04:53:19.721 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_throttle_cost_per_io": "0", 2024-02-05T04:53:19.721 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-02-05T04:53:19.721 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-02-05T04:53:19.721 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-02-05T04:53:19.721 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-02-05T04:53:19.721 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_tracing": "false", 2024-02-05T04:53:19.721 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-02-05T04:53:19.721 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-02-05T04:53:19.721 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_volume_selection_reserved": "0", 2024-02-05T04:53:19.722 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-02-05T04:53:19.722 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-02-05T04:53:19.722 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-02-05T04:53:19.722 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-02-05T04:53:19.722 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-02-05T04:53:19.722 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-02-05T04:53:19.722 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_zero_block_detection": "false", 2024-02-05T04:53:19.722 INFO:teuthology.orchestra.run.smithi062.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-02-05T04:53:19.722 INFO:teuthology.orchestra.run.smithi062.stdout: "cephfs_mirror_action_update_interval": "2", 2024-02-05T04:53:19.722 INFO:teuthology.orchestra.run.smithi062.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-02-05T04:53:19.722 INFO:teuthology.orchestra.run.smithi062.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-02-05T04:53:19.722 INFO:teuthology.orchestra.run.smithi062.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-02-05T04:53:19.722 INFO:teuthology.orchestra.run.smithi062.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-02-05T04:53:19.722 INFO:teuthology.orchestra.run.smithi062.stdout: "cephfs_mirror_mount_timeout": "10", 2024-02-05T04:53:19.722 INFO:teuthology.orchestra.run.smithi062.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-02-05T04:53:19.723 INFO:teuthology.orchestra.run.smithi062.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-02-05T04:53:19.723 INFO:teuthology.orchestra.run.smithi062.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-02-05T04:53:19.723 INFO:teuthology.orchestra.run.smithi062.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-02-05T04:53:19.723 INFO:teuthology.orchestra.run.smithi062.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-02-05T04:53:19.723 INFO:teuthology.orchestra.run.smithi062.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-02-05T04:53:19.723 INFO:teuthology.orchestra.run.smithi062.stdout: "cephx_cluster_require_signatures": "false", 2024-02-05T04:53:19.723 INFO:teuthology.orchestra.run.smithi062.stdout: "cephx_cluster_require_version": "2", 2024-02-05T04:53:19.723 INFO:teuthology.orchestra.run.smithi062.stdout: "cephx_require_signatures": "false", 2024-02-05T04:53:19.723 INFO:teuthology.orchestra.run.smithi062.stdout: "cephx_require_version": "2", 2024-02-05T04:53:19.723 INFO:teuthology.orchestra.run.smithi062.stdout: "cephx_service_require_signatures": "false", 2024-02-05T04:53:19.723 INFO:teuthology.orchestra.run.smithi062.stdout: "cephx_service_require_version": "2", 2024-02-05T04:53:19.723 INFO:teuthology.orchestra.run.smithi062.stdout: "cephx_sign_messages": "true", 2024-02-05T04:53:19.723 INFO:teuthology.orchestra.run.smithi062.stdout: "chdir": "", 2024-02-05T04:53:19.723 INFO:teuthology.orchestra.run.smithi062.stdout: "client_acl_type": "", 2024-02-05T04:53:19.724 INFO:teuthology.orchestra.run.smithi062.stdout: "client_asio_thread_count": "2", 2024-02-05T04:53:19.724 INFO:teuthology.orchestra.run.smithi062.stdout: "client_cache_mid": "0.750000", 2024-02-05T04:53:19.724 INFO:teuthology.orchestra.run.smithi062.stdout: "client_cache_size": "16384", 2024-02-05T04:53:19.724 INFO:teuthology.orchestra.run.smithi062.stdout: "client_caps_release_delay": "5", 2024-02-05T04:53:19.724 INFO:teuthology.orchestra.run.smithi062.stdout: "client_check_pool_perm": "true", 2024-02-05T04:53:19.724 INFO:teuthology.orchestra.run.smithi062.stdout: "client_collect_and_send_global_metrics": "false", 2024-02-05T04:53:19.724 INFO:teuthology.orchestra.run.smithi062.stdout: "client_debug_force_sync_read": "false", 2024-02-05T04:53:19.724 INFO:teuthology.orchestra.run.smithi062.stdout: "client_debug_getattr_caps": "false", 2024-02-05T04:53:19.724 INFO:teuthology.orchestra.run.smithi062.stdout: "client_debug_inject_tick_delay": "0", 2024-02-05T04:53:19.724 INFO:teuthology.orchestra.run.smithi062.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-02-05T04:53:19.724 INFO:teuthology.orchestra.run.smithi062.stdout: "client_die_on_failed_remount": "false", 2024-02-05T04:53:19.724 INFO:teuthology.orchestra.run.smithi062.stdout: "client_dirsize_rbytes": "true", 2024-02-05T04:53:19.724 INFO:teuthology.orchestra.run.smithi062.stdout: "client_force_lazyio": "false", 2024-02-05T04:53:19.725 INFO:teuthology.orchestra.run.smithi062.stdout: "client_fs": "", 2024-02-05T04:53:19.725 INFO:teuthology.orchestra.run.smithi062.stdout: "client_inject_fixed_oldest_tid": "false", 2024-02-05T04:53:19.725 INFO:teuthology.orchestra.run.smithi062.stdout: "client_inject_release_failure": "false", 2024-02-05T04:53:19.725 INFO:teuthology.orchestra.run.smithi062.stdout: "client_max_inline_size": "4096", 2024-02-05T04:53:19.725 INFO:teuthology.orchestra.run.smithi062.stdout: "client_max_retries_on_remount_failure": "5", 2024-02-05T04:53:19.725 INFO:teuthology.orchestra.run.smithi062.stdout: "client_mds_namespace": "", 2024-02-05T04:53:19.725 INFO:teuthology.orchestra.run.smithi062.stdout: "client_metadata": "", 2024-02-05T04:53:19.725 INFO:teuthology.orchestra.run.smithi062.stdout: "client_mount_gid": "-1", 2024-02-05T04:53:19.725 INFO:teuthology.orchestra.run.smithi062.stdout: "client_mount_timeout": "300", 2024-02-05T04:53:19.725 INFO:teuthology.orchestra.run.smithi062.stdout: "client_mount_uid": "-1", 2024-02-05T04:53:19.725 INFO:teuthology.orchestra.run.smithi062.stdout: "client_mountpoint": "/", 2024-02-05T04:53:19.725 INFO:teuthology.orchestra.run.smithi062.stdout: "client_notify_timeout": "10", 2024-02-05T04:53:19.725 INFO:teuthology.orchestra.run.smithi062.stdout: "client_oc": "true", 2024-02-05T04:53:19.726 INFO:teuthology.orchestra.run.smithi062.stdout: "client_oc_max_dirty": "104857600", 2024-02-05T04:53:19.726 INFO:teuthology.orchestra.run.smithi062.stdout: "client_oc_max_dirty_age": "5.000000", 2024-02-05T04:53:19.726 INFO:teuthology.orchestra.run.smithi062.stdout: "client_oc_max_objects": "1000", 2024-02-05T04:53:19.726 INFO:teuthology.orchestra.run.smithi062.stdout: "client_oc_size": "209715200", 2024-02-05T04:53:19.726 INFO:teuthology.orchestra.run.smithi062.stdout: "client_oc_target_dirty": "8388608", 2024-02-05T04:53:19.726 INFO:teuthology.orchestra.run.smithi062.stdout: "client_permissions": "true", 2024-02-05T04:53:19.726 INFO:teuthology.orchestra.run.smithi062.stdout: "client_quota": "true", 2024-02-05T04:53:19.726 INFO:teuthology.orchestra.run.smithi062.stdout: "client_quota_df": "true", 2024-02-05T04:53:19.726 INFO:teuthology.orchestra.run.smithi062.stdout: "client_readahead_max_bytes": "0", 2024-02-05T04:53:19.726 INFO:teuthology.orchestra.run.smithi062.stdout: "client_readahead_max_periods": "4", 2024-02-05T04:53:19.726 INFO:teuthology.orchestra.run.smithi062.stdout: "client_readahead_min": "131072", 2024-02-05T04:53:19.726 INFO:teuthology.orchestra.run.smithi062.stdout: "client_reconnect_stale": "false", 2024-02-05T04:53:19.726 INFO:teuthology.orchestra.run.smithi062.stdout: "client_shutdown_timeout": "30", 2024-02-05T04:53:19.726 INFO:teuthology.orchestra.run.smithi062.stdout: "client_snapdir": ".snap", 2024-02-05T04:53:19.726 INFO:teuthology.orchestra.run.smithi062.stdout: "client_tick_interval": "1", 2024-02-05T04:53:19.726 INFO:teuthology.orchestra.run.smithi062.stdout: "client_trace": "", 2024-02-05T04:53:19.727 INFO:teuthology.orchestra.run.smithi062.stdout: "client_try_dentry_invalidate": "false", 2024-02-05T04:53:19.727 INFO:teuthology.orchestra.run.smithi062.stdout: "client_use_faked_inos": "false", 2024-02-05T04:53:19.727 INFO:teuthology.orchestra.run.smithi062.stdout: "client_use_random_mds": "false", 2024-02-05T04:53:19.727 INFO:teuthology.orchestra.run.smithi062.stdout: "clog_to_graylog": "false", 2024-02-05T04:53:19.727 INFO:teuthology.orchestra.run.smithi062.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-02-05T04:53:19.727 INFO:teuthology.orchestra.run.smithi062.stdout: "clog_to_graylog_port": "12201", 2024-02-05T04:53:19.727 INFO:teuthology.orchestra.run.smithi062.stdout: "clog_to_monitors": "default=true", 2024-02-05T04:53:19.727 INFO:teuthology.orchestra.run.smithi062.stdout: "clog_to_syslog": "false", 2024-02-05T04:53:19.727 INFO:teuthology.orchestra.run.smithi062.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-02-05T04:53:19.727 INFO:teuthology.orchestra.run.smithi062.stdout: "clog_to_syslog_level": "info", 2024-02-05T04:53:19.727 INFO:teuthology.orchestra.run.smithi062.stdout: "cluster_addr": "-", 2024-02-05T04:53:19.727 INFO:teuthology.orchestra.run.smithi062.stdout: "cluster_network": "", 2024-02-05T04:53:19.727 INFO:teuthology.orchestra.run.smithi062.stdout: "cluster_network_interface": "", 2024-02-05T04:53:19.727 INFO:teuthology.orchestra.run.smithi062.stdout: "compressor_zlib_isal": "false", 2024-02-05T04:53:19.728 INFO:teuthology.orchestra.run.smithi062.stdout: "compressor_zlib_level": "5", 2024-02-05T04:53:19.728 INFO:teuthology.orchestra.run.smithi062.stdout: "compressor_zlib_winsize": "-15", 2024-02-05T04:53:19.728 INFO:teuthology.orchestra.run.smithi062.stdout: "compressor_zstd_level": "1", 2024-02-05T04:53:19.728 INFO:teuthology.orchestra.run.smithi062.stdout: "container_image": "docker.io/ceph/daemon-base:latest-master-devel", 2024-02-05T04:53:19.728 INFO:teuthology.orchestra.run.smithi062.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-02-05T04:53:19.728 INFO:teuthology.orchestra.run.smithi062.stdout: "crimson_alien_op_num_threads": "6", 2024-02-05T04:53:19.728 INFO:teuthology.orchestra.run.smithi062.stdout: "crimson_alien_thread_cpu_cores": "", 2024-02-05T04:53:19.728 INFO:teuthology.orchestra.run.smithi062.stdout: "crimson_osd_obc_lru_size": "10", 2024-02-05T04:53:19.728 INFO:teuthology.orchestra.run.smithi062.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-02-05T04:53:19.728 INFO:teuthology.orchestra.run.smithi062.stdout: "crimson_seastar_smp": "1", 2024-02-05T04:53:19.728 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_location": "", 2024-02-05T04:53:19.728 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_location_hook": "", 2024-02-05T04:53:19.728 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_location_hook_timeout": "10", 2024-02-05T04:53:19.728 INFO:teuthology.orchestra.run.smithi062.stdout: "daemonize": "false", 2024-02-05T04:53:19.729 INFO:teuthology.orchestra.run.smithi062.stdout: "daos_pool": "tank", 2024-02-05T04:53:19.729 INFO:teuthology.orchestra.run.smithi062.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-02-05T04:53:19.729 INFO:teuthology.orchestra.run.smithi062.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-02-05T04:53:19.729 INFO:teuthology.orchestra.run.smithi062.stdout: "dbstore_db_name_prefix": "dbstore", 2024-02-05T04:53:19.729 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_alienstore": "0/5", 2024-02-05T04:53:19.729 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_allow_any_pool_priority": "false", 2024-02-05T04:53:19.729 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_asok": "1/5", 2024-02-05T04:53:19.729 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_asok_assert_abort": "false", 2024-02-05T04:53:19.729 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_asserts_on_shutdown": "true", 2024-02-05T04:53:19.729 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_auth": "20/20", 2024-02-05T04:53:19.729 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_bdev": "1/3", 2024-02-05T04:53:19.729 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_bluefs": "1/5", 2024-02-05T04:53:19.729 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_bluestore": "1/5", 2024-02-05T04:53:19.729 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_buffer": "0/1", 2024-02-05T04:53:19.730 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_ceph_exporter": "1/5", 2024-02-05T04:53:19.730 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_cephfs_mirror": "0/5", 2024-02-05T04:53:19.730 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_cephsqlite": "0/5", 2024-02-05T04:53:19.730 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_client": "0/5", 2024-02-05T04:53:19.730 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_compressor": "1/5", 2024-02-05T04:53:19.730 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_context": "0/1", 2024-02-05T04:53:19.730 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_crimson_interrupt": "0/5", 2024-02-05T04:53:19.730 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_crush": "1/1", 2024-02-05T04:53:19.730 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_crypto": "1/5", 2024-02-05T04:53:19.730 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_cyanstore": "0/5", 2024-02-05T04:53:19.730 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_deliberately_leak_memory": "false", 2024-02-05T04:53:19.730 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_disable_randomized_ping": "false", 2024-02-05T04:53:19.730 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_dpdk": "1/5", 2024-02-05T04:53:19.730 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_eventtrace": "1/5", 2024-02-05T04:53:19.730 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_filer": "0/1", 2024-02-05T04:53:19.730 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_filestore": "1/3", 2024-02-05T04:53:19.730 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_finisher": "1/1", 2024-02-05T04:53:19.731 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_fuse": "1/5", 2024-02-05T04:53:19.731 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_heartbeat_testing_span": "0", 2024-02-05T04:53:19.731 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_heartbeatmap": "1/5", 2024-02-05T04:53:19.731 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_immutable_obj_cache": "0/5", 2024-02-05T04:53:19.731 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_javaclient": "1/5", 2024-02-05T04:53:19.731 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_journal": "1/3", 2024-02-05T04:53:19.731 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_journaler": "0/5", 2024-02-05T04:53:19.731 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_kstore": "1/5", 2024-02-05T04:53:19.731 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_lockdep": "0/1", 2024-02-05T04:53:19.731 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_mclock": "1/5", 2024-02-05T04:53:19.731 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_mds": "1/5", 2024-02-05T04:53:19.731 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_mds_balancer": "1/5", 2024-02-05T04:53:19.731 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_mds_locker": "1/5", 2024-02-05T04:53:19.731 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_mds_log": "1/5", 2024-02-05T04:53:19.731 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_mds_log_expire": "1/5", 2024-02-05T04:53:19.732 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_mds_migrator": "1/5", 2024-02-05T04:53:19.732 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_memstore": "1/5", 2024-02-05T04:53:19.732 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_mgr": "2/5", 2024-02-05T04:53:19.732 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_mgrc": "1/5", 2024-02-05T04:53:19.732 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_mon": "20/20", 2024-02-05T04:53:19.732 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_monc": "0/10", 2024-02-05T04:53:19.732 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_ms": "1/1", 2024-02-05T04:53:19.732 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_none": "0/5", 2024-02-05T04:53:19.732 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_objclass": "0/5", 2024-02-05T04:53:19.732 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_objectcacher": "0/5", 2024-02-05T04:53:19.732 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_objecter": "0/1", 2024-02-05T04:53:19.732 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_optracker": "0/5", 2024-02-05T04:53:19.732 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_osd": "1/5", 2024-02-05T04:53:19.732 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_paxos": "20/20", 2024-02-05T04:53:19.733 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_perfcounter": "1/5", 2024-02-05T04:53:19.733 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_prioritycache": "1/5", 2024-02-05T04:53:19.733 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_rados": "0/5", 2024-02-05T04:53:19.733 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_rbd": "0/5", 2024-02-05T04:53:19.733 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_rbd_mirror": "0/5", 2024-02-05T04:53:19.733 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_rbd_pwl": "0/5", 2024-02-05T04:53:19.733 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_rbd_replay": "0/5", 2024-02-05T04:53:19.733 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_refs": "0/0", 2024-02-05T04:53:19.733 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_reserver": "1/1", 2024-02-05T04:53:19.733 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_rgw": "1/5", 2024-02-05T04:53:19.733 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_rgw_access": "1/5", 2024-02-05T04:53:19.733 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_rgw_datacache": "1/5", 2024-02-05T04:53:19.733 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_rgw_dbstore": "1/5", 2024-02-05T04:53:19.733 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_rgw_flight": "1/5", 2024-02-05T04:53:19.733 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_rgw_sync": "1/5", 2024-02-05T04:53:19.734 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_rocksdb": "4/5", 2024-02-05T04:53:19.734 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_seastore": "0/5", 2024-02-05T04:53:19.734 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_seastore_backref": "0/5", 2024-02-05T04:53:19.734 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_seastore_cache": "0/5", 2024-02-05T04:53:19.734 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_seastore_cleaner": "0/5", 2024-02-05T04:53:19.734 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_seastore_device": "0/5", 2024-02-05T04:53:19.734 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_seastore_epm": "0/5", 2024-02-05T04:53:19.734 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-02-05T04:53:19.734 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_seastore_journal": "0/5", 2024-02-05T04:53:19.734 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_seastore_lba": "0/5", 2024-02-05T04:53:19.734 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_seastore_odata": "0/5", 2024-02-05T04:53:19.734 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_seastore_omap": "0/5", 2024-02-05T04:53:19.734 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_seastore_onode": "0/5", 2024-02-05T04:53:19.735 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_seastore_t": "0/5", 2024-02-05T04:53:19.735 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_seastore_tm": "0/5", 2024-02-05T04:53:19.735 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_striper": "0/1", 2024-02-05T04:53:19.735 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_test": "0/5", 2024-02-05T04:53:19.735 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_throttle": "1/1", 2024-02-05T04:53:19.735 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_timer": "0/1", 2024-02-05T04:53:19.735 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_tp": "0/5", 2024-02-05T04:53:19.735 INFO:teuthology.orchestra.run.smithi062.stdout: "debug_trace": "1/5", 2024-02-05T04:53:19.735 INFO:teuthology.orchestra.run.smithi062.stdout: "defer_client_eviction_on_laggy_osds": "true", 2024-02-05T04:53:19.735 INFO:teuthology.orchestra.run.smithi062.stdout: "device_failure_prediction_mode": "none", 2024-02-05T04:53:19.735 INFO:teuthology.orchestra.run.smithi062.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-02-05T04:53:19.735 INFO:teuthology.orchestra.run.smithi062.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-02-05T04:53:19.735 INFO:teuthology.orchestra.run.smithi062.stdout: "err_to_graylog": "false", 2024-02-05T04:53:19.736 INFO:teuthology.orchestra.run.smithi062.stdout: "err_to_journald": "false", 2024-02-05T04:53:19.736 INFO:teuthology.orchestra.run.smithi062.stdout: "err_to_stderr": "true", 2024-02-05T04:53:19.736 INFO:teuthology.orchestra.run.smithi062.stdout: "err_to_syslog": "false", 2024-02-05T04:53:19.736 INFO:teuthology.orchestra.run.smithi062.stdout: "event_tracing": "false", 2024-02-05T04:53:19.736 INFO:teuthology.orchestra.run.smithi062.stdout: "exporter_addr": "0.0.0.0", 2024-02-05T04:53:19.736 INFO:teuthology.orchestra.run.smithi062.stdout: "exporter_http_port": "9926", 2024-02-05T04:53:19.736 INFO:teuthology.orchestra.run.smithi062.stdout: "exporter_prio_limit": "5", 2024-02-05T04:53:19.736 INFO:teuthology.orchestra.run.smithi062.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-02-05T04:53:19.736 INFO:teuthology.orchestra.run.smithi062.stdout: "exporter_sort_metrics": "true", 2024-02-05T04:53:19.736 INFO:teuthology.orchestra.run.smithi062.stdout: "exporter_stats_period": "5", 2024-02-05T04:53:19.736 INFO:teuthology.orchestra.run.smithi062.stdout: "fake_statfs_for_testing": "0", 2024-02-05T04:53:19.736 INFO:teuthology.orchestra.run.smithi062.stdout: "fatal_signal_handlers": "true", 2024-02-05T04:53:19.736 INFO:teuthology.orchestra.run.smithi062.stdout: "filer_max_purge_ops": "10", 2024-02-05T04:53:19.736 INFO:teuthology.orchestra.run.smithi062.stdout: "filer_max_truncate_ops": "128", 2024-02-05T04:53:19.737 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_apply_finisher_threads": "1", 2024-02-05T04:53:19.737 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_blackhole": "false", 2024-02-05T04:53:19.737 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_btrfs_clone_range": "true", 2024-02-05T04:53:19.737 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_btrfs_snap": "true", 2024-02-05T04:53:19.737 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_caller_concurrency": "10", 2024-02-05T04:53:19.737 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_collect_device_partition_information": "true", 2024-02-05T04:53:19.737 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_commit_timeout": "600.000000", 2024-02-05T04:53:19.737 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_debug_inject_read_err": "false", 2024-02-05T04:53:19.737 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_debug_omap_check": "false", 2024-02-05T04:53:19.737 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_debug_random_read_err": "0.000000", 2024-02-05T04:53:19.737 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_debug_verify_split": "false", 2024-02-05T04:53:19.737 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_dump_file": "", 2024-02-05T04:53:19.738 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-02-05T04:53:19.738 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-02-05T04:53:19.738 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_fadvise": "true", 2024-02-05T04:53:19.738 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_fail_eio": "true", 2024-02-05T04:53:19.738 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_fd_cache_shards": "16", 2024-02-05T04:53:19.738 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_fd_cache_size": "128", 2024-02-05T04:53:19.738 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_fiemap": "false", 2024-02-05T04:53:19.738 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_fiemap_threshold": "4096", 2024-02-05T04:53:19.738 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-02-05T04:53:19.738 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_index_retry_probability": "0.000000", 2024-02-05T04:53:19.738 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_inject_stall": "0", 2024-02-05T04:53:19.738 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_journal_parallel": "false", 2024-02-05T04:53:19.738 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_journal_trailing": "false", 2024-02-05T04:53:19.739 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_journal_writeahead": "false", 2024-02-05T04:53:19.739 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_kill_at": "0", 2024-02-05T04:53:19.739 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-02-05T04:53:19.739 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_max_inline_xattr_size": "0", 2024-02-05T04:53:19.739 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-02-05T04:53:19.739 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-02-05T04:53:19.739 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-02-05T04:53:19.739 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_max_inline_xattrs": "0", 2024-02-05T04:53:19.739 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-02-05T04:53:19.739 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_max_inline_xattrs_other": "2", 2024-02-05T04:53:19.739 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-02-05T04:53:19.739 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_max_sync_interval": "5.000000", 2024-02-05T04:53:19.739 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_max_xattr_value_size": "0", 2024-02-05T04:53:19.740 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-02-05T04:53:19.740 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-02-05T04:53:19.740 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-02-05T04:53:19.740 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_merge_threshold": "-10", 2024-02-05T04:53:19.740 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_min_sync_interval": "0.010000", 2024-02-05T04:53:19.740 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_odsync_write": "false", 2024-02-05T04:53:19.740 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_omap_backend": "rocksdb", 2024-02-05T04:53:19.740 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_omap_backend_path": "", 2024-02-05T04:53:19.740 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_omap_header_cache_size": "1024", 2024-02-05T04:53:19.740 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_ondisk_finisher_threads": "1", 2024-02-05T04:53:19.740 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-02-05T04:53:19.740 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_op_thread_timeout": "60", 2024-02-05T04:53:19.741 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_op_threads": "2", 2024-02-05T04:53:19.741 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_punch_hole": "false", 2024-02-05T04:53:19.741 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-02-05T04:53:19.741 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-02-05T04:53:19.741 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-02-05T04:53:19.741 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-02-05T04:53:19.741 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-02-05T04:53:19.741 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_queue_max_bytes": "104857600", 2024-02-05T04:53:19.741 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-02-05T04:53:19.741 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-02-05T04:53:19.741 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-02-05T04:53:19.741 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_queue_max_ops": "50", 2024-02-05T04:53:19.741 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-02-05T04:53:19.741 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_seek_data_hole": "false", 2024-02-05T04:53:19.742 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_sloppy_crc": "false", 2024-02-05T04:53:19.742 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-02-05T04:53:19.742 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_splice": "false", 2024-02-05T04:53:19.742 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_split_multiple": "2", 2024-02-05T04:53:19.742 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_split_rand_factor": "20", 2024-02-05T04:53:19.742 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_update_to": "1000", 2024-02-05T04:53:19.742 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-02-05T04:53:19.742 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-02-05T04:53:19.742 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-02-05T04:53:19.742 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-02-05T04:53:19.742 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-02-05T04:53:19.742 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-02-05T04:53:19.742 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_wbthrottle_enable": "true", 2024-02-05T04:53:19.743 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-02-05T04:53:19.743 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-02-05T04:53:19.743 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-02-05T04:53:19.743 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-02-05T04:53:19.743 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-02-05T04:53:19.743 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-02-05T04:53:19.743 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_xfs_extsize": "false", 2024-02-05T04:53:19.743 INFO:teuthology.orchestra.run.smithi062.stdout: "filestore_zfs_snap": "false", 2024-02-05T04:53:19.743 INFO:teuthology.orchestra.run.smithi062.stdout: "fio_dir": "/tmp/fio", 2024-02-05T04:53:19.743 INFO:teuthology.orchestra.run.smithi062.stdout: "fsid": "88c11a2d-382e-4f21-b8af-45ab9c504b93", 2024-02-05T04:53:19.743 INFO:teuthology.orchestra.run.smithi062.stdout: "fuse_allow_other": "true", 2024-02-05T04:53:19.743 INFO:teuthology.orchestra.run.smithi062.stdout: "fuse_atomic_o_trunc": "true", 2024-02-05T04:53:19.743 INFO:teuthology.orchestra.run.smithi062.stdout: "fuse_big_writes": "true", 2024-02-05T04:53:19.743 INFO:teuthology.orchestra.run.smithi062.stdout: "fuse_debug": "false", 2024-02-05T04:53:19.744 INFO:teuthology.orchestra.run.smithi062.stdout: "fuse_default_permissions": "false", 2024-02-05T04:53:19.744 INFO:teuthology.orchestra.run.smithi062.stdout: "fuse_disable_pagecache": "false", 2024-02-05T04:53:19.744 INFO:teuthology.orchestra.run.smithi062.stdout: "fuse_max_write": "0", 2024-02-05T04:53:19.744 INFO:teuthology.orchestra.run.smithi062.stdout: "fuse_multithreaded": "true", 2024-02-05T04:53:19.744 INFO:teuthology.orchestra.run.smithi062.stdout: "fuse_require_active_mds": "true", 2024-02-05T04:53:19.744 INFO:teuthology.orchestra.run.smithi062.stdout: "fuse_set_user_groups": "true", 2024-02-05T04:53:19.744 INFO:teuthology.orchestra.run.smithi062.stdout: "fuse_splice_move": "true", 2024-02-05T04:53:19.744 INFO:teuthology.orchestra.run.smithi062.stdout: "fuse_splice_read": "true", 2024-02-05T04:53:19.744 INFO:teuthology.orchestra.run.smithi062.stdout: "fuse_splice_write": "true", 2024-02-05T04:53:19.744 INFO:teuthology.orchestra.run.smithi062.stdout: "fuse_syncfs_on_mksnap": "true", 2024-02-05T04:53:19.744 INFO:teuthology.orchestra.run.smithi062.stdout: "fuse_use_invalidate_cb": "true", 2024-02-05T04:53:19.744 INFO:teuthology.orchestra.run.smithi062.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-02-05T04:53:19.744 INFO:teuthology.orchestra.run.smithi062.stdout: "gss_target_name": "ceph", 2024-02-05T04:53:19.745 INFO:teuthology.orchestra.run.smithi062.stdout: "heartbeat_file": "", 2024-02-05T04:53:19.745 INFO:teuthology.orchestra.run.smithi062.stdout: "heartbeat_inject_failure": "0", 2024-02-05T04:53:19.745 INFO:teuthology.orchestra.run.smithi062.stdout: "heartbeat_interval": "5", 2024-02-05T04:53:19.745 INFO:teuthology.orchestra.run.smithi062.stdout: "host": "smithi062", 2024-02-05T04:53:19.745 INFO:teuthology.orchestra.run.smithi062.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-02-05T04:53:19.745 INFO:teuthology.orchestra.run.smithi062.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-02-05T04:53:19.745 INFO:teuthology.orchestra.run.smithi062.stdout: "immutable_object_cache_max_size": "1073741824", 2024-02-05T04:53:19.745 INFO:teuthology.orchestra.run.smithi062.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-02-05T04:53:19.745 INFO:teuthology.orchestra.run.smithi062.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-02-05T04:53:19.745 INFO:teuthology.orchestra.run.smithi062.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-02-05T04:53:19.745 INFO:teuthology.orchestra.run.smithi062.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-02-05T04:53:19.745 INFO:teuthology.orchestra.run.smithi062.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-02-05T04:53:19.745 INFO:teuthology.orchestra.run.smithi062.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-02-05T04:53:19.745 INFO:teuthology.orchestra.run.smithi062.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-02-05T04:53:19.746 INFO:teuthology.orchestra.run.smithi062.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-02-05T04:53:19.746 INFO:teuthology.orchestra.run.smithi062.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-02-05T04:53:19.746 INFO:teuthology.orchestra.run.smithi062.stdout: "immutable_object_cache_watermark": "0.900000", 2024-02-05T04:53:19.746 INFO:teuthology.orchestra.run.smithi062.stdout: "inject_early_sigterm": "false", 2024-02-05T04:53:19.746 INFO:teuthology.orchestra.run.smithi062.stdout: "jaeger_agent_port": "6799", 2024-02-05T04:53:19.746 INFO:teuthology.orchestra.run.smithi062.stdout: "jaeger_tracing_enable": "false", 2024-02-05T04:53:19.746 INFO:teuthology.orchestra.run.smithi062.stdout: "journal_aio": "true", 2024-02-05T04:53:19.746 INFO:teuthology.orchestra.run.smithi062.stdout: "journal_align_min_size": "65536", 2024-02-05T04:53:19.746 INFO:teuthology.orchestra.run.smithi062.stdout: "journal_block_align": "true", 2024-02-05T04:53:19.746 INFO:teuthology.orchestra.run.smithi062.stdout: "journal_block_size": "4096", 2024-02-05T04:53:19.746 INFO:teuthology.orchestra.run.smithi062.stdout: "journal_dio": "true", 2024-02-05T04:53:19.746 INFO:teuthology.orchestra.run.smithi062.stdout: "journal_discard": "false", 2024-02-05T04:53:19.746 INFO:teuthology.orchestra.run.smithi062.stdout: "journal_force_aio": "false", 2024-02-05T04:53:19.746 INFO:teuthology.orchestra.run.smithi062.stdout: "journal_ignore_corruption": "false", 2024-02-05T04:53:19.747 INFO:teuthology.orchestra.run.smithi062.stdout: "journal_max_write_bytes": "10485760", 2024-02-05T04:53:19.747 INFO:teuthology.orchestra.run.smithi062.stdout: "journal_max_write_entries": "100", 2024-02-05T04:53:19.747 INFO:teuthology.orchestra.run.smithi062.stdout: "journal_replay_from": "0", 2024-02-05T04:53:19.747 INFO:teuthology.orchestra.run.smithi062.stdout: "journal_throttle_high_multiple": "0.000000", 2024-02-05T04:53:19.747 INFO:teuthology.orchestra.run.smithi062.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-02-05T04:53:19.747 INFO:teuthology.orchestra.run.smithi062.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-02-05T04:53:19.747 INFO:teuthology.orchestra.run.smithi062.stdout: "journal_throttle_max_multiple": "0.000000", 2024-02-05T04:53:19.747 INFO:teuthology.orchestra.run.smithi062.stdout: "journal_write_header_frequency": "0", 2024-02-05T04:53:19.747 INFO:teuthology.orchestra.run.smithi062.stdout: "journal_zero_on_create": "false", 2024-02-05T04:53:19.747 INFO:teuthology.orchestra.run.smithi062.stdout: "journaler_prefetch_periods": "10", 2024-02-05T04:53:19.747 INFO:teuthology.orchestra.run.smithi062.stdout: "journaler_prezero_periods": "5", 2024-02-05T04:53:19.747 INFO:teuthology.orchestra.run.smithi062.stdout: "journaler_write_head_interval": "15", 2024-02-05T04:53:19.747 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "", 2024-02-05T04:53:19.747 INFO:teuthology.orchestra.run.smithi062.stdout: "keyfile": "", 2024-02-05T04:53:19.747 INFO:teuthology.orchestra.run.smithi062.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-02-05T04:53:19.748 INFO:teuthology.orchestra.run.smithi062.stdout: "kstore_backend": "rocksdb", 2024-02-05T04:53:19.748 INFO:teuthology.orchestra.run.smithi062.stdout: "kstore_default_stripe_size": "65536", 2024-02-05T04:53:19.748 INFO:teuthology.orchestra.run.smithi062.stdout: "kstore_fsck_on_mount": "false", 2024-02-05T04:53:19.748 INFO:teuthology.orchestra.run.smithi062.stdout: "kstore_fsck_on_mount_deep": "true", 2024-02-05T04:53:19.748 INFO:teuthology.orchestra.run.smithi062.stdout: "kstore_max_bytes": "67108864", 2024-02-05T04:53:19.748 INFO:teuthology.orchestra.run.smithi062.stdout: "kstore_max_ops": "512", 2024-02-05T04:53:19.748 INFO:teuthology.orchestra.run.smithi062.stdout: "kstore_nid_prealloc": "1024", 2024-02-05T04:53:19.748 INFO:teuthology.orchestra.run.smithi062.stdout: "kstore_onode_map_size": "1024", 2024-02-05T04:53:19.748 INFO:teuthology.orchestra.run.smithi062.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-02-05T04:53:19.748 INFO:teuthology.orchestra.run.smithi062.stdout: "kstore_sync_submit_transaction": "false", 2024-02-05T04:53:19.748 INFO:teuthology.orchestra.run.smithi062.stdout: "kstore_sync_transaction": "false", 2024-02-05T04:53:19.748 INFO:teuthology.orchestra.run.smithi062.stdout: "librados_thread_count": "2", 2024-02-05T04:53:19.748 INFO:teuthology.orchestra.run.smithi062.stdout: "lockdep": "false", 2024-02-05T04:53:19.748 INFO:teuthology.orchestra.run.smithi062.stdout: "lockdep_force_backtrace": "false", 2024-02-05T04:53:19.749 INFO:teuthology.orchestra.run.smithi062.stdout: "log_coarse_timestamps": "true", 2024-02-05T04:53:19.749 INFO:teuthology.orchestra.run.smithi062.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-02-05T04:53:19.749 INFO:teuthology.orchestra.run.smithi062.stdout: "log_flush_on_exit": "false", 2024-02-05T04:53:19.749 INFO:teuthology.orchestra.run.smithi062.stdout: "log_graylog_host": "127.0.0.1", 2024-02-05T04:53:19.749 INFO:teuthology.orchestra.run.smithi062.stdout: "log_graylog_port": "12201", 2024-02-05T04:53:19.749 INFO:teuthology.orchestra.run.smithi062.stdout: "log_max_new": "1000", 2024-02-05T04:53:19.749 INFO:teuthology.orchestra.run.smithi062.stdout: "log_max_recent": "10000", 2024-02-05T04:53:19.749 INFO:teuthology.orchestra.run.smithi062.stdout: "log_stderr_prefix": "", 2024-02-05T04:53:19.749 INFO:teuthology.orchestra.run.smithi062.stdout: "log_stop_at_utilization": "0.970000", 2024-02-05T04:53:19.749 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": "true", 2024-02-05T04:53:19.749 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_graylog": "false", 2024-02-05T04:53:19.749 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_journald": "false", 2024-02-05T04:53:19.749 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_stderr": "false", 2024-02-05T04:53:19.749 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_syslog": "false", 2024-02-05T04:53:19.749 INFO:teuthology.orchestra.run.smithi062.stdout: "mandatory_topic_permissions": "false", 2024-02-05T04:53:19.750 INFO:teuthology.orchestra.run.smithi062.stdout: "max_rotating_auth_attempts": "10", 2024-02-05T04:53:19.750 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-02-05T04:53:19.750 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_action_on_write_error": "1", 2024-02-05T04:53:19.750 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_alternate_name_max": "8192", 2024-02-05T04:53:19.750 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_asio_thread_count": "2", 2024-02-05T04:53:19.750 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_bal_export_pin": "true", 2024-02-05T04:53:19.750 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_bal_fragment_dirs": "true", 2024-02-05T04:53:19.750 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-02-05T04:53:19.750 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_bal_fragment_interval": "5", 2024-02-05T04:53:19.750 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_bal_fragment_size_max": "100000", 2024-02-05T04:53:19.750 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_bal_idle_threshold": "0.000000", 2024-02-05T04:53:19.750 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_bal_interval": "10", 2024-02-05T04:53:19.750 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_bal_max": "-1", 2024-02-05T04:53:19.750 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_bal_max_until": "-1", 2024-02-05T04:53:19.751 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_bal_merge_size": "50", 2024-02-05T04:53:19.751 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_bal_midchunk": "0.300000", 2024-02-05T04:53:19.751 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_bal_min_rebalance": "0.100000", 2024-02-05T04:53:19.751 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_bal_min_start": "0.200000", 2024-02-05T04:53:19.751 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_bal_minchunk": "0.001000", 2024-02-05T04:53:19.751 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_bal_mode": "0", 2024-02-05T04:53:19.751 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_bal_need_max": "1.200000", 2024-02-05T04:53:19.751 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_bal_need_min": "0.800000", 2024-02-05T04:53:19.751 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_bal_overload_epochs": "2", 2024-02-05T04:53:19.751 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-02-05T04:53:19.751 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_bal_sample_interval": "3.000000", 2024-02-05T04:53:19.751 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_bal_split_bits": "3", 2024-02-05T04:53:19.751 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_bal_split_rd": "25000.000000", 2024-02-05T04:53:19.751 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_bal_split_size": "10000", 2024-02-05T04:53:19.752 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_bal_split_wr": "10000.000000", 2024-02-05T04:53:19.752 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_bal_target_decay": "10.000000", 2024-02-05T04:53:19.752 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-02-05T04:53:19.752 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_beacon_grace": "15.000000", 2024-02-05T04:53:19.752 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_beacon_interval": "4.000000", 2024-02-05T04:53:19.752 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_beacon_mon_down_grace": "60", 2024-02-05T04:53:19.752 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_cache_memory_limit": "4294967296", 2024-02-05T04:53:19.752 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_cache_mid": "0.700000", 2024-02-05T04:53:19.752 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_cache_release_free_interval": "10", 2024-02-05T04:53:19.752 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_cache_reservation": "0.050000", 2024-02-05T04:53:19.752 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-02-05T04:53:19.752 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_cache_trim_interval": "1", 2024-02-05T04:53:19.752 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_cache_trim_threshold": "262144", 2024-02-05T04:53:19.752 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-02-05T04:53:19.752 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-02-05T04:53:19.753 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_client_delegate_inos_pct": "50", 2024-02-05T04:53:19.753 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_client_prealloc_inos": "1000", 2024-02-05T04:53:19.753 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-02-05T04:53:19.753 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_connect_bootstrapping": "false", 2024-02-05T04:53:19.753 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_damage_table_max_entries": "10000", 2024-02-05T04:53:19.753 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-02-05T04:53:19.753 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_debug_auth_pins": "false", 2024-02-05T04:53:19.753 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_debug_frag": "false", 2024-02-05T04:53:19.753 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_debug_scatterstat": "false", 2024-02-05T04:53:19.753 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_debug_subtrees": "false", 2024-02-05T04:53:19.753 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_decay_halflife": "5.000000", 2024-02-05T04:53:19.753 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_default_dir_hash": "2", 2024-02-05T04:53:19.753 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_defer_session_stale": "true", 2024-02-05T04:53:19.753 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_deny_all_reconnect": "false", 2024-02-05T04:53:19.753 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_dir_keys_per_op": "16384", 2024-02-05T04:53:19.754 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_dir_max_commit_size": "10", 2024-02-05T04:53:19.754 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_dir_max_entries": "0", 2024-02-05T04:53:19.754 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_dir_prefetch": "true", 2024-02-05T04:53:19.754 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_dirstat_min_interval": "1.000000", 2024-02-05T04:53:19.754 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_dump_cache_after_rejoin": "false", 2024-02-05T04:53:19.754 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_dump_cache_on_map": "false", 2024-02-05T04:53:19.754 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_dump_cache_threshold_file": "0", 2024-02-05T04:53:19.754 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-02-05T04:53:19.754 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_early_reply": "true", 2024-02-05T04:53:19.754 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_enable_op_tracker": "true", 2024-02-05T04:53:19.754 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_enforce_unique_name": "true", 2024-02-05T04:53:19.754 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_export_ephemeral_distributed": "true", 2024-02-05T04:53:19.754 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-02-05T04:53:19.754 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_export_ephemeral_random": "true", 2024-02-05T04:53:19.755 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-02-05T04:53:19.755 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-02-05T04:53:19.755 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_forward_all_requests_to_auth": "false", 2024-02-05T04:53:19.755 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-02-05T04:53:19.755 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-02-05T04:53:19.755 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-02-05T04:53:19.755 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-02-05T04:53:19.755 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_health_cache_threshold": "1.500000", 2024-02-05T04:53:19.755 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_health_summarize_threshold": "10", 2024-02-05T04:53:19.755 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_heartbeat_grace": "15.000000", 2024-02-05T04:53:19.755 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_heartbeat_reset_grace": "1000", 2024-02-05T04:53:19.755 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_inject_health_dummy": "false", 2024-02-05T04:53:19.755 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-02-05T04:53:19.755 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_inject_migrator_session_race": "false", 2024-02-05T04:53:19.755 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-02-05T04:53:19.756 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-02-05T04:53:19.756 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-02-05T04:53:19.756 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_join_fs": "", 2024-02-05T04:53:19.756 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_journal_format": "1", 2024-02-05T04:53:19.756 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_kill_create_at": "0", 2024-02-05T04:53:19.756 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_kill_export_at": "0", 2024-02-05T04:53:19.756 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_kill_import_at": "0", 2024-02-05T04:53:19.756 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_kill_journal_at": "0", 2024-02-05T04:53:19.756 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_kill_journal_expire_at": "0", 2024-02-05T04:53:19.756 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_kill_journal_replay_at": "0", 2024-02-05T04:53:19.756 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_kill_link_at": "0", 2024-02-05T04:53:19.756 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_kill_mdstable_at": "0", 2024-02-05T04:53:19.756 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_kill_openc_at": "0", 2024-02-05T04:53:19.757 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_kill_rename_at": "0", 2024-02-05T04:53:19.757 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_kill_shutdown_at": "0", 2024-02-05T04:53:19.757 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_kill_skip_replaying_inotable": "false", 2024-02-05T04:53:19.757 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_log_event_large_threshold": "524288", 2024-02-05T04:53:19.757 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_log_events_per_segment": "1024", 2024-02-05T04:53:19.757 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_log_major_segment_event_ratio": "12", 2024-02-05T04:53:19.757 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_log_max_events": "-1", 2024-02-05T04:53:19.757 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_log_max_segments": "128", 2024-02-05T04:53:19.757 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_log_pause": "false", 2024-02-05T04:53:19.757 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_log_segment_size": "0", 2024-02-05T04:53:19.757 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_log_skip_corrupt_events": "false", 2024-02-05T04:53:19.757 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_log_skip_unbounded_events": "false", 2024-02-05T04:53:19.757 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_log_warn_factor": "2.000000", 2024-02-05T04:53:19.757 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_max_caps_per_client": "1048576", 2024-02-05T04:53:19.757 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_max_completed_flushes": "100000", 2024-02-05T04:53:19.758 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_max_completed_requests": "100000", 2024-02-05T04:53:19.758 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_max_export_size": "20971520", 2024-02-05T04:53:19.758 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_max_file_recover": "32", 2024-02-05T04:53:19.758 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_max_purge_files": "64", 2024-02-05T04:53:19.758 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_max_purge_ops": "8192", 2024-02-05T04:53:19.758 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-02-05T04:53:19.758 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-02-05T04:53:19.758 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_max_snaps_per_dir": "100", 2024-02-05T04:53:19.758 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_metrics_update_interval": "2", 2024-02-05T04:53:19.758 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_min_caps_per_client": "100", 2024-02-05T04:53:19.758 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_min_caps_working_set": "10000", 2024-02-05T04:53:19.758 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-02-05T04:53:19.759 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_numa_node": "-1", 2024-02-05T04:53:19.759 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-02-05T04:53:19.759 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_op_complaint_time": "30.000000", 2024-02-05T04:53:19.759 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_op_history_duration": "600", 2024-02-05T04:53:19.759 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_op_history_size": "20", 2024-02-05T04:53:19.759 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_op_history_slow_op_size": "20", 2024-02-05T04:53:19.759 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_op_history_slow_op_threshold": "10", 2024-02-05T04:53:19.759 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_op_log_threshold": "5", 2024-02-05T04:53:19.759 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_ping_grace": "15", 2024-02-05T04:53:19.759 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_ping_interval": "5", 2024-02-05T04:53:19.759 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-02-05T04:53:19.759 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-02-05T04:53:19.759 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_recall_max_caps": "30000", 2024-02-05T04:53:19.759 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-02-05T04:53:19.760 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_recall_max_decay_threshold": "131072", 2024-02-05T04:53:19.760 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-02-05T04:53:19.760 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_recall_warning_threshold": "262144", 2024-02-05T04:53:19.760 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_reconnect_timeout": "45.000000", 2024-02-05T04:53:19.760 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_replay_interval": "1.000000", 2024-02-05T04:53:19.760 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-02-05T04:53:19.760 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-02-05T04:53:19.760 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_root_ino_gid": "0", 2024-02-05T04:53:19.760 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_root_ino_uid": "0", 2024-02-05T04:53:19.760 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-02-05T04:53:19.760 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_session_blocklist_on_evict": "true", 2024-02-05T04:53:19.760 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_session_blocklist_on_timeout": "true", 2024-02-05T04:53:19.760 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-02-05T04:53:19.760 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-02-05T04:53:19.760 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-02-05T04:53:19.761 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-02-05T04:53:19.761 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-02-05T04:53:19.761 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_session_metadata_threshold": "16777216", 2024-02-05T04:53:19.761 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-02-05T04:53:19.761 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_shutdown_check": "0", 2024-02-05T04:53:19.761 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_skip_ino": "0", 2024-02-05T04:53:19.761 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_sleep_rank_change": "0.000000", 2024-02-05T04:53:19.761 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_snap_max_uid": "4294967294", 2024-02-05T04:53:19.761 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_snap_min_uid": "0", 2024-02-05T04:53:19.761 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_snap_rstat": "false", 2024-02-05T04:53:19.761 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_standby_replay_damaged": "false", 2024-02-05T04:53:19.761 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_symlink_recovery": "true", 2024-02-05T04:53:19.761 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_task_status_update_interval": "2.000000", 2024-02-05T04:53:19.761 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_thrash_exports": "0", 2024-02-05T04:53:19.761 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_thrash_fragments": "0", 2024-02-05T04:53:19.761 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_tick_interval": "5.000000", 2024-02-05T04:53:19.762 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_valgrind_exit": "false", 2024-02-05T04:53:19.762 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_verify_backtrace": "1", 2024-02-05T04:53:19.762 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_verify_scatter": "false", 2024-02-05T04:53:19.762 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_wipe_ino_prealloc": "false", 2024-02-05T04:53:19.762 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_wipe_sessions": "false", 2024-02-05T04:53:19.762 INFO:teuthology.orchestra.run.smithi062.stdout: "mempool_debug": "false", 2024-02-05T04:53:19.762 INFO:teuthology.orchestra.run.smithi062.stdout: "memstore_debug_omit_block_device_write": "false", 2024-02-05T04:53:19.762 INFO:teuthology.orchestra.run.smithi062.stdout: "memstore_device_bytes": "1073741824", 2024-02-05T04:53:19.762 INFO:teuthology.orchestra.run.smithi062.stdout: "memstore_page_set": "false", 2024-02-05T04:53:19.762 INFO:teuthology.orchestra.run.smithi062.stdout: "memstore_page_size": "65536", 2024-02-05T04:53:19.762 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr_client_bytes": "134217728", 2024-02-05T04:53:19.762 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr_client_messages": "512", 2024-02-05T04:53:19.762 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-02-05T04:53:19.762 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr_connect_retry_interval": "1.000000", 2024-02-05T04:53:19.762 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-02-05T04:53:19.762 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-02-05T04:53:19.762 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr_disabled_modules": "", 2024-02-05T04:53:19.763 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-02-05T04:53:19.763 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr_max_pg_num_change": "128", 2024-02-05T04:53:19.763 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr_mds_bytes": "134217728", 2024-02-05T04:53:19.763 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr_mds_messages": "128", 2024-02-05T04:53:19.763 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-02-05T04:53:19.763 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr_mon_bytes": "134217728", 2024-02-05T04:53:19.763 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr_mon_messages": "128", 2024-02-05T04:53:19.763 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr_osd_bytes": "536870912", 2024-02-05T04:53:19.763 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr_osd_messages": "8192", 2024-02-05T04:53:19.763 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr_pool": "true", 2024-02-05T04:53:19.763 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr_service_beacon_grace": "60.000000", 2024-02-05T04:53:19.763 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr_standby_modules": "true", 2024-02-05T04:53:19.763 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr_stats_period": "5", 2024-02-05T04:53:19.763 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr_stats_threshold": "5", 2024-02-05T04:53:19.764 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr_tick_period": "2", 2024-02-05T04:53:19.764 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-02-05T04:53:19.764 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_accept_timeout_factor": "2.000000", 2024-02-05T04:53:19.764 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_allow_pool_delete": "true", 2024-02-05T04:53:19.764 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_allow_pool_size_one": "true", 2024-02-05T04:53:19.764 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_auth_validate_all_caps": "true", 2024-02-05T04:53:19.764 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-02-05T04:53:19.764 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-02-05T04:53:19.764 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_client_bytes": "104857600", 2024-02-05T04:53:19.764 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_client_directed_command_retry": "5", 2024-02-05T04:53:19.764 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_client_hunt_interval": "3.000000", 2024-02-05T04:53:19.764 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-02-05T04:53:19.764 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-02-05T04:53:19.765 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-02-05T04:53:19.765 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_client_hunt_parallel": "3", 2024-02-05T04:53:19.765 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_client_log_interval": "1.000000", 2024-02-05T04:53:19.765 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-02-05T04:53:19.765 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_client_ping_interval": "10.000000", 2024-02-05T04:53:19.765 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_client_ping_timeout": "30.000000", 2024-02-05T04:53:19.765 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_client_target_rank": "-1", 2024-02-05T04:53:19.765 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_clock_drift_allowed": "1.000000", 2024-02-05T04:53:19.765 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-02-05T04:53:19.765 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-02-05T04:53:19.765 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_cluster_log_file_level": "debug", 2024-02-05T04:53:19.765 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_cluster_log_to_file": "true", 2024-02-05T04:53:19.765 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_cluster_log_to_graylog": "false", 2024-02-05T04:53:19.765 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-02-05T04:53:19.766 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-02-05T04:53:19.766 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_cluster_log_to_journald": "false", 2024-02-05T04:53:19.766 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_cluster_log_to_stderr": "false", 2024-02-05T04:53:19.766 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-02-05T04:53:19.766 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-02-05T04:53:19.766 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-02-05T04:53:19.766 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_compact_on_bootstrap": "false", 2024-02-05T04:53:19.766 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_compact_on_start": "false", 2024-02-05T04:53:19.766 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_compact_on_trim": "true", 2024-02-05T04:53:19.766 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_con_tracker_persist_interval": "10", 2024-02-05T04:53:19.766 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_con_tracker_score_halflife": "43200", 2024-02-05T04:53:19.766 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_config_key_max_entry_size": "65536", 2024-02-05T04:53:19.766 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_cpu_threads": "4", 2024-02-05T04:53:19.767 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_crush_min_required_version": "hammer", 2024-02-05T04:53:19.767 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_daemon_bytes": "419430400", 2024-02-05T04:53:19.767 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-02-05T04:53:19.767 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_data_avail_crit": "5", 2024-02-05T04:53:19.767 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_data_avail_warn": "5", 2024-02-05T04:53:19.767 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_data_size_warn": "16106127360", 2024-02-05T04:53:19.767 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_debug_block_osdmap_trim": "false", 2024-02-05T04:53:19.767 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-02-05T04:53:19.767 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_debug_dump_json": "false", 2024-02-05T04:53:19.767 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-02-05T04:53:19.767 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_debug_dump_transactions": "false", 2024-02-05T04:53:19.767 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_debug_extra_checks": "false", 2024-02-05T04:53:19.767 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-02-05T04:53:19.768 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-02-05T04:53:19.768 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_debug_no_require_reef": "false", 2024-02-05T04:53:19.768 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_debug_no_require_squid": "false", 2024-02-05T04:53:19.768 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-02-05T04:53:19.768 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_delta_reset_interval": "10.000000", 2024-02-05T04:53:19.768 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_dns_srv_name": "ceph-mon", 2024-02-05T04:53:19.768 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_down_mkfs_grace": "120", 2024-02-05T04:53:19.768 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_election_default_strategy": "3", 2024-02-05T04:53:19.768 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_election_timeout": "5.000000", 2024-02-05T04:53:19.768 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-02-05T04:53:19.768 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_elector_ping_divisor": "2", 2024-02-05T04:53:19.768 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_elector_ping_timeout": "2.000000", 2024-02-05T04:53:19.769 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_enable_op_tracker": "true", 2024-02-05T04:53:19.769 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_fake_pool_delete": "false", 2024-02-05T04:53:19.769 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_force_quorum_join": "false", 2024-02-05T04:53:19.769 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_globalid_prealloc": "10000", 2024-02-05T04:53:19.769 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_health_detail_to_clog": "false", 2024-02-05T04:53:19.769 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_health_log_update_period": "5", 2024-02-05T04:53:19.769 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_health_max_detail": "50", 2024-02-05T04:53:19.769 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_health_to_clog": "true", 2024-02-05T04:53:19.769 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_health_to_clog_interval": "600", 2024-02-05T04:53:19.769 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-02-05T04:53:19.769 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_host": "172.21.15.62,172.21.15.91,172.21.15.150", 2024-02-05T04:53:19.769 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_host_override": "", 2024-02-05T04:53:19.769 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_initial_members": "", 2024-02-05T04:53:19.769 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-02-05T04:53:19.770 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-02-05T04:53:19.770 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-02-05T04:53:19.770 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-02-05T04:53:19.770 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_keyvaluedb": "rocksdb", 2024-02-05T04:53:19.770 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_lease": "5.000000", 2024-02-05T04:53:19.770 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-02-05T04:53:19.770 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-02-05T04:53:19.770 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_log_full_interval": "50", 2024-02-05T04:53:19.770 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_log_max": "10000", 2024-02-05T04:53:19.770 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_log_max_summary": "50", 2024-02-05T04:53:19.770 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_max_log_entries_per_event": "4096", 2024-02-05T04:53:19.770 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_max_log_epochs": "500", 2024-02-05T04:53:19.770 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_max_mdsmap_epochs": "500", 2024-02-05T04:53:19.770 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_max_mgrmap_epochs": "500", 2024-02-05T04:53:19.771 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_max_osd": "10000", 2024-02-05T04:53:19.771 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_max_pg_per_osd": "10000", 2024-02-05T04:53:19.771 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_max_pool_pg_num": "65536", 2024-02-05T04:53:19.771 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-02-05T04:53:19.771 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-02-05T04:53:19.771 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_mds_force_trim_to": "0", 2024-02-05T04:53:19.771 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_mds_skip_sanity": "false", 2024-02-05T04:53:19.771 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_memory_autotune": "true", 2024-02-05T04:53:19.771 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_memory_target": "2147483648", 2024-02-05T04:53:19.771 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_mgr_beacon_grace": "30", 2024-02-05T04:53:19.771 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-02-05T04:53:19.771 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_mgr_digest_period": "5", 2024-02-05T04:53:19.771 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_mgr_inactive_grace": "60", 2024-02-05T04:53:19.771 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_mgr_mkfs_grace": "240", 2024-02-05T04:53:19.772 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-02-05T04:53:19.772 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_min_osdmap_epochs": "500", 2024-02-05T04:53:19.772 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_op_complaint_time": "30", 2024-02-05T04:53:19.772 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_op_history_duration": "600", 2024-02-05T04:53:19.772 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_op_history_size": "20", 2024-02-05T04:53:19.772 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_op_history_slow_op_size": "20", 2024-02-05T04:53:19.772 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_op_history_slow_op_threshold": "10", 2024-02-05T04:53:19.772 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_op_log_threshold": "5", 2024-02-05T04:53:19.772 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-02-05T04:53:19.772 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-02-05T04:53:19.772 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-02-05T04:53:19.772 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_osd_auto_mark_in": "false", 2024-02-05T04:53:19.772 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_osd_auto_mark_new_in": "true", 2024-02-05T04:53:19.772 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-02-05T04:53:19.773 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-02-05T04:53:19.773 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_osd_cache_size": "500", 2024-02-05T04:53:19.773 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_osd_cache_size_min": "134217728", 2024-02-05T04:53:19.773 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_osd_crush_smoke_test": "true", 2024-02-05T04:53:19.773 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_osd_destroyed_out_interval": "600", 2024-02-05T04:53:19.773 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_osd_down_out_interval": "600", 2024-02-05T04:53:19.773 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-02-05T04:53:19.773 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-02-05T04:53:19.773 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_osd_force_trim_to": "0", 2024-02-05T04:53:19.773 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_osd_full_ratio": "0.950000", 2024-02-05T04:53:19.773 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-02-05T04:53:19.773 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_osd_laggy_halflife": "3600", 2024-02-05T04:53:19.773 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_osd_laggy_max_interval": "300", 2024-02-05T04:53:19.774 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_osd_laggy_weight": "0.300000", 2024-02-05T04:53:19.774 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-02-05T04:53:19.774 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_osd_max_creating_pgs": "1024", 2024-02-05T04:53:19.774 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_osd_max_initial_pgs": "1024", 2024-02-05T04:53:19.774 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_osd_min_down_reporters": "2", 2024-02-05T04:53:19.774 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-02-05T04:53:19.774 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-02-05T04:53:19.774 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-02-05T04:53:19.774 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_osd_prime_pg_temp": "true", 2024-02-05T04:53:19.774 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-02-05T04:53:19.774 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-02-05T04:53:19.774 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_osd_report_timeout": "900", 2024-02-05T04:53:19.774 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-02-05T04:53:19.775 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-02-05T04:53:19.775 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_osd_warn_num_repaired": "10", 2024-02-05T04:53:19.775 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_osd_warn_op_age": "32.000000", 2024-02-05T04:53:19.775 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-02-05T04:53:19.775 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_osdmap_full_prune_interval": "10", 2024-02-05T04:53:19.775 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_osdmap_full_prune_min": "10000", 2024-02-05T04:53:19.775 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_osdmap_full_prune_txsize": "100", 2024-02-05T04:53:19.775 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-02-05T04:53:19.775 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_pg_stuck_threshold": "60", 2024-02-05T04:53:19.775 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-02-05T04:53:19.775 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_pg_warn_min_objects": "10000", 2024-02-05T04:53:19.775 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_pg_warn_min_per_osd": "0", 2024-02-05T04:53:19.775 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-02-05T04:53:19.775 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_pool_quota_crit_threshold": "0", 2024-02-05T04:53:19.776 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_pool_quota_warn_threshold": "0", 2024-02-05T04:53:19.776 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_probe_timeout": "2.000000", 2024-02-05T04:53:19.776 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_reweight_max_change": "0.050000", 2024-02-05T04:53:19.776 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_reweight_max_osds": "4", 2024-02-05T04:53:19.776 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-02-05T04:53:19.776 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-02-05T04:53:19.776 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-02-05T04:53:19.776 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-02-05T04:53:19.776 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-02-05T04:53:19.776 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_scrub_interval": "300", 2024-02-05T04:53:19.776 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_scrub_max_keys": "100", 2024-02-05T04:53:19.776 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_scrub_timeout": "300", 2024-02-05T04:53:19.776 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_session_timeout": "300", 2024-02-05T04:53:19.776 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_smart_report_timeout": "5", 2024-02-05T04:53:19.777 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_stat_smooth_intervals": "6", 2024-02-05T04:53:19.777 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-02-05T04:53:19.777 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_stretch_pool_min_size": "2", 2024-02-05T04:53:19.777 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_stretch_pool_size": "4", 2024-02-05T04:53:19.777 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-02-05T04:53:19.777 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_subscribe_interval": "86400.000000", 2024-02-05T04:53:19.777 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_sync_debug": "false", 2024-02-05T04:53:19.777 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_sync_max_payload_keys": "2000", 2024-02-05T04:53:19.777 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_sync_max_payload_size": "1048576", 2024-02-05T04:53:19.777 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_sync_provider_kill_at": "0", 2024-02-05T04:53:19.777 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_sync_requester_kill_at": "0", 2024-02-05T04:53:19.777 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_sync_timeout": "60.000000", 2024-02-05T04:53:19.777 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_target_pg_per_osd": "100", 2024-02-05T04:53:19.777 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_tick_interval": "5", 2024-02-05T04:53:19.777 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_timecheck_interval": "300.000000", 2024-02-05T04:53:19.778 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-02-05T04:53:19.778 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_use_min_delay_socket": "false", 2024-02-05T04:53:19.778 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_warn_older_version_delay": "604800", 2024-02-05T04:53:19.778 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-02-05T04:53:19.778 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-02-05T04:53:19.778 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-02-05T04:53:19.778 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_warn_on_filestore_osds": "false", 2024-02-05T04:53:19.778 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-02-05T04:53:19.778 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-02-05T04:53:19.778 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-02-05T04:53:19.778 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_warn_on_misplaced": "false", 2024-02-05T04:53:19.778 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-02-05T04:53:19.778 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_warn_on_older_version": "true", 2024-02-05T04:53:19.778 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-02-05T04:53:19.779 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_warn_on_pool_no_app": "true", 2024-02-05T04:53:19.779 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_warn_on_pool_no_redundancy": "false", 2024-02-05T04:53:19.779 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-02-05T04:53:19.779 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-02-05T04:53:19.779 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-02-05T04:53:19.779 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_warn_on_too_few_osds": "false", 2024-02-05T04:53:19.779 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-02-05T04:53:19.779 INFO:teuthology.orchestra.run.smithi062.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-02-05T04:53:19.779 INFO:teuthology.orchestra.run.smithi062.stdout: "monmap": "", 2024-02-05T04:53:19.779 INFO:teuthology.orchestra.run.smithi062.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-02-05T04:53:19.779 INFO:teuthology.orchestra.run.smithi062.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-02-05T04:53:19.779 INFO:teuthology.orchestra.run.smithi062.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-02-05T04:53:19.779 INFO:teuthology.orchestra.run.smithi062.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-02-05T04:53:19.780 INFO:teuthology.orchestra.run.smithi062.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-02-05T04:53:19.780 INFO:teuthology.orchestra.run.smithi062.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-02-05T04:53:19.780 INFO:teuthology.orchestra.run.smithi062.stdout: "motr_tracing_enabled": "false", 2024-02-05T04:53:19.780 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_async_op_threads": "3", 2024-02-05T04:53:19.780 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_async_rdma_buffer_size": "131072", 2024-02-05T04:53:19.780 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_async_rdma_cm": "false", 2024-02-05T04:53:19.780 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_async_rdma_device_name": "", 2024-02-05T04:53:19.780 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_async_rdma_dscp": "96", 2024-02-05T04:53:19.780 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-02-05T04:53:19.780 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_async_rdma_gid_idx": "0", 2024-02-05T04:53:19.780 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_async_rdma_local_gid": "", 2024-02-05T04:53:19.780 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_async_rdma_polling_us": "1000", 2024-02-05T04:53:19.780 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_async_rdma_port_num": "1", 2024-02-05T04:53:19.780 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-02-05T04:53:19.780 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-02-05T04:53:19.781 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_async_rdma_roce_ver": "1", 2024-02-05T04:53:19.781 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_async_rdma_send_buffers": "1024", 2024-02-05T04:53:19.781 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_async_rdma_sl": "3", 2024-02-05T04:53:19.781 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_async_rdma_support_srq": "true", 2024-02-05T04:53:19.781 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_async_rdma_type": "ib", 2024-02-05T04:53:19.781 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_async_reap_threshold": "5", 2024-02-05T04:53:19.781 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_bind_before_connect": "false", 2024-02-05T04:53:19.781 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_bind_ipv4": "true", 2024-02-05T04:53:19.781 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_bind_ipv6": "false", 2024-02-05T04:53:19.781 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_bind_msgr1": "true", 2024-02-05T04:53:19.781 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_bind_msgr2": "true", 2024-02-05T04:53:19.781 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_bind_port_max": "7568", 2024-02-05T04:53:19.781 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_bind_port_min": "6800", 2024-02-05T04:53:19.781 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_bind_prefer_ipv4": "false", 2024-02-05T04:53:19.782 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_bind_retry_count": "3", 2024-02-05T04:53:19.782 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_bind_retry_delay": "5", 2024-02-05T04:53:19.782 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_blackhole_client": "false", 2024-02-05T04:53:19.782 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_blackhole_mds": "false", 2024-02-05T04:53:19.782 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_blackhole_mgr": "false", 2024-02-05T04:53:19.782 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_blackhole_mon": "false", 2024-02-05T04:53:19.782 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_blackhole_osd": "false", 2024-02-05T04:53:19.782 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_client_mode": "crc secure", 2024-02-05T04:53:19.782 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_cluster_mode": "crc secure", 2024-02-05T04:53:19.782 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_cluster_type": "", 2024-02-05T04:53:19.782 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_compress_secure": "false", 2024-02-05T04:53:19.782 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_connection_idle_timeout": "900", 2024-02-05T04:53:19.782 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_connection_ready_timeout": "10", 2024-02-05T04:53:19.782 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_crc_data": "true", 2024-02-05T04:53:19.783 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_crc_header": "true", 2024-02-05T04:53:19.783 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_die_on_bad_msg": "false", 2024-02-05T04:53:19.783 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_die_on_bug": "true", 2024-02-05T04:53:19.783 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_die_on_old_message": "true", 2024-02-05T04:53:19.783 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_die_on_skipped_message": "false", 2024-02-05T04:53:19.783 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_die_on_unhandled_msg": "false", 2024-02-05T04:53:19.783 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-02-05T04:53:19.783 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_dpdk_coremask": "0xF", 2024-02-05T04:53:19.783 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-02-05T04:53:19.783 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_dpdk_devs_allowlist": "", 2024-02-05T04:53:19.783 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_dpdk_enable_tso": "true", 2024-02-05T04:53:19.783 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-02-05T04:53:19.783 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-02-05T04:53:19.783 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_dpdk_hugepages": "", 2024-02-05T04:53:19.783 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_dpdk_hw_flow_control": "true", 2024-02-05T04:53:19.784 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-02-05T04:53:19.784 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_dpdk_lro": "true", 2024-02-05T04:53:19.784 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_dpdk_memory_channel": "4", 2024-02-05T04:53:19.784 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-02-05T04:53:19.784 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_dpdk_pmd": "", 2024-02-05T04:53:19.784 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_dpdk_port_id": "0", 2024-02-05T04:53:19.784 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-02-05T04:53:19.784 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_dump_corrupt_message_level": "1", 2024-02-05T04:53:19.784 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_dump_on_send": "false", 2024-02-05T04:53:19.784 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_initial_backoff": "0.200000", 2024-02-05T04:53:19.784 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_inject_delay_max": "1.000000", 2024-02-05T04:53:19.784 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_inject_delay_probability": "0.000000", 2024-02-05T04:53:19.784 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_inject_delay_type": "", 2024-02-05T04:53:19.784 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_inject_internal_delays": "0.000000", 2024-02-05T04:53:19.784 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_inject_network_congestion": "0", 2024-02-05T04:53:19.785 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_inject_socket_failures": "2500", 2024-02-05T04:53:19.785 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_learn_addr_from_peer": "true", 2024-02-05T04:53:19.785 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_max_accept_failures": "4", 2024-02-05T04:53:19.785 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_max_backoff": "15.000000", 2024-02-05T04:53:19.785 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_mon_client_mode": "secure crc", 2024-02-05T04:53:19.785 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_mon_cluster_mode": "secure crc", 2024-02-05T04:53:19.785 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_mon_service_mode": "secure crc", 2024-02-05T04:53:19.785 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_osd_compress_min_size": "1024", 2024-02-05T04:53:19.785 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_osd_compress_mode": "none", 2024-02-05T04:53:19.785 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_osd_compression_algorithm": "snappy", 2024-02-05T04:53:19.785 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-02-05T04:53:19.785 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_pq_min_cost": "65536", 2024-02-05T04:53:19.785 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_public_type": "", 2024-02-05T04:53:19.785 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_service_mode": "crc secure", 2024-02-05T04:53:19.785 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_tcp_listen_backlog": "512", 2024-02-05T04:53:19.786 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_tcp_nodelay": "true", 2024-02-05T04:53:19.786 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-02-05T04:53:19.786 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_tcp_rcvbuf": "0", 2024-02-05T04:53:19.786 INFO:teuthology.orchestra.run.smithi062.stdout: "ms_type": "async+posix", 2024-02-05T04:53:19.786 INFO:teuthology.orchestra.run.smithi062.stdout: "no_config_file": "false", 2024-02-05T04:53:19.786 INFO:teuthology.orchestra.run.smithi062.stdout: "objecter_completion_locks_per_session": "32", 2024-02-05T04:53:19.786 INFO:teuthology.orchestra.run.smithi062.stdout: "objecter_debug_inject_relock_delay": "false", 2024-02-05T04:53:19.786 INFO:teuthology.orchestra.run.smithi062.stdout: "objecter_inflight_op_bytes": "104857600", 2024-02-05T04:53:19.786 INFO:teuthology.orchestra.run.smithi062.stdout: "objecter_inflight_ops": "1024", 2024-02-05T04:53:19.786 INFO:teuthology.orchestra.run.smithi062.stdout: "objecter_inject_no_watch_ping": "false", 2024-02-05T04:53:19.786 INFO:teuthology.orchestra.run.smithi062.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-02-05T04:53:19.786 INFO:teuthology.orchestra.run.smithi062.stdout: "objecter_tick_interval": "5.000000", 2024-02-05T04:53:19.786 INFO:teuthology.orchestra.run.smithi062.stdout: "objecter_timeout": "10.000000", 2024-02-05T04:53:19.786 INFO:teuthology.orchestra.run.smithi062.stdout: "objectstore_blackhole": "false", 2024-02-05T04:53:19.786 INFO:teuthology.orchestra.run.smithi062.stdout: "openssl_engine_opts": "", 2024-02-05T04:53:19.787 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_agent_delay_time": "5.000000", 2024-02-05T04:53:19.787 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_agent_hist_halflife": "1000", 2024-02-05T04:53:19.787 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_agent_max_low_ops": "2", 2024-02-05T04:53:19.787 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_agent_max_ops": "4", 2024-02-05T04:53:19.787 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-02-05T04:53:19.787 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_agent_quantize_effort": "0.100000", 2024-02-05T04:53:19.787 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_agent_slop": "0.020000", 2024-02-05T04:53:19.787 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-02-05T04:53:19.787 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_allow_recovery_below_min_size": "true", 2024-02-05T04:53:19.787 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_asio_thread_count": "2", 2024-02-05T04:53:19.787 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_async_recovery_min_cost": "1", 2024-02-05T04:53:19.787 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_auto_mark_unfound_lost": "false", 2024-02-05T04:53:19.787 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_backfill_retry_interval": "30.000000", 2024-02-05T04:53:19.787 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_backfill_scan_max": "512", 2024-02-05T04:53:19.788 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_backfill_scan_min": "64", 2024-02-05T04:53:19.788 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_backoff_on_degraded": "false", 2024-02-05T04:53:19.788 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_backoff_on_peering": "false", 2024-02-05T04:53:19.788 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_backoff_on_unfound": "true", 2024-02-05T04:53:19.788 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_beacon_report_interval": "300", 2024-02-05T04:53:19.788 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_bench_duration": "30", 2024-02-05T04:53:19.788 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-02-05T04:53:19.788 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_bench_max_block_size": "67108864", 2024-02-05T04:53:19.788 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_bench_small_size_max_iops": "100", 2024-02-05T04:53:19.788 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_blkin_trace_all": "false", 2024-02-05T04:53:19.788 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_blocked_scrub_grace_period": "120", 2024-02-05T04:53:19.788 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-02-05T04:53:19.788 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-02-05T04:53:19.788 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-02-05T04:53:19.788 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_check_for_log_corruption": "false", 2024-02-05T04:53:19.789 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-02-05T04:53:19.789 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-02-05T04:53:19.789 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-02-05T04:53:19.789 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-02-05T04:53:19.789 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_class_update_on_start": "true", 2024-02-05T04:53:19.789 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_client_message_cap": "256", 2024-02-05T04:53:19.789 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_client_message_size_cap": "524288000", 2024-02-05T04:53:19.789 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_client_op_priority": "63", 2024-02-05T04:53:19.789 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_client_watch_timeout": "30", 2024-02-05T04:53:19.789 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_command_max_records": "256", 2024-02-05T04:53:19.789 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_compact_on_start": "false", 2024-02-05T04:53:19.789 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-02-05T04:53:19.789 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_crush_chooseleaf_type": "0", 2024-02-05T04:53:19.789 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_crush_initial_weight": "-1.000000", 2024-02-05T04:53:19.789 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_crush_update_on_start": "true", 2024-02-05T04:53:19.789 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_crush_update_weight_set": "true", 2024-02-05T04:53:19.790 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-02-05T04:53:19.790 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-02-05T04:53:19.790 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-02-05T04:53:19.790 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_debug_drop_ping_duration": "0", 2024-02-05T04:53:19.790 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-02-05T04:53:19.790 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_debug_feed_pullee": "-1", 2024-02-05T04:53:19.790 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-02-05T04:53:19.790 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-02-05T04:53:19.790 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-02-05T04:53:19.790 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_debug_misdirected_ops": "false", 2024-02-05T04:53:19.790 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_debug_no_acting_change": "false", 2024-02-05T04:53:19.790 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_debug_no_purge_strays": "false", 2024-02-05T04:53:19.790 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_debug_op_order": "false", 2024-02-05T04:53:19.790 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_debug_pg_log_writeout": "false", 2024-02-05T04:53:19.791 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_debug_pretend_recovery_active": "false", 2024-02-05T04:53:19.791 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-02-05T04:53:19.791 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-02-05T04:53:19.791 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_debug_shutdown": "false", 2024-02-05T04:53:19.791 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-02-05T04:53:19.791 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-02-05T04:53:19.791 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_debug_trim_objects": "false", 2024-02-05T04:53:19.791 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_debug_verify_cached_snaps": "false", 2024-02-05T04:53:19.791 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_debug_verify_missing_on_start": "false", 2024-02-05T04:53:19.791 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_debug_verify_snaps": "false", 2024-02-05T04:53:19.791 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-02-05T04:53:19.791 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-02-05T04:53:19.791 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_deep_scrub_keys": "1024", 2024-02-05T04:53:19.791 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-02-05T04:53:19.791 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-02-05T04:53:19.792 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-02-05T04:53:19.792 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_deep_scrub_stride": "524288", 2024-02-05T04:53:19.792 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-02-05T04:53:19.792 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_default_data_pool_replay_window": "5", 2024-02-05T04:53:19.792 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_default_notify_timeout": "30", 2024-02-05T04:53:19.792 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_delete_sleep": "0.000000", 2024-02-05T04:53:19.792 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-02-05T04:53:19.792 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-02-05T04:53:19.792 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-02-05T04:53:19.792 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_discard_disconnected_ops": "true", 2024-02-05T04:53:19.792 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_enable_op_tracker": "true", 2024-02-05T04:53:19.792 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-02-05T04:53:19.792 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_extblkdev_plugins": "vdo", 2024-02-05T04:53:19.792 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-02-05T04:53:19.793 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-02-05T04:53:19.793 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_fast_info": "true", 2024-02-05T04:53:19.793 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_fast_shutdown": "true", 2024-02-05T04:53:19.793 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-02-05T04:53:19.793 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_fast_shutdown_timeout": "15", 2024-02-05T04:53:19.793 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-02-05T04:53:19.793 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-02-05T04:53:19.793 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-02-05T04:53:19.793 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_function_tracing": "false", 2024-02-05T04:53:19.793 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_heartbeat_grace": "20", 2024-02-05T04:53:19.793 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_heartbeat_interval": "6", 2024-02-05T04:53:19.793 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-02-05T04:53:19.793 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_heartbeat_min_peers": "10", 2024-02-05T04:53:19.793 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_heartbeat_min_size": "2000", 2024-02-05T04:53:19.794 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_heartbeat_stale": "600", 2024-02-05T04:53:19.794 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-02-05T04:53:19.794 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_hit_set_max_size": "100000", 2024-02-05T04:53:19.794 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_hit_set_min_size": "1000", 2024-02-05T04:53:19.794 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-02-05T04:53:19.794 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-02-05T04:53:19.794 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-02-05T04:53:19.794 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-02-05T04:53:19.794 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-02-05T04:53:19.794 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_journal_flush_on_shutdown": "true", 2024-02-05T04:53:19.794 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_journal_size": "5120", 2024-02-05T04:53:19.794 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_kill_backfill_at": "0", 2024-02-05T04:53:19.794 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_loop_before_reset_tphandle": "64", 2024-02-05T04:53:19.794 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_map_cache_size": "50", 2024-02-05T04:53:19.794 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_map_dedup": "true", 2024-02-05T04:53:19.794 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_map_message_max": "40", 2024-02-05T04:53:19.795 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_map_message_max_bytes": "10485760", 2024-02-05T04:53:19.795 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_map_share_max_epochs": "40", 2024-02-05T04:53:19.795 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_max_attr_name_len": "100", 2024-02-05T04:53:19.795 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_max_attr_size": "0", 2024-02-05T04:53:19.795 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_max_backfills": "1", 2024-02-05T04:53:19.795 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_max_markdown_count": "5", 2024-02-05T04:53:19.795 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_max_markdown_period": "600", 2024-02-05T04:53:19.795 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_max_object_name_len": "2048", 2024-02-05T04:53:19.795 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_max_object_namespace_len": "256", 2024-02-05T04:53:19.795 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_max_object_size": "134217728", 2024-02-05T04:53:19.795 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-02-05T04:53:19.795 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_max_omap_entries_per_request": "1024", 2024-02-05T04:53:19.795 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_max_pg_blocked_by": "16", 2024-02-05T04:53:19.795 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_max_pg_log_entries": "10000", 2024-02-05T04:53:19.796 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-02-05T04:53:19.796 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_max_pgls": "1024", 2024-02-05T04:53:19.796 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_max_push_cost": "8388608", 2024-02-05T04:53:19.796 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_max_push_objects": "10", 2024-02-05T04:53:19.796 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_max_scrubs": "3", 2024-02-05T04:53:19.796 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-02-05T04:53:19.796 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_max_trimming_pgs": "2", 2024-02-05T04:53:19.796 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_max_write_op_reply_len": "64", 2024-02-05T04:53:19.796 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_max_write_size": "90", 2024-02-05T04:53:19.796 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-02-05T04:53:19.796 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-02-05T04:53:19.796 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-02-05T04:53:19.796 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-02-05T04:53:19.796 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-02-05T04:53:19.797 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-02-05T04:53:19.797 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-02-05T04:53:19.797 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_mclock_override_recovery_settings": "false", 2024-02-05T04:53:19.797 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_mclock_profile": "balanced", 2024-02-05T04:53:19.797 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-02-05T04:53:19.797 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-02-05T04:53:19.797 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-02-05T04:53:19.797 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-02-05T04:53:19.797 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-02-05T04:53:19.797 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-02-05T04:53:19.797 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-02-05T04:53:19.797 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-02-05T04:53:19.797 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-02-05T04:53:19.797 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-02-05T04:53:19.798 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_mclock_skip_benchmark": "false", 2024-02-05T04:53:19.798 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_memory_base": "805306368", 2024-02-05T04:53:19.798 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_memory_cache_min": "134217728", 2024-02-05T04:53:19.798 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-02-05T04:53:19.798 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-02-05T04:53:19.798 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_memory_target": "4294967296", 2024-02-05T04:53:19.798 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_memory_target_autotune": "false", 2024-02-05T04:53:19.798 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-02-05T04:53:19.798 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_min_pg_log_entries": "250", 2024-02-05T04:53:19.798 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_min_recovery_priority": "0", 2024-02-05T04:53:19.798 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_mon_heartbeat_interval": "30", 2024-02-05T04:53:19.798 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-02-05T04:53:19.798 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_mon_report_interval": "5", 2024-02-05T04:53:19.798 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_mon_report_max_in_flight": "2", 2024-02-05T04:53:19.798 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-02-05T04:53:19.799 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_num_cache_shards": "32", 2024-02-05T04:53:19.799 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_num_op_tracker_shard": "32", 2024-02-05T04:53:19.799 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_numa_auto_affinity": "true", 2024-02-05T04:53:19.799 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_numa_node": "-1", 2024-02-05T04:53:19.799 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_numa_prefer_iface": "true", 2024-02-05T04:53:19.799 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_object_clean_region_max_num_intervals": "1000", 2024-02-05T04:53:19.799 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_objecter_finishers": "1", 2024-02-05T04:53:19.799 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_objectstore": "bluestore", 2024-02-05T04:53:19.799 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_objectstore_fuse": "false", 2024-02-05T04:53:19.799 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_objectstore_tracing": "false", 2024-02-05T04:53:19.799 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_op_complaint_time": "30.000000", 2024-02-05T04:53:19.799 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_op_history_duration": "600", 2024-02-05T04:53:19.799 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_op_history_size": "20", 2024-02-05T04:53:19.799 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_op_history_slow_op_size": "20", 2024-02-05T04:53:19.799 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-02-05T04:53:19.800 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_op_log_threshold": "5", 2024-02-05T04:53:19.800 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_op_num_shards": "0", 2024-02-05T04:53:19.800 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_op_num_shards_hdd": "5", 2024-02-05T04:53:19.800 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_op_num_shards_ssd": "8", 2024-02-05T04:53:19.800 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_op_num_threads_per_shard": "0", 2024-02-05T04:53:19.800 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-02-05T04:53:19.800 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-02-05T04:53:19.800 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-02-05T04:53:19.800 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_op_pq_min_cost": "65536", 2024-02-05T04:53:19.800 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_op_queue": "mclock_scheduler", 2024-02-05T04:53:19.800 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_op_queue_cut_off": "high", 2024-02-05T04:53:19.800 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_op_thread_suicide_timeout": "150", 2024-02-05T04:53:19.800 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_op_thread_timeout": "15", 2024-02-05T04:53:19.800 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_open_classes_on_start": "true", 2024-02-05T04:53:19.800 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_os_flags": "0", 2024-02-05T04:53:19.801 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_peering_op_priority": "255", 2024-02-05T04:53:19.801 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_pg_delete_cost": "1048576", 2024-02-05T04:53:19.801 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_pg_delete_priority": "5", 2024-02-05T04:53:19.801 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-02-05T04:53:19.801 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-02-05T04:53:19.801 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_pg_log_dups_tracked": "3000", 2024-02-05T04:53:19.801 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_pg_log_trim_max": "10000", 2024-02-05T04:53:19.801 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_pg_log_trim_min": "100", 2024-02-05T04:53:19.801 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-02-05T04:53:19.801 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_pg_object_context_cache_count": "64", 2024-02-05T04:53:19.801 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_pg_stat_report_interval_max": "500", 2024-02-05T04:53:19.801 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-02-05T04:53:19.801 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-02-05T04:53:19.801 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-02-05T04:53:19.801 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-02-05T04:53:19.801 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-02-05T04:53:19.802 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-02-05T04:53:19.802 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_pool_default_crimson": "false", 2024-02-05T04:53:19.802 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_pool_default_crush_rule": "-1", 2024-02-05T04:53:19.802 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_pool_default_ec_fast_read": "false", 2024-02-05T04:53:19.802 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-02-05T04:53:19.802 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_pool_default_flag_bulk": "false", 2024-02-05T04:53:19.802 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-02-05T04:53:19.802 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_pool_default_flag_nodelete": "false", 2024-02-05T04:53:19.802 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-02-05T04:53:19.802 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-02-05T04:53:19.802 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_pool_default_flags": "0", 2024-02-05T04:53:19.802 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-02-05T04:53:19.802 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_pool_default_min_size": "0", 2024-02-05T04:53:19.802 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-02-05T04:53:19.802 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_pool_default_pg_num": "32", 2024-02-05T04:53:19.803 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_pool_default_pgp_num": "0", 2024-02-05T04:53:19.803 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-02-05T04:53:19.803 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_pool_default_size": "2", 2024-02-05T04:53:19.803 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_pool_default_type": "replicated", 2024-02-05T04:53:19.803 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-02-05T04:53:19.803 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_pool_use_gmt_hitset": "true", 2024-02-05T04:53:19.803 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_push_per_object_cost": "1000", 2024-02-05T04:53:19.803 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_read_ec_check_for_errors": "false", 2024-02-05T04:53:19.803 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_recover_clone_overlap": "true", 2024-02-05T04:53:19.803 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_recover_clone_overlap_limit": "10", 2024-02-05T04:53:19.803 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_recovery_cost": "20971520", 2024-02-05T04:53:19.803 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_recovery_delay_start": "0.000000", 2024-02-05T04:53:19.803 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_recovery_max_active": "0", 2024-02-05T04:53:19.803 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_recovery_max_active_hdd": "3", 2024-02-05T04:53:19.803 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_recovery_max_active_ssd": "10", 2024-02-05T04:53:19.803 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_recovery_max_chunk": "8388608", 2024-02-05T04:53:19.804 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-02-05T04:53:19.804 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_recovery_max_single_start": "1", 2024-02-05T04:53:19.804 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_recovery_op_priority": "3", 2024-02-05T04:53:19.804 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_recovery_op_warn_multiple": "16", 2024-02-05T04:53:19.804 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_recovery_priority": "5", 2024-02-05T04:53:19.804 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_recovery_retry_interval": "30.000000", 2024-02-05T04:53:19.804 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_recovery_sleep": "0.000000", 2024-02-05T04:53:19.804 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-02-05T04:53:19.804 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-02-05T04:53:19.804 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-02-05T04:53:19.804 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_repair_during_recovery": "false", 2024-02-05T04:53:19.804 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_requested_scrub_priority": "120", 2024-02-05T04:53:19.804 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-02-05T04:53:19.805 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_rollback_to_cluster_snap": "", 2024-02-05T04:53:19.805 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_scrub_auto_repair": "false", 2024-02-05T04:53:19.805 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-02-05T04:53:19.805 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-02-05T04:53:19.805 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_scrub_begin_hour": "0", 2024-02-05T04:53:19.805 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_scrub_begin_week_day": "0", 2024-02-05T04:53:19.805 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_scrub_chunk_max": "25", 2024-02-05T04:53:19.805 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_scrub_chunk_min": "5", 2024-02-05T04:53:19.805 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_scrub_cost": "52428800", 2024-02-05T04:53:19.805 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_scrub_during_recovery": "false", 2024-02-05T04:53:19.805 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_scrub_end_hour": "0", 2024-02-05T04:53:19.805 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_scrub_end_week_day": "0", 2024-02-05T04:53:19.805 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_scrub_event_cost": "4096", 2024-02-05T04:53:19.806 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-02-05T04:53:19.806 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-02-05T04:53:19.806 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_scrub_invalid_stats": "true", 2024-02-05T04:53:19.806 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_scrub_load_threshold": "0.500000", 2024-02-05T04:53:19.806 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_scrub_max_interval": "604800.000000", 2024-02-05T04:53:19.806 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_scrub_max_preemptions": "5", 2024-02-05T04:53:19.806 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_scrub_min_interval": "86400.000000", 2024-02-05T04:53:19.806 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_scrub_priority": "5", 2024-02-05T04:53:19.806 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_scrub_reservation_timeout": "300000", 2024-02-05T04:53:19.806 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_scrub_sleep": "0.000000", 2024-02-05T04:53:19.806 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_scrub_slow_reservation_response": "30000", 2024-02-05T04:53:19.806 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-02-05T04:53:19.806 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-02-05T04:53:19.806 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_shutdown_pgref_assert": "false", 2024-02-05T04:53:19.807 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_skip_data_digest": "false", 2024-02-05T04:53:19.807 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_smart_report_timeout": "5", 2024-02-05T04:53:19.807 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_snap_trim_cost": "1048576", 2024-02-05T04:53:19.807 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_snap_trim_priority": "5", 2024-02-05T04:53:19.807 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_snap_trim_sleep": "0.000000", 2024-02-05T04:53:19.807 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-02-05T04:53:19.807 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-02-05T04:53:19.807 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-02-05T04:53:19.807 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-02-05T04:53:19.807 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_stats_update_period_scrubbing": "15", 2024-02-05T04:53:19.807 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-02-05T04:53:19.807 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_target_transaction_size": "30", 2024-02-05T04:53:19.807 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-02-05T04:53:19.807 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-02-05T04:53:19.808 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-02-05T04:53:19.808 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-02-05T04:53:19.808 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-02-05T04:53:19.808 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-02-05T04:53:19.808 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-02-05T04:53:19.808 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_tier_default_cache_mode": "writeback", 2024-02-05T04:53:19.808 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-02-05T04:53:19.808 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-02-05T04:53:19.808 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_tracing": "false", 2024-02-05T04:53:19.808 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_use_stale_snap": "false", 2024-02-05T04:53:19.808 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-02-05T04:53:19.808 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_warn_on_max_pglog_size": "4096", 2024-02-05T04:53:19.808 INFO:teuthology.orchestra.run.smithi062.stdout: "osdc_blkin_trace_all": "false", 2024-02-05T04:53:19.808 INFO:teuthology.orchestra.run.smithi062.stdout: "paxos_kill_at": "0", 2024-02-05T04:53:19.809 INFO:teuthology.orchestra.run.smithi062.stdout: "paxos_max_join_drift": "10", 2024-02-05T04:53:19.809 INFO:teuthology.orchestra.run.smithi062.stdout: "paxos_min": "500", 2024-02-05T04:53:19.809 INFO:teuthology.orchestra.run.smithi062.stdout: "paxos_min_wait": "0.050000", 2024-02-05T04:53:19.809 INFO:teuthology.orchestra.run.smithi062.stdout: "paxos_propose_interval": "1.000000", 2024-02-05T04:53:19.809 INFO:teuthology.orchestra.run.smithi062.stdout: "paxos_service_trim_max": "500", 2024-02-05T04:53:19.809 INFO:teuthology.orchestra.run.smithi062.stdout: "paxos_service_trim_max_multiplier": "20", 2024-02-05T04:53:19.809 INFO:teuthology.orchestra.run.smithi062.stdout: "paxos_service_trim_min": "250", 2024-02-05T04:53:19.809 INFO:teuthology.orchestra.run.smithi062.stdout: "paxos_stash_full_interval": "25", 2024-02-05T04:53:19.809 INFO:teuthology.orchestra.run.smithi062.stdout: "paxos_trim_max": "500", 2024-02-05T04:53:19.809 INFO:teuthology.orchestra.run.smithi062.stdout: "paxos_trim_min": "250", 2024-02-05T04:53:19.809 INFO:teuthology.orchestra.run.smithi062.stdout: "perf": "true", 2024-02-05T04:53:19.809 INFO:teuthology.orchestra.run.smithi062.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2024-02-05T04:53:19.809 INFO:teuthology.orchestra.run.smithi062.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-02-05T04:53:19.809 INFO:teuthology.orchestra.run.smithi062.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-02-05T04:53:19.810 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addr": "-", 2024-02-05T04:53:19.810 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addrv": "", 2024-02-05T04:53:19.810 INFO:teuthology.orchestra.run.smithi062.stdout: "public_bind_addr": "-", 2024-02-05T04:53:19.810 INFO:teuthology.orchestra.run.smithi062.stdout: "public_network": "", 2024-02-05T04:53:19.810 INFO:teuthology.orchestra.run.smithi062.stdout: "public_network_interface": "", 2024-02-05T04:53:19.810 INFO:teuthology.orchestra.run.smithi062.stdout: "qat_compressor_enabled": "false", 2024-02-05T04:53:19.810 INFO:teuthology.orchestra.run.smithi062.stdout: "qat_compressor_session_max_number": "256", 2024-02-05T04:53:19.810 INFO:teuthology.orchestra.run.smithi062.stdout: "rados_mon_op_timeout": "0", 2024-02-05T04:53:19.810 INFO:teuthology.orchestra.run.smithi062.stdout: "rados_osd_op_timeout": "0", 2024-02-05T04:53:19.810 INFO:teuthology.orchestra.run.smithi062.stdout: "rados_tracing": "false", 2024-02-05T04:53:19.810 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_atime_update_interval": "60", 2024-02-05T04:53:19.810 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-02-05T04:53:19.810 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_balance_parent_reads": "false", 2024-02-05T04:53:19.810 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_balance_snap_reads": "false", 2024-02-05T04:53:19.810 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_blkin_trace_all": "false", 2024-02-05T04:53:19.811 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_blocklist_expire_seconds": "0", 2024-02-05T04:53:19.811 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_blocklist_on_break_lock": "true", 2024-02-05T04:53:19.811 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_cache": "true", 2024-02-05T04:53:19.811 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_cache_block_writes_upfront": "false", 2024-02-05T04:53:19.811 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_cache_max_dirty": "25165824", 2024-02-05T04:53:19.811 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-02-05T04:53:19.811 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_cache_max_dirty_object": "0", 2024-02-05T04:53:19.811 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_cache_policy": "writearound", 2024-02-05T04:53:19.811 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_cache_size": "33554432", 2024-02-05T04:53:19.811 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_cache_target_dirty": "16777216", 2024-02-05T04:53:19.811 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-02-05T04:53:19.811 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_clone_copy_on_read": "false", 2024-02-05T04:53:19.811 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_compression_hint": "none", 2024-02-05T04:53:19.811 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_concurrent_management_ops": "10", 2024-02-05T04:53:19.812 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-02-05T04:53:19.812 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_default_clone_format": "auto", 2024-02-05T04:53:19.812 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_default_data_pool": "", 2024-02-05T04:53:19.812 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_default_features": "61", 2024-02-05T04:53:19.812 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_default_format": "2", 2024-02-05T04:53:19.812 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_default_map_options": "", 2024-02-05T04:53:19.812 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_default_order": "22", 2024-02-05T04:53:19.812 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_default_pool": "rbd", 2024-02-05T04:53:19.812 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-02-05T04:53:19.812 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_default_stripe_count": "0", 2024-02-05T04:53:19.812 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_default_stripe_unit": "0", 2024-02-05T04:53:19.812 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_disable_zero_copy_writes": "true", 2024-02-05T04:53:19.812 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_discard_granularity_bytes": "65536", 2024-02-05T04:53:19.812 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-02-05T04:53:19.812 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_enable_alloc_hint": "true", 2024-02-05T04:53:19.813 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-02-05T04:53:19.813 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_io_scheduler": "simple", 2024-02-05T04:53:19.813 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-02-05T04:53:19.813 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_journal_commit_age": "5.000000", 2024-02-05T04:53:19.813 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-02-05T04:53:19.813 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-02-05T04:53:19.813 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-02-05T04:53:19.813 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-02-05T04:53:19.813 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_journal_object_flush_interval": "0", 2024-02-05T04:53:19.813 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-02-05T04:53:19.813 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-02-05T04:53:19.813 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_journal_order": "24", 2024-02-05T04:53:19.813 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_journal_pool": "", 2024-02-05T04:53:19.813 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_journal_splay_width": "4", 2024-02-05T04:53:19.813 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_localize_parent_reads": "false", 2024-02-05T04:53:19.814 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_localize_snap_reads": "false", 2024-02-05T04:53:19.814 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-02-05T04:53:19.814 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-02-05T04:53:19.814 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-02-05T04:53:19.814 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-02-05T04:53:19.814 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-02-05T04:53:19.814 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-02-05T04:53:19.814 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_mirror_image_policy_type": "simple", 2024-02-05T04:53:19.814 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-02-05T04:53:19.814 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-02-05T04:53:19.814 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-02-05T04:53:19.814 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-02-05T04:53:19.814 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-02-05T04:53:19.814 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-02-05T04:53:19.815 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-02-05T04:53:19.815 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_mirror_memory_autotune": "true", 2024-02-05T04:53:19.815 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_mirror_memory_base": "805306368", 2024-02-05T04:53:19.815 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-02-05T04:53:19.815 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-02-05T04:53:19.815 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-02-05T04:53:19.815 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-02-05T04:53:19.815 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_mirror_memory_target": "4294967296", 2024-02-05T04:53:19.815 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-02-05T04:53:19.815 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-02-05T04:53:19.815 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-02-05T04:53:19.815 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_mirroring_delete_delay": "0", 2024-02-05T04:53:19.815 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-02-05T04:53:19.815 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_mirroring_replay_delay": "0", 2024-02-05T04:53:19.815 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-02-05T04:53:19.816 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-02-05T04:53:19.816 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_move_to_trash_on_remove": "false", 2024-02-05T04:53:19.816 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-02-05T04:53:19.816 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_mtime_update_interval": "60", 2024-02-05T04:53:19.816 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_non_blocking_aio": "true", 2024-02-05T04:53:19.816 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_op_thread_timeout": "60", 2024-02-05T04:53:19.816 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_op_threads": "1", 2024-02-05T04:53:19.816 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_parent_cache_enabled": "false", 2024-02-05T04:53:19.816 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_persistent_cache_mode": "disabled", 2024-02-05T04:53:19.816 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_persistent_cache_path": "/tmp", 2024-02-05T04:53:19.816 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_persistent_cache_size": "1073741824", 2024-02-05T04:53:19.816 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_plugins": "", 2024-02-05T04:53:19.816 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_qos_bps_burst": "0", 2024-02-05T04:53:19.816 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-02-05T04:53:19.817 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_qos_bps_limit": "0", 2024-02-05T04:53:19.817 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_qos_exclude_ops": "0", 2024-02-05T04:53:19.817 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_qos_iops_burst": "0", 2024-02-05T04:53:19.817 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-02-05T04:53:19.817 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_qos_iops_limit": "0", 2024-02-05T04:53:19.817 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_qos_read_bps_burst": "0", 2024-02-05T04:53:19.817 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-02-05T04:53:19.817 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_qos_read_bps_limit": "0", 2024-02-05T04:53:19.817 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_qos_read_iops_burst": "0", 2024-02-05T04:53:19.817 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-02-05T04:53:19.817 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_qos_read_iops_limit": "0", 2024-02-05T04:53:19.817 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_qos_schedule_tick_min": "50", 2024-02-05T04:53:19.817 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_qos_write_bps_burst": "0", 2024-02-05T04:53:19.817 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-02-05T04:53:19.817 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_qos_write_bps_limit": "0", 2024-02-05T04:53:19.818 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_qos_write_iops_burst": "0", 2024-02-05T04:53:19.818 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-02-05T04:53:19.818 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_qos_write_iops_limit": "0", 2024-02-05T04:53:19.818 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_quiesce_notification_attempts": "10", 2024-02-05T04:53:19.818 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_read_from_replica_policy": "default", 2024-02-05T04:53:19.818 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-02-05T04:53:19.818 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_readahead_max_bytes": "524288", 2024-02-05T04:53:19.818 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_readahead_trigger_requests": "10", 2024-02-05T04:53:19.818 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_request_timed_out_seconds": "30", 2024-02-05T04:53:19.818 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_skip_partial_discard": "true", 2024-02-05T04:53:19.818 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-02-05T04:53:19.818 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_tracing": "false", 2024-02-05T04:53:19.818 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_validate_names": "true", 2024-02-05T04:53:19.818 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_validate_pool": "true", 2024-02-05T04:53:19.819 INFO:teuthology.orchestra.run.smithi062.stdout: "restapi_base_url": "", 2024-02-05T04:53:19.819 INFO:teuthology.orchestra.run.smithi062.stdout: "restapi_log_level": "", 2024-02-05T04:53:19.819 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_acl_grants_max_num": "100", 2024-02-05T04:53:19.819 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_admin_entry": "admin", 2024-02-05T04:53:19.819 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-02-05T04:53:19.819 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_backend_store": "rados", 2024-02-05T04:53:19.819 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_barbican_url": "", 2024-02-05T04:53:19.819 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_beast_enable_async": "true", 2024-02-05T04:53:19.819 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_bucket_counters_cache": "false", 2024-02-05T04:53:19.819 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-02-05T04:53:19.819 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-02-05T04:53:19.819 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-02-05T04:53:19.819 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_bucket_index_max_aio": "128", 2024-02-05T04:53:19.819 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-02-05T04:53:19.819 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-02-05T04:53:19.819 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_bucket_quota_ttl": "600", 2024-02-05T04:53:19.820 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-02-05T04:53:19.820 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_cache_enabled": "true", 2024-02-05T04:53:19.820 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_cache_expiry_interval": "900", 2024-02-05T04:53:19.820 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_cache_lru_size": "10000", 2024-02-05T04:53:19.820 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_config_store": "rados", 2024-02-05T04:53:19.820 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_content_length_compat": "false", 2024-02-05T04:53:19.820 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_copy_obj_progress": "true", 2024-02-05T04:53:19.820 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-02-05T04:53:19.820 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_cors_rules_max_num": "100", 2024-02-05T04:53:19.820 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_cross_domain_policy": "", 2024-02-05T04:53:19.820 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_crypt_default_encryption_key": "", 2024-02-05T04:53:19.820 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_crypt_kmip_addr": "", 2024-02-05T04:53:19.820 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_crypt_kmip_ca_path": "", 2024-02-05T04:53:19.820 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_crypt_kmip_client_cert": "", 2024-02-05T04:53:19.821 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_crypt_kmip_client_key": "", 2024-02-05T04:53:19.821 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-02-05T04:53:19.821 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_crypt_kmip_password": "", 2024-02-05T04:53:19.821 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-02-05T04:53:19.821 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_crypt_kmip_username": "", 2024-02-05T04:53:19.821 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_crypt_require_ssl": "true", 2024-02-05T04:53:19.821 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-02-05T04:53:19.821 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-02-05T04:53:19.821 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-02-05T04:53:19.821 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-02-05T04:53:19.821 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-02-05T04:53:19.821 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-02-05T04:53:19.821 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-02-05T04:53:19.822 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-02-05T04:53:19.822 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-02-05T04:53:19.822 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-02-05T04:53:19.822 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-02-05T04:53:19.822 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-02-05T04:53:19.822 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-02-05T04:53:19.822 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-02-05T04:53:19.822 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_crypt_suppress_logs": "true", 2024-02-05T04:53:19.822 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_crypt_vault_addr": "", 2024-02-05T04:53:19.822 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_crypt_vault_auth": "token", 2024-02-05T04:53:19.822 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_crypt_vault_namespace": "", 2024-02-05T04:53:19.822 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_crypt_vault_prefix": "", 2024-02-05T04:53:19.822 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-02-05T04:53:19.822 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-02-05T04:53:19.823 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-02-05T04:53:19.823 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-02-05T04:53:19.823 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_crypt_vault_token_file": "", 2024-02-05T04:53:19.823 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-02-05T04:53:19.823 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_curl_buffersize": "524288", 2024-02-05T04:53:19.823 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_curl_low_speed_limit": "1024", 2024-02-05T04:53:19.823 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_curl_low_speed_time": "300", 2024-02-05T04:53:19.823 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_curl_tcp_keepalive": "0", 2024-02-05T04:53:19.823 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-02-05T04:53:19.823 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-02-05T04:53:19.823 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-02-05T04:53:19.823 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-02-05T04:53:19.823 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-02-05T04:53:19.824 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_d3n_l1_fadvise": "4", 2024-02-05T04:53:19.824 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-02-05T04:53:19.824 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-02-05T04:53:19.824 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-02-05T04:53:19.824 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_d4n_host": "127.0.0.1", 2024-02-05T04:53:19.824 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_d4n_port": "6379", 2024-02-05T04:53:19.824 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-02-05T04:53:19.824 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_data_log_changes_size": "1000", 2024-02-05T04:53:19.824 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_data_log_num_shards": "128", 2024-02-05T04:53:19.824 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-02-05T04:53:19.824 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_data_log_window": "30", 2024-02-05T04:53:19.824 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_data_notify_interval_msec": "0", 2024-02-05T04:53:19.824 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_data_sync_poll_interval": "20", 2024-02-05T04:53:19.824 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_data_sync_spawn_window": "20", 2024-02-05T04:53:19.825 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-02-05T04:53:19.825 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-02-05T04:53:19.825 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_default_data_log_backing": "fifo", 2024-02-05T04:53:19.825 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-02-05T04:53:19.825 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_default_region_info_oid": "default.region", 2024-02-05T04:53:19.825 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-02-05T04:53:19.825 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-02-05T04:53:19.825 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_defer_to_bucket_acls": "", 2024-02-05T04:53:19.825 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-02-05T04:53:19.825 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-02-05T04:53:19.825 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-02-05T04:53:19.825 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-02-05T04:53:19.825 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-02-05T04:53:19.825 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-02-05T04:53:19.826 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-02-05T04:53:19.826 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-02-05T04:53:19.826 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_dmclock_data_res": "500.000000", 2024-02-05T04:53:19.826 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-02-05T04:53:19.826 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-02-05T04:53:19.826 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-02-05T04:53:19.826 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-02-05T04:53:19.826 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_dns_name": "", 2024-02-05T04:53:19.826 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_dns_s3website_name": "", 2024-02-05T04:53:19.826 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_dynamic_resharding": "true", 2024-02-05T04:53:19.826 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-02-05T04:53:19.826 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_enable_gc_threads": "true", 2024-02-05T04:53:19.826 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_enable_lc_threads": "true", 2024-02-05T04:53:19.826 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_enable_ops_log": "false", 2024-02-05T04:53:19.826 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_enable_quota_threads": "true", 2024-02-05T04:53:19.827 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_enable_static_website": "false", 2024-02-05T04:53:19.827 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_enable_usage_log": "false", 2024-02-05T04:53:19.827 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_enforce_swift_acls": "true", 2024-02-05T04:53:19.827 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_exit_timeout_secs": "120", 2024-02-05T04:53:19.827 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_expose_bucket": "false", 2024-02-05T04:53:19.827 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_extended_http_attrs": "", 2024-02-05T04:53:19.827 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_filter": "none", 2024-02-05T04:53:19.827 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-02-05T04:53:19.827 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_frontends": "beast port=7480", 2024-02-05T04:53:19.827 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_gc_max_concurrent_io": "10", 2024-02-05T04:53:19.827 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_gc_max_deferred": "50", 2024-02-05T04:53:19.827 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-02-05T04:53:19.827 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_gc_max_objs": "32", 2024-02-05T04:53:19.827 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_gc_max_queue_size": "134213632", 2024-02-05T04:53:19.827 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_gc_max_trim_chunk": "16", 2024-02-05T04:53:19.828 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_gc_obj_min_wait": "7200", 2024-02-05T04:53:19.828 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_gc_processor_max_time": "3600", 2024-02-05T04:53:19.828 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_gc_processor_period": "3600", 2024-02-05T04:53:19.828 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-02-05T04:53:19.828 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_get_obj_window_size": "16777216", 2024-02-05T04:53:19.828 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_healthcheck_disabling_path": "", 2024-02-05T04:53:19.828 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_ignore_get_invalid_range": "false", 2024-02-05T04:53:19.828 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_init_timeout": "300", 2024-02-05T04:53:19.828 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-02-05T04:53:19.828 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-02-05T04:53:19.828 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_kafka_connection_idle": "30", 2024-02-05T04:53:19.828 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_kafka_sleep_timeout": "10", 2024-02-05T04:53:19.828 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-02-05T04:53:19.828 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-02-05T04:53:19.829 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-02-05T04:53:19.829 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_keystone_admin_domain": "", 2024-02-05T04:53:19.829 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_keystone_admin_password": "", 2024-02-05T04:53:19.829 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_keystone_admin_password_path": "", 2024-02-05T04:53:19.829 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_keystone_admin_project": "", 2024-02-05T04:53:19.829 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_keystone_admin_tenant": "", 2024-02-05T04:53:19.829 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_keystone_admin_token": "", 2024-02-05T04:53:19.829 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_keystone_admin_token_path": "", 2024-02-05T04:53:19.829 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_keystone_admin_user": "", 2024-02-05T04:53:19.829 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_keystone_api_version": "2", 2024-02-05T04:53:19.829 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_keystone_barbican_domain": "", 2024-02-05T04:53:19.829 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_keystone_barbican_password": "", 2024-02-05T04:53:19.829 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_keystone_barbican_project": "", 2024-02-05T04:53:19.829 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_keystone_barbican_tenant": "", 2024-02-05T04:53:19.830 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_keystone_barbican_user": "", 2024-02-05T04:53:19.830 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-02-05T04:53:19.830 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_keystone_implicit_tenants": "false", 2024-02-05T04:53:19.830 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-02-05T04:53:19.830 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_keystone_service_token_enabled": "false", 2024-02-05T04:53:19.830 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_keystone_token_cache_size": "10000", 2024-02-05T04:53:19.830 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_keystone_url": "", 2024-02-05T04:53:19.830 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_keystone_verify_ssl": "true", 2024-02-05T04:53:19.830 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_lc_debug_interval": "-1", 2024-02-05T04:53:19.830 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_lc_lock_max_time": "90", 2024-02-05T04:53:19.830 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_lc_max_objs": "32", 2024-02-05T04:53:19.830 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_lc_max_rules": "1000", 2024-02-05T04:53:19.830 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_lc_max_worker": "3", 2024-02-05T04:53:19.831 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_lc_max_wp_worker": "3", 2024-02-05T04:53:19.831 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_lc_thread_delay": "0", 2024-02-05T04:53:19.831 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-02-05T04:53:19.831 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_ldap_dnattr": "uid", 2024-02-05T04:53:19.831 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-02-05T04:53:19.831 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_ldap_searchfilter": "", 2024-02-05T04:53:19.831 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-02-05T04:53:19.831 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_ldap_uri": "ldaps://", 2024-02-05T04:53:19.831 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-02-05T04:53:19.831 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-02-05T04:53:19.831 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-02-05T04:53:19.831 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_log_http_headers": "", 2024-02-05T04:53:19.831 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_log_nonexistent_bucket": "false", 2024-02-05T04:53:19.832 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-02-05T04:53:19.832 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_log_object_name_utc": "false", 2024-02-05T04:53:19.832 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-02-05T04:53:19.832 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-02-05T04:53:19.832 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_max_attr_name_len": "0", 2024-02-05T04:53:19.832 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_max_attr_size": "0", 2024-02-05T04:53:19.832 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_max_attrs_num_in_req": "0", 2024-02-05T04:53:19.832 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_max_chunk_size": "4194304", 2024-02-05T04:53:19.832 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_max_concurrent_requests": "1024", 2024-02-05T04:53:19.832 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-02-05T04:53:19.832 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_max_dynamic_shards": "1999", 2024-02-05T04:53:19.833 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_max_listing_results": "1000", 2024-02-05T04:53:19.833 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_max_notify_retries": "10", 2024-02-05T04:53:19.833 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_max_objs_per_shard": "100000", 2024-02-05T04:53:19.833 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_max_put_param_size": "1048576", 2024-02-05T04:53:19.833 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_max_put_size": "5368709120", 2024-02-05T04:53:19.833 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_max_slo_entries": "1000", 2024-02-05T04:53:19.833 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_md_log_max_shards": "64", 2024-02-05T04:53:19.833 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_md_notify_interval_msec": "200", 2024-02-05T04:53:19.833 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_meta_sync_poll_interval": "20", 2024-02-05T04:53:19.833 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_meta_sync_spawn_window": "20", 2024-02-05T04:53:19.833 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-02-05T04:53:19.833 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_mp_lock_max_time": "600", 2024-02-05T04:53:19.834 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-02-05T04:53:19.834 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_multipart_min_part_size": "5242880", 2024-02-05T04:53:19.834 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-02-05T04:53:19.834 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-02-05T04:53:19.834 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_nfs_fhcache_size": "2017", 2024-02-05T04:53:19.834 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-02-05T04:53:19.834 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-02-05T04:53:19.834 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_nfs_lru_lanes": "5", 2024-02-05T04:53:19.834 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_nfs_max_gc": "300", 2024-02-05T04:53:19.834 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-02-05T04:53:19.834 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_nfs_run_gc_threads": "false", 2024-02-05T04:53:19.834 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_nfs_run_lc_threads": "false", 2024-02-05T04:53:19.834 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_nfs_run_quota_threads": "false", 2024-02-05T04:53:19.834 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_nfs_run_sync_thread": "false", 2024-02-05T04:53:19.835 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-02-05T04:53:19.835 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-02-05T04:53:19.835 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_num_async_rados_threads": "32", 2024-02-05T04:53:19.835 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_num_control_oids": "8", 2024-02-05T04:53:19.835 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_numa_node": "-1", 2024-02-05T04:53:19.835 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_obj_stripe_size": "4194304", 2024-02-05T04:53:19.835 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-02-05T04:53:19.835 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_objexp_chunk_size": "100", 2024-02-05T04:53:19.835 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_objexp_gc_interval": "600", 2024-02-05T04:53:19.835 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_objexp_hints_num_shards": "127", 2024-02-05T04:53:19.835 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-02-05T04:53:19.835 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-02-05T04:53:19.835 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_op_thread_timeout": "600", 2024-02-05T04:53:19.835 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_op_tracing": "false", 2024-02-05T04:53:19.835 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_opa_token": "", 2024-02-05T04:53:19.836 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_opa_url": "", 2024-02-05T04:53:19.836 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_opa_verify_ssl": "true", 2024-02-05T04:53:19.836 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-02-05T04:53:19.836 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-02-05T04:53:19.836 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_ops_log_rados": "false", 2024-02-05T04:53:19.836 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_ops_log_socket_path": "", 2024-02-05T04:53:19.836 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-02-05T04:53:19.836 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-02-05T04:53:19.836 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-02-05T04:53:19.836 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_period_push_interval": "2.000000", 2024-02-05T04:53:19.836 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_period_push_interval_max": "30.000000", 2024-02-05T04:53:19.836 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_period_root_pool": ".rgw.root", 2024-02-05T04:53:19.836 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-02-05T04:53:19.836 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-02-05T04:53:19.836 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_posix_cache_lanes": "3", 2024-02-05T04:53:19.837 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-02-05T04:53:19.837 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_posix_cache_max_buckets": "100", 2024-02-05T04:53:19.837 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_posix_cache_partitions": "3", 2024-02-05T04:53:19.837 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-02-05T04:53:19.837 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_print_continue": "true", 2024-02-05T04:53:19.837 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_print_prohibited_content_length": "false", 2024-02-05T04:53:19.837 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-02-05T04:53:19.837 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-02-05T04:53:19.837 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-02-05T04:53:19.837 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-02-05T04:53:19.837 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_rados_tracing": "false", 2024-02-05T04:53:19.837 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_realm": "", 2024-02-05T04:53:19.837 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_realm_id": "", 2024-02-05T04:53:19.837 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-02-05T04:53:19.837 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_region": "", 2024-02-05T04:53:19.838 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_region_root_pool": ".rgw.root", 2024-02-05T04:53:19.838 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_relaxed_region_enforcement": "false", 2024-02-05T04:53:19.838 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-02-05T04:53:19.838 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-02-05T04:53:19.838 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_request_uri": "", 2024-02-05T04:53:19.838 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_reshard_batch_size": "64", 2024-02-05T04:53:19.838 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-02-05T04:53:19.838 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_reshard_max_aio": "128", 2024-02-05T04:53:19.838 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_reshard_num_logs": "16", 2024-02-05T04:53:19.838 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_reshard_thread_interval": "600", 2024-02-05T04:53:19.838 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_resolve_cname": "false", 2024-02-05T04:53:19.838 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_rest_getusage_op_compat": "false", 2024-02-05T04:53:19.838 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_run_sync_thread": "true", 2024-02-05T04:53:19.838 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-02-05T04:53:19.839 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_s3_auth_use_keystone": "false", 2024-02-05T04:53:19.839 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_s3_auth_use_ldap": "false", 2024-02-05T04:53:19.839 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_s3_auth_use_rados": "true", 2024-02-05T04:53:19.839 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_s3_auth_use_sts": "false", 2024-02-05T04:53:19.839 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-02-05T04:53:19.839 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_s3_success_create_obj_status": "0", 2024-02-05T04:53:19.839 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-02-05T04:53:19.839 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_scheduler_type": "throttler", 2024-02-05T04:53:19.839 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_script_uri": "", 2024-02-05T04:53:19.839 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_service_provider_name": "", 2024-02-05T04:53:19.839 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-02-05T04:53:19.839 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_sts_client_id": "", 2024-02-05T04:53:19.839 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_sts_client_secret": "", 2024-02-05T04:53:19.839 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_sts_entry": "sts", 2024-02-05T04:53:19.840 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_sts_key": "sts", 2024-02-05T04:53:19.840 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_sts_max_session_duration": "43200", 2024-02-05T04:53:19.840 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_sts_min_session_duration": "900", 2024-02-05T04:53:19.840 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_sts_token_introspection_url": "", 2024-02-05T04:53:19.840 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_swift_account_in_url": "false", 2024-02-05T04:53:19.840 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_swift_auth_entry": "auth", 2024-02-05T04:53:19.840 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_swift_auth_url": "", 2024-02-05T04:53:19.840 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_swift_custom_header": "", 2024-02-05T04:53:19.840 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_swift_enforce_content_length": "false", 2024-02-05T04:53:19.840 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_swift_need_stats": "true", 2024-02-05T04:53:19.840 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_swift_tenant_name": "", 2024-02-05T04:53:19.840 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_swift_token_expiration": "86400", 2024-02-05T04:53:19.840 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_swift_url": "", 2024-02-05T04:53:19.840 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_swift_url_prefix": "swift", 2024-02-05T04:53:19.841 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_swift_versioning_enabled": "false", 2024-02-05T04:53:19.841 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-02-05T04:53:19.841 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-02-05T04:53:19.841 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_sync_lease_period": "120", 2024-02-05T04:53:19.841 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-02-05T04:53:19.841 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_sync_log_trim_interval": "1200", 2024-02-05T04:53:19.841 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-02-05T04:53:19.841 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-02-05T04:53:19.841 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-02-05T04:53:19.841 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_sync_obj_etag_verify": "false", 2024-02-05T04:53:19.841 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_sync_trace_history_size": "4096", 2024-02-05T04:53:19.841 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-02-05T04:53:19.841 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-02-05T04:53:19.841 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_thread_pool_size": "512", 2024-02-05T04:53:19.841 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_topic_persistency_max_retries": "0", 2024-02-05T04:53:19.842 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-02-05T04:53:19.842 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-02-05T04:53:19.842 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_torrent_comment": "", 2024-02-05T04:53:19.842 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_torrent_createby": "", 2024-02-05T04:53:19.842 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_torrent_encoding": "", 2024-02-05T04:53:19.842 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_torrent_flag": "false", 2024-02-05T04:53:19.842 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_torrent_max_size": "5368709120", 2024-02-05T04:53:19.842 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_torrent_origin": "", 2024-02-05T04:53:19.842 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_torrent_sha_unit": "524288", 2024-02-05T04:53:19.842 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_torrent_tracker": "", 2024-02-05T04:53:19.842 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_trust_forwarded_https": "false", 2024-02-05T04:53:19.842 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-02-05T04:53:19.842 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_usage_log_tick_interval": "30", 2024-02-05T04:53:19.842 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_usage_max_shards": "32", 2024-02-05T04:53:19.842 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_usage_max_user_shards": "1", 2024-02-05T04:53:19.843 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_use_opa_authz": "false", 2024-02-05T04:53:19.843 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_user_counters_cache": "false", 2024-02-05T04:53:19.843 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_user_counters_cache_size": "10000", 2024-02-05T04:53:19.843 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-02-05T04:53:19.843 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_user_default_quota_max_size": "-1", 2024-02-05T04:53:19.843 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_user_max_buckets": "1000", 2024-02-05T04:53:19.843 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_user_policies_max_num": "100", 2024-02-05T04:53:19.843 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-02-05T04:53:19.843 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-02-05T04:53:19.843 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_user_quota_sync_interval": "86400", 2024-02-05T04:53:19.843 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-02-05T04:53:19.843 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_user_unique_email": "true", 2024-02-05T04:53:19.843 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_verify_ssl": "true", 2024-02-05T04:53:19.843 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_website_routing_rules_max_num": "50", 2024-02-05T04:53:19.843 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_zone": "", 2024-02-05T04:53:19.844 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_zone_id": "", 2024-02-05T04:53:19.844 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-02-05T04:53:19.844 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_zonegroup": "", 2024-02-05T04:53:19.844 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_zonegroup_id": "", 2024-02-05T04:53:19.844 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-02-05T04:53:19.844 INFO:teuthology.orchestra.run.smithi062.stdout: "rgwlc_auto_session_clear": "true", 2024-02-05T04:53:19.844 INFO:teuthology.orchestra.run.smithi062.stdout: "rgwlc_skip_bucket_step": "false", 2024-02-05T04:53:19.844 INFO:teuthology.orchestra.run.smithi062.stdout: "rocksdb_block_size": "4096", 2024-02-05T04:53:19.844 INFO:teuthology.orchestra.run.smithi062.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-02-05T04:53:19.844 INFO:teuthology.orchestra.run.smithi062.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-02-05T04:53:19.844 INFO:teuthology.orchestra.run.smithi062.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-02-05T04:53:19.844 INFO:teuthology.orchestra.run.smithi062.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-02-05T04:53:19.844 INFO:teuthology.orchestra.run.smithi062.stdout: "rocksdb_cache_shard_bits": "4", 2024-02-05T04:53:19.844 INFO:teuthology.orchestra.run.smithi062.stdout: "rocksdb_cache_size": "536870912", 2024-02-05T04:53:19.844 INFO:teuthology.orchestra.run.smithi062.stdout: "rocksdb_cache_type": "binned_lru", 2024-02-05T04:53:19.845 INFO:teuthology.orchestra.run.smithi062.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-02-05T04:53:19.845 INFO:teuthology.orchestra.run.smithi062.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-02-05T04:53:19.845 INFO:teuthology.orchestra.run.smithi062.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-02-05T04:53:19.845 INFO:teuthology.orchestra.run.smithi062.stdout: "rocksdb_collect_compaction_stats": "false", 2024-02-05T04:53:19.845 INFO:teuthology.orchestra.run.smithi062.stdout: "rocksdb_collect_extended_stats": "false", 2024-02-05T04:53:19.845 INFO:teuthology.orchestra.run.smithi062.stdout: "rocksdb_collect_memory_stats": "false", 2024-02-05T04:53:19.845 INFO:teuthology.orchestra.run.smithi062.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-02-05T04:53:19.845 INFO:teuthology.orchestra.run.smithi062.stdout: "rocksdb_index_type": "binary_search", 2024-02-05T04:53:19.845 INFO:teuthology.orchestra.run.smithi062.stdout: "rocksdb_log_to_ceph_log": "true", 2024-02-05T04:53:19.845 INFO:teuthology.orchestra.run.smithi062.stdout: "rocksdb_metadata_block_size": "4096", 2024-02-05T04:53:19.845 INFO:teuthology.orchestra.run.smithi062.stdout: "rocksdb_partition_filters": "false", 2024-02-05T04:53:19.845 INFO:teuthology.orchestra.run.smithi062.stdout: "rocksdb_perf": "false", 2024-02-05T04:53:19.846 INFO:teuthology.orchestra.run.smithi062.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-02-05T04:53:19.846 INFO:teuthology.orchestra.run.smithi062.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-02-05T04:53:19.846 INFO:teuthology.orchestra.run.smithi062.stdout: "rotating_keys_renewal_timeout": "10", 2024-02-05T04:53:19.846 INFO:teuthology.orchestra.run.smithi062.stdout: "run_dir": "/var/run/ceph", 2024-02-05T04:53:19.846 INFO:teuthology.orchestra.run.smithi062.stdout: "seastore_block_create": "true", 2024-02-05T04:53:19.846 INFO:teuthology.orchestra.run.smithi062.stdout: "seastore_cache_lru_size": "67108864", 2024-02-05T04:53:19.846 INFO:teuthology.orchestra.run.smithi062.stdout: "seastore_cbjournal_size": "5368709120", 2024-02-05T04:53:19.846 INFO:teuthology.orchestra.run.smithi062.stdout: "seastore_data_delta_based_overwrite": "0", 2024-02-05T04:53:19.846 INFO:teuthology.orchestra.run.smithi062.stdout: "seastore_default_max_object_size": "16777216", 2024-02-05T04:53:19.846 INFO:teuthology.orchestra.run.smithi062.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-02-05T04:53:19.846 INFO:teuthology.orchestra.run.smithi062.stdout: "seastore_device_size": "53687091200", 2024-02-05T04:53:19.846 INFO:teuthology.orchestra.run.smithi062.stdout: "seastore_journal_batch_capacity": "16", 2024-02-05T04:53:19.846 INFO:teuthology.orchestra.run.smithi062.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-02-05T04:53:19.847 INFO:teuthology.orchestra.run.smithi062.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-02-05T04:53:19.847 INFO:teuthology.orchestra.run.smithi062.stdout: "seastore_journal_iodepth_limit": "5", 2024-02-05T04:53:19.847 INFO:teuthology.orchestra.run.smithi062.stdout: "seastore_main_device_type": "SSD", 2024-02-05T04:53:19.847 INFO:teuthology.orchestra.run.smithi062.stdout: "seastore_max_concurrent_transactions": "8", 2024-02-05T04:53:19.847 INFO:teuthology.orchestra.run.smithi062.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-02-05T04:53:19.847 INFO:teuthology.orchestra.run.smithi062.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-02-05T04:53:19.847 INFO:teuthology.orchestra.run.smithi062.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-02-05T04:53:19.847 INFO:teuthology.orchestra.run.smithi062.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-02-05T04:53:19.847 INFO:teuthology.orchestra.run.smithi062.stdout: "seastore_segment_size": "67108864", 2024-02-05T04:53:19.847 INFO:teuthology.orchestra.run.smithi062.stdout: "set_keepcaps": "false", 2024-02-05T04:53:19.847 INFO:teuthology.orchestra.run.smithi062.stdout: "setgroup": "", 2024-02-05T04:53:19.847 INFO:teuthology.orchestra.run.smithi062.stdout: "setuser": "", 2024-02-05T04:53:19.847 INFO:teuthology.orchestra.run.smithi062.stdout: "setuser_match_path": "", 2024-02-05T04:53:19.847 INFO:teuthology.orchestra.run.smithi062.stdout: "target_max_misplaced_ratio": "0.050000", 2024-02-05T04:53:19.847 INFO:teuthology.orchestra.run.smithi062.stdout: "thp": "false", 2024-02-05T04:53:19.847 INFO:teuthology.orchestra.run.smithi062.stdout: "threadpool_default_timeout": "60", 2024-02-05T04:53:19.848 INFO:teuthology.orchestra.run.smithi062.stdout: "threadpool_empty_queue_max_wait": "2", 2024-02-05T04:53:19.848 INFO:teuthology.orchestra.run.smithi062.stdout: "throttler_perf_counter": "true" 2024-02-05T04:53:19.848 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-02-05T04:53:19.850 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-02-05T04:53:19.966 INFO:teuthology.orchestra.run.smithi062.stdout:mon.a: {} 2024-02-05T04:53:19.967 INFO:teuthology.orchestra.run.smithi062.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-02-05T04:53:19.976 INFO:teuthology.orchestra.run.smithi062.stdout:mon.b: {} 2024-02-05T04:53:19.976 INFO:teuthology.orchestra.run.smithi062.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-02-05T04:53:19.986 INFO:teuthology.orchestra.run.smithi062.stdout:mon.c: {} 2024-02-05T04:53:19.986 INFO:teuthology.orchestra.run.smithi062.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-02-05T04:53:19.997 DEBUG:teuthology.orchestra.run.smithi062:> type ceph-objectstore-tool 2024-02-05T04:53:20.010 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-02-05T04:53:20.011 DEBUG:teuthology.orchestra.run.smithi150:> type ceph-objectstore-tool 2024-02-05T04:53:20.028 INFO:teuthology.orchestra.run.smithi150.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-02-05T04:53:20.028 DEBUG:teuthology.orchestra.run.smithi091:> type ceph-objectstore-tool 2024-02-05T04:53:20.046 INFO:teuthology.orchestra.run.smithi091.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-02-05T04:53:20.046 DEBUG:teuthology.orchestra.run.smithi196:> type ceph-objectstore-tool 2024-02-05T04:53:20.063 INFO:teuthology.orchestra.run.smithi196.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-02-05T04:53:20.064 INFO:teuthology.run_tasks:Running task rados... 2024-02-05T04:53:20.074 INFO:tasks.rados:Beginning rados... 2024-02-05T04:53:20.075 DEBUG:teuthology.run_tasks:Unwinding manager rados 2024-02-05T04:53:20.084 INFO:tasks.rados:joining rados 2024-02-05T04:53:20.084 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-02-05T04:53:20.084 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-02-05T04:53:20.085 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-02-05T04:53:20.085 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-02-05T04:53:20.085 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-02-05T04:53:20.085 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T04:53:20.086 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-02-05T04:53:20.086 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-02-05T04:53:20.086 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-02-05T04:53:20.087 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-02-05T04:53:20.087 INFO:tasks.rados:clients are ['client.0'] 2024-02-05T04:53:20.087 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set shecprofile c=2 crush-failure-domain=osd k=4 m=3 name=shecprofile plugin=shec 2024-02-05T04:53:20.089 INFO:tasks.ceph.osd.2.smithi150.stderr:2024-02-05T04:53:20.085+0000 7f7337bf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 34014) UID: 0 2024-02-05T04:53:20.187 INFO:tasks.ceph.osd.5.smithi091.stderr:2024-02-05T04:53:20.187+0000 7f63983b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34277) UID: 0 2024-02-05T04:53:20.228 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0: {} 2024-02-05T04:53:20.229 INFO:teuthology.orchestra.run.smithi062.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T04:53:20.249 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1: {} 2024-02-05T04:53:20.249 INFO:teuthology.orchestra.run.smithi062.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T04:53:20.261 INFO:teuthology.orchestra.run.smithi062.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:20.262 INFO:teuthology.orchestra.run.smithi062.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:20.272 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2: {} 2024-02-05T04:53:20.272 INFO:teuthology.orchestra.run.smithi062.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T04:53:20.275 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-02-05T04:53:20.288 INFO:tasks.ceph.osd.13.smithi091.stderr:2024-02-05T04:53:20.287+0000 7f526fd4d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 34281) UID: 0 2024-02-05T04:53:20.293 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3: {} 2024-02-05T04:53:20.293 INFO:teuthology.orchestra.run.smithi062.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T04:53:20.311 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4: {} 2024-02-05T04:53:20.311 INFO:teuthology.orchestra.run.smithi062.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T04:53:20.345 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5: {} 2024-02-05T04:53:20.345 INFO:teuthology.orchestra.run.smithi062.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T04:53:20.380 INFO:teuthology.orchestra.run.smithi062.stdout:osd.6: {} 2024-02-05T04:53:20.380 INFO:teuthology.orchestra.run.smithi062.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T04:53:20.388 INFO:tasks.ceph.osd.10.smithi150.stderr:2024-02-05T04:53:20.388+0000 7f3075a4d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 34018) UID: 0 2024-02-05T04:53:20.415 INFO:teuthology.orchestra.run.smithi062.stdout:osd.7: {} 2024-02-05T04:53:20.415 INFO:teuthology.orchestra.run.smithi062.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T04:53:20.449 INFO:teuthology.orchestra.run.smithi062.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:20.449 INFO:teuthology.orchestra.run.smithi062.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:20.452 INFO:teuthology.orchestra.run.smithi062.stdout:osd.8: {} 2024-02-05T04:53:20.452 INFO:teuthology.orchestra.run.smithi062.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-05T04:53:20.460 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-02-05T04:53:20.472 INFO:teuthology.orchestra.run.smithi062.stdout:osd.9: {} 2024-02-05T04:53:20.473 INFO:teuthology.orchestra.run.smithi062.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-05T04:53:20.490 INFO:tasks.ceph.osd.9.smithi091.stderr:2024-02-05T04:53:20.489+0000 7efc18827640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 34279) UID: 0 2024-02-05T04:53:20.490 INFO:teuthology.orchestra.run.smithi062.stdout:osd.10: {} 2024-02-05T04:53:20.491 INFO:teuthology.orchestra.run.smithi062.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-05T04:53:20.509 INFO:teuthology.orchestra.run.smithi062.stdout:osd.11: {} 2024-02-05T04:53:20.509 INFO:teuthology.orchestra.run.smithi062.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-05T04:53:20.526 INFO:teuthology.orchestra.run.smithi062.stdout:osd.12: {} 2024-02-05T04:53:20.526 INFO:teuthology.orchestra.run.smithi062.stderr:osd.12: osd_enable_op_tracker = '' 2024-02-05T04:53:20.544 INFO:teuthology.orchestra.run.smithi062.stdout:osd.13: {} 2024-02-05T04:53:20.545 INFO:teuthology.orchestra.run.smithi062.stderr:osd.13: osd_enable_op_tracker = '' 2024-02-05T04:53:20.563 INFO:teuthology.orchestra.run.smithi062.stdout:osd.14: {} 2024-02-05T04:53:20.563 INFO:teuthology.orchestra.run.smithi062.stderr:osd.14: osd_enable_op_tracker = '' 2024-02-05T04:53:20.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-02-05T04:53:20.563+0000 7fe2b37fe640 0 -- 172.21.15.62:0/992719910 >> [v2:172.21.15.196:6826/3187639683,v1:172.21.15.196:6827/3187639683] conn(0x7fe2b421fb90 msgr2=0x7fe2b4221f80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-02-05T04:53:20.580 INFO:teuthology.orchestra.run.smithi062.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:20.581 INFO:teuthology.orchestra.run.smithi062.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:20.582 INFO:teuthology.orchestra.run.smithi062.stdout:osd.15: {} 2024-02-05T04:53:20.582 INFO:teuthology.orchestra.run.smithi062.stderr:osd.15: osd_enable_op_tracker = '' 2024-02-05T04:53:20.591 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-02-05T04:53:20.591 INFO:tasks.ceph.osd.4.smithi062.stderr:2024-02-05T04:53:20.589+0000 7f708c308640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34689) UID: 0 2024-02-05T04:53:20.691 INFO:tasks.ceph.osd.12.smithi062.stderr:2024-02-05T04:53:20.690+0000 7f5e5847e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 34695) UID: 0 2024-02-05T04:53:20.741 INFO:teuthology.orchestra.run.smithi062.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:20.741 INFO:teuthology.orchestra.run.smithi062.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:20.750 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-02-05T04:53:20.792 INFO:tasks.ceph.osd.6.smithi150.stderr:2024-02-05T04:53:20.791+0000 7f6bea9f0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34015) UID: 0 2024-02-05T04:53:20.893 INFO:tasks.ceph.osd.6.smithi150.stderr:2024-02-05T04:53:20.892+0000 7f6bea9f0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34015) UID: 0 2024-02-05T04:53:20.897 INFO:teuthology.orchestra.run.smithi062.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:20.897 INFO:teuthology.orchestra.run.smithi062.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:20.905 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-02-05T04:53:20.994 INFO:tasks.ceph.osd.6.smithi150.stderr:2024-02-05T04:53:20.992+0000 7f6bea9f0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34015) UID: 0 2024-02-05T04:53:21.013 INFO:teuthology.orchestra.run.smithi062.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:21.013 INFO:teuthology.orchestra.run.smithi062.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:21.021 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-02-05T04:53:21.061 INFO:teuthology.orchestra.run.smithi062.stderr:noscrub is set 2024-02-05T04:53:21.065 INFO:teuthology.orchestra.run.smithi062.stderr:reweighted osd.0 to 1 (10000) 2024-02-05T04:53:21.065 INFO:teuthology.orchestra.run.smithi062.stderr:load: shec 2024-02-05T04:53:21.081 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-02-05T04:53:21.081 INFO:tasks.rados:starting run 0 out of 1 2024-02-05T04:53:21.081 INFO:tasks.ceph.ceph_manager.ceph:creating pool_name unique_pool_0 2024-02-05T04:53:21.082 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_0 16 16 erasure shecprofile 2024-02-05T04:53:21.122 INFO:tasks.ceph.osd.8.smithi062.stderr:2024-02-05T04:53:21.121+0000 7f9385bf2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 34693) UID: 0 2024-02-05T04:53:21.133 INFO:teuthology.orchestra.run.smithi062.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:21.134 INFO:teuthology.orchestra.run.smithi062.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:21.144 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-02-05T04:53:21.195 INFO:tasks.ceph.osd.10.smithi150.stderr:2024-02-05T04:53:21.194+0000 7f3075a4d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 34018) UID: 0 2024-02-05T04:53:21.296 INFO:tasks.ceph.osd.3.smithi196.stderr:2024-02-05T04:53:21.295+0000 7fc8817c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34605) UID: 0 2024-02-05T04:53:21.302 INFO:teuthology.orchestra.run.smithi062.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:21.303 INFO:teuthology.orchestra.run.smithi062.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:21.312 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-02-05T04:53:21.396 INFO:tasks.ceph.osd.8.smithi062.stderr:2024-02-05T04:53:21.396+0000 7f9385bf2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 34693) UID: 0 2024-02-05T04:53:21.420 INFO:teuthology.orchestra.run.smithi062.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:21.420 INFO:teuthology.orchestra.run.smithi062.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:21.429 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-02-05T04:53:21.497 INFO:tasks.ceph.osd.14.smithi150.stderr:2024-02-05T04:53:21.497+0000 7f5f81083640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 34020) UID: 0 2024-02-05T04:53:21.576 INFO:teuthology.orchestra.run.smithi062.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:21.576 INFO:teuthology.orchestra.run.smithi062.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:21.584 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-02-05T04:53:21.598 INFO:tasks.ceph.osd.11.smithi196.stderr:2024-02-05T04:53:21.598+0000 7f85a8d86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 34609) UID: 0 2024-02-05T04:53:21.691 INFO:teuthology.orchestra.run.smithi062.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:21.691 INFO:teuthology.orchestra.run.smithi062.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:21.700 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-02-05T04:53:21.700 INFO:tasks.ceph.osd.9.smithi091.stderr:2024-02-05T04:53:21.699+0000 7efc18827640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 34279) UID: 0 2024-02-05T04:53:21.801 INFO:tasks.ceph.osd.10.smithi150.stderr:2024-02-05T04:53:21.800+0000 7f3075a4d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 34018) UID: 0 2024-02-05T04:53:21.807 INFO:teuthology.orchestra.run.smithi062.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:21.807 INFO:teuthology.orchestra.run.smithi062.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:21.816 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-02-05T04:53:21.902 INFO:tasks.ceph.osd.4.smithi062.stderr:2024-02-05T04:53:21.901+0000 7f708c308640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34689) UID: 0 2024-02-05T04:53:21.933 INFO:teuthology.orchestra.run.smithi091.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:21.934 INFO:teuthology.orchestra.run.smithi091.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:21.943 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-02-05T04:53:22.003 INFO:tasks.ceph.osd.6.smithi150.stderr:2024-02-05T04:53:22.003+0000 7f6bea9f0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34015) UID: 0 2024-02-05T04:53:22.048 INFO:teuthology.orchestra.run.smithi091.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:22.048 INFO:teuthology.orchestra.run.smithi091.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:22.057 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-02-05T04:53:22.067 INFO:teuthology.orchestra.run.smithi062.stderr:reweighted osd.1 to 1 (10000) 2024-02-05T04:53:22.080 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-02-05T04:53:22.122 INFO:tasks.ceph.osd.4.smithi062.stderr:2024-02-05T04:53:22.121+0000 7f708c308640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34689) UID: 0 2024-02-05T04:53:22.163 INFO:teuthology.orchestra.run.smithi091.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:22.163 INFO:teuthology.orchestra.run.smithi091.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:22.171 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-02-05T04:53:22.215 INFO:tasks.ceph.osd.5.smithi091.stderr:2024-02-05T04:53:22.215+0000 7f63983b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34277) UID: 0 2024-02-05T04:53:22.278 INFO:teuthology.orchestra.run.smithi091.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:22.278 INFO:teuthology.orchestra.run.smithi091.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:22.287 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-02-05T04:53:22.305 INFO:tasks.ceph.osd.11.smithi196.stderr:2024-02-05T04:53:22.305+0000 7f85a8d86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 34609) UID: 0 2024-02-05T04:53:22.406 INFO:tasks.ceph.osd.8.smithi062.stderr:2024-02-05T04:53:22.405+0000 7f9385bf2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 34693) UID: 0 2024-02-05T04:53:22.432 INFO:teuthology.orchestra.run.smithi091.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:22.432 INFO:teuthology.orchestra.run.smithi091.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:22.441 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-02-05T04:53:22.507 INFO:tasks.ceph.osd.14.smithi150.stderr:2024-02-05T04:53:22.506+0000 7f5f81083640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 34020) UID: 0 2024-02-05T04:53:22.546 INFO:teuthology.orchestra.run.smithi091.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:22.546 INFO:teuthology.orchestra.run.smithi091.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:22.554 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-02-05T04:53:22.594 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T04:53:22.640 INFO:tasks.ceph.osd.12.smithi062.stderr:2024-02-05T04:53:22.639+0000 7f5e5847e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 34695) UID: 0 2024-02-05T04:53:22.659 INFO:teuthology.orchestra.run.smithi091.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:22.659 INFO:teuthology.orchestra.run.smithi091.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:22.667 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-02-05T04:53:22.709 INFO:tasks.ceph.osd.6.smithi150.stderr:2024-02-05T04:53:22.708+0000 7f6bea9f0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34015) UID: 0 2024-02-05T04:53:22.720 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0: {} 2024-02-05T04:53:22.720 INFO:teuthology.orchestra.run.smithi062.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T04:53:22.738 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1: {} 2024-02-05T04:53:22.738 INFO:teuthology.orchestra.run.smithi062.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T04:53:22.756 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2: {} 2024-02-05T04:53:22.757 INFO:teuthology.orchestra.run.smithi062.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T04:53:22.772 INFO:teuthology.orchestra.run.smithi091.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:22.772 INFO:teuthology.orchestra.run.smithi091.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:22.775 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3: {} 2024-02-05T04:53:22.775 INFO:teuthology.orchestra.run.smithi062.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T04:53:22.781 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-02-05T04:53:22.791 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4: {} 2024-02-05T04:53:22.792 INFO:teuthology.orchestra.run.smithi062.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T04:53:22.809 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5: {} 2024-02-05T04:53:22.809 INFO:teuthology.orchestra.run.smithi062.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T04:53:22.824 INFO:tasks.ceph.osd.13.smithi091.stderr:2024-02-05T04:53:22.824+0000 7f526fd4d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 34281) UID: 0 2024-02-05T04:53:22.827 INFO:teuthology.orchestra.run.smithi062.stdout:osd.6: {} 2024-02-05T04:53:22.828 INFO:teuthology.orchestra.run.smithi062.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T04:53:22.845 INFO:teuthology.orchestra.run.smithi062.stdout:osd.7: {} 2024-02-05T04:53:22.846 INFO:teuthology.orchestra.run.smithi062.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T04:53:22.862 INFO:teuthology.orchestra.run.smithi062.stdout:osd.8: {} 2024-02-05T04:53:22.862 INFO:teuthology.orchestra.run.smithi062.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-05T04:53:22.886 INFO:teuthology.orchestra.run.smithi091.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:22.886 INFO:teuthology.orchestra.run.smithi091.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:22.895 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-02-05T04:53:22.935 INFO:tasks.ceph.osd.9.smithi091.stderr:2024-02-05T04:53:22.935+0000 7efc18827640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 34279) UID: 0 2024-02-05T04:53:23.011 INFO:tasks.ceph.osd.9.smithi091.stderr:2024-02-05T04:53:23.011+0000 7efc18827640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 34279) UID: 0 2024-02-05T04:53:23.041 INFO:teuthology.orchestra.run.smithi091.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:23.041 INFO:teuthology.orchestra.run.smithi091.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:23.049 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-02-05T04:53:23.068 INFO:teuthology.orchestra.run.smithi062.stderr:pool 'unique_pool_0' created 2024-02-05T04:53:23.069 INFO:teuthology.orchestra.run.smithi062.stderr:reweighted osd.2 to 1 (10000) 2024-02-05T04:53:23.079 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-02-05T04:53:23.090 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-02-05T04:53:23.091 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-02-05T04:53:23.092 INFO:teuthology.orchestra.run.smithi062.stdout:osd.9: {} 2024-02-05T04:53:23.092 INFO:teuthology.orchestra.run.smithi062.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-05T04:53:23.109 INFO:teuthology.orchestra.run.smithi062.stdout:osd.10: {} 2024-02-05T04:53:23.109 INFO:teuthology.orchestra.run.smithi062.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-05T04:53:23.112 INFO:tasks.ceph.osd.11.smithi196.stderr:2024-02-05T04:53:23.111+0000 7f85a8d86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 34609) UID: 0 2024-02-05T04:53:23.127 INFO:teuthology.orchestra.run.smithi062.stdout:osd.11: {} 2024-02-05T04:53:23.128 INFO:teuthology.orchestra.run.smithi062.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-05T04:53:23.148 INFO:teuthology.orchestra.run.smithi062.stdout:osd.12: {} 2024-02-05T04:53:23.148 INFO:teuthology.orchestra.run.smithi062.stderr:osd.12: osd_enable_op_tracker = '' 2024-02-05T04:53:23.166 INFO:teuthology.orchestra.run.smithi062.stdout:osd.13: {} 2024-02-05T04:53:23.166 INFO:teuthology.orchestra.run.smithi062.stderr:osd.13: osd_enable_op_tracker = '' 2024-02-05T04:53:23.184 INFO:teuthology.orchestra.run.smithi062.stdout:osd.14: {} 2024-02-05T04:53:23.185 INFO:teuthology.orchestra.run.smithi062.stderr:osd.14: osd_enable_op_tracker = '' 2024-02-05T04:53:23.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-02-05T04:53:23.184+0000 7fd6a1d10640 0 -- 172.21.15.62:0/1816898628 >> [v2:172.21.15.196:6826/3187639683,v1:172.21.15.196:6827/3187639683] conn(0x7fd69c221cf0 msgr2=0x7fd69c2240e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-02-05T04:53:23.197 INFO:teuthology.orchestra.run.smithi091.stdout:{ 2024-02-05T04:53:23.197 INFO:teuthology.orchestra.run.smithi091.stdout: "ops": [], 2024-02-05T04:53:23.197 INFO:teuthology.orchestra.run.smithi091.stdout: "complaint_time": 30, 2024-02-05T04:53:23.198 INFO:teuthology.orchestra.run.smithi091.stdout: "num_blocked_ops": 0 2024-02-05T04:53:23.198 INFO:teuthology.orchestra.run.smithi091.stdout:} 2024-02-05T04:53:23.206 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-02-05T04:53:23.213 INFO:tasks.ceph.osd.2.smithi150.stderr:2024-02-05T04:53:23.212+0000 7f7337bf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 34014) UID: 0 2024-02-05T04:53:23.311 INFO:teuthology.orchestra.run.smithi091.stdout:{ 2024-02-05T04:53:23.311 INFO:teuthology.orchestra.run.smithi091.stdout: "size": 20, 2024-02-05T04:53:23.311 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 600, 2024-02-05T04:53:23.312 INFO:teuthology.orchestra.run.smithi091.stdout: "ops": [] 2024-02-05T04:53:23.312 INFO:teuthology.orchestra.run.smithi091.stdout:} 2024-02-05T04:53:23.314 INFO:tasks.ceph.osd.1.smithi091.stderr:2024-02-05T04:53:23.313+0000 7ff9dc283640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34275) UID: 0 2024-02-05T04:53:23.320 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-02-05T04:53:23.404 INFO:teuthology.orchestra.run.smithi062.stdout:osd.15: {} 2024-02-05T04:53:23.404 INFO:teuthology.orchestra.run.smithi062.stderr:osd.15: osd_enable_op_tracker = '' 2024-02-05T04:53:23.415 INFO:tasks.ceph.osd.9.smithi091.stderr:2024-02-05T04:53:23.414+0000 7efc18827640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 34279) UID: 0 2024-02-05T04:53:23.436 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-02-05T04:53:23.436 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [], 2024-02-05T04:53:23.436 INFO:teuthology.orchestra.run.smithi150.stdout: "num_ops": 0 2024-02-05T04:53:23.436 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-02-05T04:53:23.445 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-02-05T04:53:23.516 INFO:tasks.ceph.osd.0.smithi062.stderr:2024-02-05T04:53:23.515+0000 7fece8e03640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34688) UID: 0 2024-02-05T04:53:23.550 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-02-05T04:53:23.550 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [], 2024-02-05T04:53:23.550 INFO:teuthology.orchestra.run.smithi150.stdout: "complaint_time": 30, 2024-02-05T04:53:23.550 INFO:teuthology.orchestra.run.smithi150.stdout: "num_blocked_ops": 0 2024-02-05T04:53:23.551 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-02-05T04:53:23.559 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-02-05T04:53:23.617 INFO:tasks.ceph.osd.7.smithi196.stderr:2024-02-05T04:53:23.616+0000 7fb281419640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34606) UID: 0 2024-02-05T04:53:23.666 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-02-05T04:53:23.666 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 20, 2024-02-05T04:53:23.666 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 600, 2024-02-05T04:53:23.666 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [ 2024-02-05T04:53:23.666 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.667 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.4209.0:85 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'55, mlcod=24'50)", 2024-02-05T04:53:23.667 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:09.697543+0000", 2024-02-05T04:53:23.667 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 13.966808285000001, 2024-02-05T04:53:23.667 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0081615030000000005, 2024-02-05T04:53:23.667 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:23.667 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:23.667 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:23.667 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.667 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:23.667 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.697543+0000", 2024-02-05T04:53:23.667 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:23.667 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.667 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.667 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:23.667 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.697543+0000", 2024-02-05T04:53:23.667 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:23.668 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.668 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.668 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:23.668 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.697544+0000", 2024-02-05T04:53:23.668 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.7899999999999999e-07 2024-02-05T04:53:23.668 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.668 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.668 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:23.668 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.697546+0000", 2024-02-05T04:53:23.668 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.7970000000000001e-06 2024-02-05T04:53:23.668 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.668 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.668 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:23.668 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.697553+0000", 2024-02-05T04:53:23.669 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.2929999999999997e-06 2024-02-05T04:53:23.669 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.669 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.669 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:23.669 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.697556+0000", 2024-02-05T04:53:23.669 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.4309999999999998e-06 2024-02-05T04:53:23.669 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.669 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.669 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:23.669 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.697596+0000", 2024-02-05T04:53:23.669 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.977e-05 2024-02-05T04:53:23.669 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.669 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.669 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:53:23.669 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.697607+0000", 2024-02-05T04:53:23.669 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1459e-05 2024-02-05T04:53:23.670 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.670 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.670 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-02-05T04:53:23.670 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.705669+0000", 2024-02-05T04:53:23.670 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0080620009999999992 2024-02-05T04:53:23.670 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.670 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.670 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:23.670 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.705705+0000", 2024-02-05T04:53:23.671 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.5373000000000002e-05 2024-02-05T04:53:23.671 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:23.671 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:23.671 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:23.671 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.671 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.671 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.4209.0:86 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'56, mlcod=24'52)", 2024-02-05T04:53:23.671 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:09.698014+0000", 2024-02-05T04:53:23.671 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 13.966337185, 2024-02-05T04:53:23.671 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0077071750000000001, 2024-02-05T04:53:23.671 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:23.671 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:23.671 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:23.671 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.671 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:23.672 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698014+0000", 2024-02-05T04:53:23.672 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:23.672 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.672 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.672 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:23.672 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698014+0000", 2024-02-05T04:53:23.672 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:23.672 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.672 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.672 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:23.672 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698015+0000", 2024-02-05T04:53:23.672 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.9900000000000002e-07 2024-02-05T04:53:23.672 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.672 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.673 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:23.673 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698018+0000", 2024-02-05T04:53:23.673 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.0019999999999998e-06 2024-02-05T04:53:23.673 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.673 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.673 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:23.673 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698024+0000", 2024-02-05T04:53:23.673 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.251e-06 2024-02-05T04:53:23.673 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.673 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.673 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:23.673 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698028+0000", 2024-02-05T04:53:23.673 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.76e-06 2024-02-05T04:53:23.673 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.673 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.674 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:23.674 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698066+0000", 2024-02-05T04:53:23.674 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.8556999999999997e-05 2024-02-05T04:53:23.674 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.674 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.674 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:53:23.674 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698078+0000", 2024-02-05T04:53:23.674 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1620999999999999e-05 2024-02-05T04:53:23.674 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.674 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.674 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-02-05T04:53:23.674 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.705706+0000", 2024-02-05T04:53:23.674 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0076284400000000002 2024-02-05T04:53:23.674 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.674 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.674 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:23.675 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.705721+0000", 2024-02-05T04:53:23.675 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.5245000000000001e-05 2024-02-05T04:53:23.675 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:23.675 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:23.675 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:23.675 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.675 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.675 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.4209.0:87 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'57, mlcod=24'52)", 2024-02-05T04:53:23.675 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:09.698307+0000", 2024-02-05T04:53:23.675 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 13.966044257, 2024-02-05T04:53:23.675 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0074282530000000001, 2024-02-05T04:53:23.675 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:23.675 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:23.675 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:23.675 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.676 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:23.676 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698307+0000", 2024-02-05T04:53:23.676 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:23.676 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.676 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.676 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:23.676 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698307+0000", 2024-02-05T04:53:23.676 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:23.676 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.676 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.676 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:23.676 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698308+0000", 2024-02-05T04:53:23.676 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.9900000000000001e-07 2024-02-05T04:53:23.676 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.677 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.677 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:23.677 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698358+0000", 2024-02-05T04:53:23.677 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.0649000000000002e-05 2024-02-05T04:53:23.677 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.677 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.677 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:23.677 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698366+0000", 2024-02-05T04:53:23.677 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.9000000000000006e-06 2024-02-05T04:53:23.677 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.677 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.677 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:23.677 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698370+0000", 2024-02-05T04:53:23.677 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.7270000000000001e-06 2024-02-05T04:53:23.677 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.678 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.678 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:23.678 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698410+0000", 2024-02-05T04:53:23.678 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.0117000000000002e-05 2024-02-05T04:53:23.678 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.678 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.678 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:53:23.678 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698423+0000", 2024-02-05T04:53:23.678 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2531000000000001e-05 2024-02-05T04:53:23.678 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.678 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.678 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-02-05T04:53:23.678 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.705722+0000", 2024-02-05T04:53:23.678 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0072995270000000001 2024-02-05T04:53:23.678 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.679 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.679 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:23.679 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.705735+0000", 2024-02-05T04:53:23.679 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.3403e-05 2024-02-05T04:53:23.679 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:23.679 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:23.679 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:23.679 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.679 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.679 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.4209.0:88 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'58, mlcod=24'52)", 2024-02-05T04:53:23.679 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:09.698756+0000", 2024-02-05T04:53:23.679 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 13.965595169, 2024-02-05T04:53:23.679 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0072080540000000002, 2024-02-05T04:53:23.679 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:23.680 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:23.680 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:23.680 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.680 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:23.680 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698756+0000", 2024-02-05T04:53:23.680 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:23.680 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.680 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.680 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:23.680 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698756+0000", 2024-02-05T04:53:23.680 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:23.680 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.680 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.680 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:23.681 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698757+0000", 2024-02-05T04:53:23.681 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.6899999999999997e-07 2024-02-05T04:53:23.681 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.681 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.681 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:23.681 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698762+0000", 2024-02-05T04:53:23.681 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.8790000000000002e-06 2024-02-05T04:53:23.681 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.681 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.681 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:23.681 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698773+0000", 2024-02-05T04:53:23.681 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1e-05 2024-02-05T04:53:23.681 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.681 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.681 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:23.682 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698780+0000", 2024-02-05T04:53:23.682 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.8920000000000003e-06 2024-02-05T04:53:23.682 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.682 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.682 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:23.682 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.700009+0000", 2024-02-05T04:53:23.682 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.001229665 2024-02-05T04:53:23.682 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.682 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.682 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:53:23.682 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.700022+0000", 2024-02-05T04:53:23.682 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2699e-05 2024-02-05T04:53:23.682 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.682 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.682 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-02-05T04:53:23.683 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.705943+0000", 2024-02-05T04:53:23.683 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0059209759999999997 2024-02-05T04:53:23.683 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.683 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.683 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:23.683 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.705964+0000", 2024-02-05T04:53:23.683 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.1274e-05 2024-02-05T04:53:23.683 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:23.683 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:23.683 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:23.683 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.683 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.683 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.4209.0:109 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'71, mlcod=24'67)", 2024-02-05T04:53:23.683 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:09.853158+0000", 2024-02-05T04:53:23.684 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 13.81119355, 2024-02-05T04:53:23.684 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0046048460000000001, 2024-02-05T04:53:23.684 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:23.684 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:23.684 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:23.684 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.684 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:23.684 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.853158+0000", 2024-02-05T04:53:23.684 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:23.684 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.684 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.684 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:23.684 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.853158+0000", 2024-02-05T04:53:23.684 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:23.684 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.685 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.685 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:23.685 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.853158+0000", 2024-02-05T04:53:23.685 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.44e-07 2024-02-05T04:53:23.685 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.685 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.685 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:23.685 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.853236+0000", 2024-02-05T04:53:23.685 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.7917999999999995e-05 2024-02-05T04:53:23.685 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.685 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.685 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:23.685 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.853248+0000", 2024-02-05T04:53:23.685 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1861999999999999e-05 2024-02-05T04:53:23.685 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.686 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.686 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:23.686 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.853254+0000", 2024-02-05T04:53:23.686 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.8719999999999999e-06 2024-02-05T04:53:23.686 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.686 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.686 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:23.686 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.853297+0000", 2024-02-05T04:53:23.686 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.2527000000000001e-05 2024-02-05T04:53:23.686 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.686 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.686 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:53:23.686 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.853310+0000", 2024-02-05T04:53:23.686 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.3560999999999999e-05 2024-02-05T04:53:23.686 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.687 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.687 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-02-05T04:53:23.687 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.857733+0000", 2024-02-05T04:53:23.687 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.004422558 2024-02-05T04:53:23.687 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.687 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.687 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:23.687 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.857763+0000", 2024-02-05T04:53:23.687 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.0003999999999999e-05 2024-02-05T04:53:23.687 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:23.687 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:23.687 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:23.687 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.687 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.687 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.4209.0:110 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'72, mlcod=24'67)", 2024-02-05T04:53:23.688 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:09.853702+0000", 2024-02-05T04:53:23.688 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 13.810648987, 2024-02-05T04:53:23.688 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0063171329999999999, 2024-02-05T04:53:23.688 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:23.688 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:23.688 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:23.688 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.688 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:23.688 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.853702+0000", 2024-02-05T04:53:23.688 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:23.688 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.688 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.688 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:23.688 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.853702+0000", 2024-02-05T04:53:23.688 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:23.689 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.689 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.689 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:23.689 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.853703+0000", 2024-02-05T04:53:23.689 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.1100000000000002e-07 2024-02-05T04:53:23.689 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.689 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.689 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:23.689 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.853706+0000", 2024-02-05T04:53:23.689 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.3629999999999998e-06 2024-02-05T04:53:23.689 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.689 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.689 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:23.689 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.853713+0000", 2024-02-05T04:53:23.689 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.8410000000000002e-06 2024-02-05T04:53:23.690 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.690 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.690 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:23.690 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.853717+0000", 2024-02-05T04:53:23.690 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.777e-06 2024-02-05T04:53:23.690 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.690 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.690 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:23.690 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.856746+0000", 2024-02-05T04:53:23.690 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0030292800000000001 2024-02-05T04:53:23.690 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.690 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.690 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:53:23.690 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.856762+0000", 2024-02-05T04:53:23.690 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.5651999999999999e-05 2024-02-05T04:53:23.690 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.691 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.691 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-02-05T04:53:23.691 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.859984+0000", 2024-02-05T04:53:23.691 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0032218289999999998 2024-02-05T04:53:23.691 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.691 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.691 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:23.691 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.860020+0000", 2024-02-05T04:53:23.691 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.608e-05 2024-02-05T04:53:23.691 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:23.691 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:23.691 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:23.691 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.691 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.691 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.4209.0:111 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'73, mlcod=24'68)", 2024-02-05T04:53:23.692 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:09.854221+0000", 2024-02-05T04:53:23.692 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 13.810130656, 2024-02-05T04:53:23.692 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0058190960000000002, 2024-02-05T04:53:23.692 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:23.692 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:23.692 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:23.692 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.692 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:23.692 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.854221+0000", 2024-02-05T04:53:23.692 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:23.692 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.692 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.692 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:23.692 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.854221+0000", 2024-02-05T04:53:23.692 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:23.693 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.693 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.693 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:23.693 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.854221+0000", 2024-02-05T04:53:23.693 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.72e-07 2024-02-05T04:53:23.693 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.693 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.693 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:23.693 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.854224+0000", 2024-02-05T04:53:23.693 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.1779999999999999e-06 2024-02-05T04:53:23.693 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.693 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.693 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:23.693 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.854231+0000", 2024-02-05T04:53:23.693 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.7440000000000001e-06 2024-02-05T04:53:23.694 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.694 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.694 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:23.694 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.854235+0000", 2024-02-05T04:53:23.694 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.8349999999999997e-06 2024-02-05T04:53:23.694 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.694 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.694 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:23.694 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.856952+0000", 2024-02-05T04:53:23.694 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0027175839999999999 2024-02-05T04:53:23.694 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.694 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.694 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:53:23.694 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.856966+0000", 2024-02-05T04:53:23.694 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.3739e-05 2024-02-05T04:53:23.695 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.695 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.695 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-02-05T04:53:23.695 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.860023+0000", 2024-02-05T04:53:23.695 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0030564979999999999 2024-02-05T04:53:23.695 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.695 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.695 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:23.695 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.860040+0000", 2024-02-05T04:53:23.695 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.7146000000000002e-05 2024-02-05T04:53:23.695 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:23.695 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:23.695 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:23.695 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.695 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.696 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.4209.0:112 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'74, mlcod=24'69)", 2024-02-05T04:53:23.696 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:09.854776+0000", 2024-02-05T04:53:23.696 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 13.809575646000001, 2024-02-05T04:53:23.696 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0052805700000000001, 2024-02-05T04:53:23.696 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:23.696 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:23.696 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:23.696 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.696 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:23.696 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.854776+0000", 2024-02-05T04:53:23.696 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:23.696 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.696 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.696 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:23.697 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.854776+0000", 2024-02-05T04:53:23.697 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:23.697 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.697 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.697 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:23.697 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.854777+0000", 2024-02-05T04:53:23.697 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.1999999999999999e-07 2024-02-05T04:53:23.697 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.697 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.697 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:23.697 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.854805+0000", 2024-02-05T04:53:23.697 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.8827999999999999e-05 2024-02-05T04:53:23.697 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.697 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.697 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:23.697 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.854817+0000", 2024-02-05T04:53:23.698 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.165e-05 2024-02-05T04:53:23.698 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.698 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.698 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:23.698 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.854822+0000", 2024-02-05T04:53:23.698 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.293e-06 2024-02-05T04:53:23.698 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.698 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.698 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:23.698 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.857167+0000", 2024-02-05T04:53:23.698 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0023451879999999998 2024-02-05T04:53:23.698 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.698 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.698 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:53:23.698 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.857181+0000", 2024-02-05T04:53:23.698 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.3455000000000001e-05 2024-02-05T04:53:23.699 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.699 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.699 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-02-05T04:53:23.699 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.860042+0000", 2024-02-05T04:53:23.699 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0028610839999999998 2024-02-05T04:53:23.699 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.699 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.699 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:23.699 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.860056+0000", 2024-02-05T04:53:23.699 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.4352000000000001e-05 2024-02-05T04:53:23.699 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:23.699 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:23.699 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:23.699 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.699 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.700 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.4209.0:113 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'75, mlcod=24'69)", 2024-02-05T04:53:23.700 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:09.855187+0000", 2024-02-05T04:53:23.700 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 13.809164244, 2024-02-05T04:53:23.700 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.004895684, 2024-02-05T04:53:23.700 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:23.700 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:23.700 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:23.700 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.700 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:23.700 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.855187+0000", 2024-02-05T04:53:23.700 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:23.700 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.700 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.700 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:23.700 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.855187+0000", 2024-02-05T04:53:23.701 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:23.701 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.701 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.701 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:23.701 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.855188+0000", 2024-02-05T04:53:23.701 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.5299999999999999e-07 2024-02-05T04:53:23.701 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.701 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.701 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:23.701 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.855191+0000", 2024-02-05T04:53:23.701 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.4790000000000001e-06 2024-02-05T04:53:23.701 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.701 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.701 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:23.701 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.855199+0000", 2024-02-05T04:53:23.701 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.4769999999999997e-06 2024-02-05T04:53:23.701 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.702 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.702 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:23.702 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.855203+0000", 2024-02-05T04:53:23.702 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.6149999999999999e-06 2024-02-05T04:53:23.702 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.702 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.702 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:23.702 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.857370+0000", 2024-02-05T04:53:23.702 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.002166692 2024-02-05T04:53:23.702 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.702 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.702 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:53:23.702 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.857381+0000", 2024-02-05T04:53:23.702 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1394e-05 2024-02-05T04:53:23.702 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.702 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.703 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-02-05T04:53:23.703 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.860058+0000", 2024-02-05T04:53:23.703 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0026767190000000001 2024-02-05T04:53:23.703 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.703 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.703 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:23.703 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.860083+0000", 2024-02-05T04:53:23.703 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.4854999999999998e-05 2024-02-05T04:53:23.703 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:23.703 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:23.703 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:23.703 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.703 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.703 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.4209.0:114 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'76, mlcod=24'69)", 2024-02-05T04:53:23.703 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:09.855745+0000", 2024-02-05T04:53:23.703 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 13.808606396, 2024-02-05T04:53:23.703 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0043552950000000003, 2024-02-05T04:53:23.704 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:23.704 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:23.704 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:23.704 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.704 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:23.704 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.855745+0000", 2024-02-05T04:53:23.704 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:23.704 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.704 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.704 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:23.704 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.855745+0000", 2024-02-05T04:53:23.704 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:23.704 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.704 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.704 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:23.704 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.855747+0000", 2024-02-05T04:53:23.705 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.57e-06 2024-02-05T04:53:23.705 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.705 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.705 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:23.705 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.855757+0000", 2024-02-05T04:53:23.705 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0560000000000001e-05 2024-02-05T04:53:23.705 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.705 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.705 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:23.705 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.855783+0000", 2024-02-05T04:53:23.705 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.6162000000000001e-05 2024-02-05T04:53:23.705 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.705 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.705 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:23.705 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.855796+0000", 2024-02-05T04:53:23.706 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2403e-05 2024-02-05T04:53:23.706 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.706 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.706 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:23.706 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.857579+0000", 2024-02-05T04:53:23.706 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0017831050000000001 2024-02-05T04:53:23.706 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.706 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.706 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:53:23.706 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.857591+0000", 2024-02-05T04:53:23.706 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1909000000000001e-05 2024-02-05T04:53:23.706 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.706 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.706 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-02-05T04:53:23.706 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.860085+0000", 2024-02-05T04:53:23.706 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0024943209999999999 2024-02-05T04:53:23.706 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.707 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.707 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:23.707 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.860100+0000", 2024-02-05T04:53:23.707 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.5265e-05 2024-02-05T04:53:23.707 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:23.707 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:23.707 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:23.707 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.707 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.707 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.4209.0:115 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'77, mlcod=24'69)", 2024-02-05T04:53:23.707 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:09.856748+0000", 2024-02-05T04:53:23.707 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 13.80760377, 2024-02-05T04:53:23.707 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0058814849999999997, 2024-02-05T04:53:23.707 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:23.707 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:23.708 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:23.708 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.708 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:23.708 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.856748+0000", 2024-02-05T04:53:23.708 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:23.708 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.708 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.708 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:23.708 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.856748+0000", 2024-02-05T04:53:23.708 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:23.708 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.708 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.708 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:23.708 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.856748+0000", 2024-02-05T04:53:23.708 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.4099999999999998e-07 2024-02-05T04:53:23.709 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.709 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.709 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:23.709 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.856781+0000", 2024-02-05T04:53:23.709 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.3056999999999999e-05 2024-02-05T04:53:23.709 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.709 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.709 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:23.709 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.856791+0000", 2024-02-05T04:53:23.709 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.5659999999999996e-06 2024-02-05T04:53:23.709 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.709 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.709 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:23.709 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.856799+0000", 2024-02-05T04:53:23.710 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.8720000000000002e-06 2024-02-05T04:53:23.710 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.710 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.710 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:23.710 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.857805+0000", 2024-02-05T04:53:23.710 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0010061180000000001 2024-02-05T04:53:23.710 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.710 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.710 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:53:23.710 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.857820+0000", 2024-02-05T04:53:23.710 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.4883e-05 2024-02-05T04:53:23.710 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.710 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.710 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-02-05T04:53:23.710 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.862596+0000", 2024-02-05T04:53:23.710 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0047763249999999997 2024-02-05T04:53:23.711 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.711 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.711 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:23.711 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.862629+0000", 2024-02-05T04:53:23.711 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.2923000000000003e-05 2024-02-05T04:53:23.711 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:23.711 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:23.711 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:23.711 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.711 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.711 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.4209.0:116 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'78, mlcod=24'69)", 2024-02-05T04:53:23.711 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:09.857396+0000", 2024-02-05T04:53:23.711 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 13.806955435999999, 2024-02-05T04:53:23.711 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0052485070000000003, 2024-02-05T04:53:23.711 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:23.712 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:23.712 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:23.712 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.712 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:23.712 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.857396+0000", 2024-02-05T04:53:23.712 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:23.712 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.712 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.712 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:23.712 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.857396+0000", 2024-02-05T04:53:23.712 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:23.712 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.712 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.712 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:23.712 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.857397+0000", 2024-02-05T04:53:23.713 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.3799999999999996e-07 2024-02-05T04:53:23.713 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.713 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.713 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:23.713 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.857402+0000", 2024-02-05T04:53:23.713 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.3149999999999999e-06 2024-02-05T04:53:23.713 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.713 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.713 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:23.713 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.857411+0000", 2024-02-05T04:53:23.713 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.4180000000000003e-06 2024-02-05T04:53:23.713 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.713 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.713 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:23.713 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.857417+0000", 2024-02-05T04:53:23.713 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.2789999999999998e-06 2024-02-05T04:53:23.714 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.714 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.714 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:23.714 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.859810+0000", 2024-02-05T04:53:23.714 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0023927530000000001 2024-02-05T04:53:23.714 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.714 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.714 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:53:23.714 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.859824+0000", 2024-02-05T04:53:23.714 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.419e-05 2024-02-05T04:53:23.714 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.714 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.714 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-02-05T04:53:23.714 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.862631+0000", 2024-02-05T04:53:23.714 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0028069380000000001 2024-02-05T04:53:23.715 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.715 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.715 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:23.715 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.862645+0000", 2024-02-05T04:53:23.715 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.3876000000000001e-05 2024-02-05T04:53:23.715 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:23.715 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:23.715 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:23.715 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.715 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.715 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.4209.0:117 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'79, mlcod=24'69)", 2024-02-05T04:53:23.715 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:09.857803+0000", 2024-02-05T04:53:23.715 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 13.806548045, 2024-02-05T04:53:23.715 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0076301729999999996, 2024-02-05T04:53:23.715 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:23.715 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:23.716 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:23.716 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.716 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:23.716 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.857803+0000", 2024-02-05T04:53:23.716 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:23.716 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.716 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.716 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:23.716 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.857803+0000", 2024-02-05T04:53:23.716 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:23.716 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.716 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.716 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:23.716 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.857804+0000", 2024-02-05T04:53:23.716 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.1999999999999999e-07 2024-02-05T04:53:23.717 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.717 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.717 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:23.717 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.857810+0000", 2024-02-05T04:53:23.717 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.8690000000000001e-06 2024-02-05T04:53:23.717 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.717 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.717 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:23.717 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.857821+0000", 2024-02-05T04:53:23.717 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1059e-05 2024-02-05T04:53:23.717 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.717 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.717 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:23.717 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.857827+0000", 2024-02-05T04:53:23.717 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.1430000000000003e-06 2024-02-05T04:53:23.718 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.718 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.718 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:23.718 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.860138+0000", 2024-02-05T04:53:23.718 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0023111160000000002 2024-02-05T04:53:23.718 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.718 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.718 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:53:23.718 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.860152+0000", 2024-02-05T04:53:23.718 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.3896999999999999e-05 2024-02-05T04:53:23.718 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.718 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.718 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-02-05T04:53:23.718 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.865400+0000", 2024-02-05T04:53:23.718 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0052475040000000001 2024-02-05T04:53:23.719 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.719 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.719 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:23.719 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.865434+0000", 2024-02-05T04:53:23.719 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.3865000000000002e-05 2024-02-05T04:53:23.719 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:23.719 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:23.719 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:23.719 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.719 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.719 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.4209.0:119 1.0 e24/23 1:30306672:devicehealth::main.db.0000000000000000:head v 24'81, mlcod=24'78)", 2024-02-05T04:53:23.719 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:09.867166+0000", 2024-02-05T04:53:23.719 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 13.797185194000001, 2024-02-05T04:53:23.719 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0033777429999999999, 2024-02-05T04:53:23.719 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:23.720 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:23.720 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:23.720 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.720 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:23.720 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.867166+0000", 2024-02-05T04:53:23.720 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:23.720 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.720 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.720 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:23.720 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.867166+0000", 2024-02-05T04:53:23.720 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:23.720 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.720 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.720 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:23.721 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.867167+0000", 2024-02-05T04:53:23.721 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.1999999999999999e-07 2024-02-05T04:53:23.721 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.721 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.721 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:23.721 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.867239+0000", 2024-02-05T04:53:23.721 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.1870000000000001e-05 2024-02-05T04:53:23.721 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.721 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.721 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:23.721 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.867250+0000", 2024-02-05T04:53:23.721 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.099e-05 2024-02-05T04:53:23.721 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.721 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.721 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:23.721 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.867256+0000", 2024-02-05T04:53:23.722 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.0480000000000004e-06 2024-02-05T04:53:23.722 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.722 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.722 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:23.722 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.867291+0000", 2024-02-05T04:53:23.722 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.4777000000000003e-05 2024-02-05T04:53:23.722 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.722 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.722 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:53:23.722 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.867302+0000", 2024-02-05T04:53:23.722 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1775e-05 2024-02-05T04:53:23.722 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.722 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.722 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-02-05T04:53:23.722 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.870506+0000", 2024-02-05T04:53:23.723 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0032040570000000002 2024-02-05T04:53:23.723 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.723 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.723 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:23.723 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.870544+0000", 2024-02-05T04:53:23.723 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.7605999999999999e-05 2024-02-05T04:53:23.723 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:23.723 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:23.723 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:23.723 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.723 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.723 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.4209.0:139 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'97, mlcod=25'93)", 2024-02-05T04:53:23.723 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:10.001153+0000", 2024-02-05T04:53:23.723 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 13.663198926, 2024-02-05T04:53:23.723 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0050519220000000004, 2024-02-05T04:53:23.724 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:23.724 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:23.724 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:23.724 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.724 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:23.724 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:10.001153+0000", 2024-02-05T04:53:23.724 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:23.724 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.724 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.724 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:23.724 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:10.001153+0000", 2024-02-05T04:53:23.724 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:23.724 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.724 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.724 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:23.725 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:10.001153+0000", 2024-02-05T04:53:23.725 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.4200000000000001e-07 2024-02-05T04:53:23.725 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.725 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.725 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:23.725 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:10.001156+0000", 2024-02-05T04:53:23.725 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.3699999999999999e-06 2024-02-05T04:53:23.725 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.725 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.725 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:23.725 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:10.001163+0000", 2024-02-05T04:53:23.725 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.917e-06 2024-02-05T04:53:23.725 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.725 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.725 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:23.725 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:10.001168+0000", 2024-02-05T04:53:23.726 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.0359999999999997e-06 2024-02-05T04:53:23.726 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.726 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.726 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:23.726 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:10.001202+0000", 2024-02-05T04:53:23.726 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.3686000000000002e-05 2024-02-05T04:53:23.726 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.726 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.726 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:53:23.726 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:10.001215+0000", 2024-02-05T04:53:23.726 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2782999999999999e-05 2024-02-05T04:53:23.726 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.726 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.726 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-02-05T04:53:23.726 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:10.006093+0000", 2024-02-05T04:53:23.727 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0048779560000000001 2024-02-05T04:53:23.727 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.727 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.727 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:23.727 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:10.006204+0000", 2024-02-05T04:53:23.727 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00011173199999999999 2024-02-05T04:53:23.727 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:23.727 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:23.727 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:23.727 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.727 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.727 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.4209.0:140 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'98, mlcod=25'93)", 2024-02-05T04:53:23.727 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:10.001630+0000", 2024-02-05T04:53:23.727 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 13.662721812000001, 2024-02-05T04:53:23.727 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0045928540000000004, 2024-02-05T04:53:23.727 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:23.728 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:23.728 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:23.728 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.728 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:23.728 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:10.001630+0000", 2024-02-05T04:53:23.728 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:23.728 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.728 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.728 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:23.728 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:10.001630+0000", 2024-02-05T04:53:23.728 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:23.728 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.728 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.728 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:23.728 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:10.001630+0000", 2024-02-05T04:53:23.728 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.8200000000000001e-07 2024-02-05T04:53:23.729 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.729 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.729 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:23.729 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:10.001674+0000", 2024-02-05T04:53:23.729 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.4338999999999997e-05 2024-02-05T04:53:23.729 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.729 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.729 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:23.729 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:10.001682+0000", 2024-02-05T04:53:23.729 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.1579999999999999e-06 2024-02-05T04:53:23.729 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.729 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.729 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:23.729 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:10.001686+0000", 2024-02-05T04:53:23.729 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.6770000000000003e-06 2024-02-05T04:53:23.729 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.730 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.730 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:23.730 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:10.001757+0000", 2024-02-05T04:53:23.730 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.0531999999999995e-05 2024-02-05T04:53:23.730 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.730 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.730 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:53:23.730 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:10.001771+0000", 2024-02-05T04:53:23.730 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.4041e-05 2024-02-05T04:53:23.730 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.730 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.730 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-02-05T04:53:23.730 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:10.006207+0000", 2024-02-05T04:53:23.730 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0044360780000000004 2024-02-05T04:53:23.730 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.731 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.731 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:23.731 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:10.006222+0000", 2024-02-05T04:53:23.731 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.5647000000000001e-05 2024-02-05T04:53:23.731 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:23.731 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:23.731 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:23.731 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.731 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.731 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.4209.0:141 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'99, mlcod=25'94)", 2024-02-05T04:53:23.731 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:10.002242+0000", 2024-02-05T04:53:23.731 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 13.662109191999999, 2024-02-05T04:53:23.731 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0039937180000000003, 2024-02-05T04:53:23.731 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:23.731 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:23.732 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:23.732 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.732 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:23.732 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:10.002242+0000", 2024-02-05T04:53:23.732 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:23.732 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.732 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.732 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:23.732 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:10.002242+0000", 2024-02-05T04:53:23.732 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:23.732 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.732 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.732 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:23.732 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:10.002243+0000", 2024-02-05T04:53:23.732 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.8599999999999999e-07 2024-02-05T04:53:23.733 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.733 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.733 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:23.733 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:10.002246+0000", 2024-02-05T04:53:23.733 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.4529999999999998e-06 2024-02-05T04:53:23.733 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.733 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.733 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:23.733 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:10.002253+0000", 2024-02-05T04:53:23.733 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.7680000000000003e-06 2024-02-05T04:53:23.733 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.733 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.733 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:23.733 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:10.002258+0000", 2024-02-05T04:53:23.733 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.78e-06 2024-02-05T04:53:23.733 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.734 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.734 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:23.734 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:10.002290+0000", 2024-02-05T04:53:23.734 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.1977999999999998e-05 2024-02-05T04:53:23.734 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.734 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.734 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:53:23.734 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:10.002301+0000", 2024-02-05T04:53:23.734 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1568e-05 2024-02-05T04:53:23.734 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.734 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.734 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-02-05T04:53:23.734 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:10.006223+0000", 2024-02-05T04:53:23.734 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0039221200000000003 2024-02-05T04:53:23.734 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.735 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.735 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:23.735 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:10.006236+0000", 2024-02-05T04:53:23.735 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2665e-05 2024-02-05T04:53:23.735 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:23.735 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:23.735 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:23.735 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.735 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.735 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.4209.0:206 1.0 e26/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 26'152, mlcod=26'147)", 2024-02-05T04:53:23.735 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:11.937628+0000", 2024-02-05T04:53:23.735 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 11.726723005, 2024-02-05T04:53:23.735 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0042675109999999999, 2024-02-05T04:53:23.735 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:23.735 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:23.736 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:23.736 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.736 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:23.736 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:11.937628+0000", 2024-02-05T04:53:23.736 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:23.736 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.736 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.736 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:23.736 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:11.937628+0000", 2024-02-05T04:53:23.736 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:23.736 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.736 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.736 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:23.736 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:11.937629+0000", 2024-02-05T04:53:23.736 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.0199999999999998e-07 2024-02-05T04:53:23.737 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.737 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.737 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:23.737 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:11.937697+0000", 2024-02-05T04:53:23.737 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.8546000000000004e-05 2024-02-05T04:53:23.737 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.737 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.737 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:23.737 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:11.937704+0000", 2024-02-05T04:53:23.737 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.3049999999999998e-06 2024-02-05T04:53:23.737 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.737 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.737 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:23.737 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:11.937708+0000", 2024-02-05T04:53:23.737 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.5759999999999999e-06 2024-02-05T04:53:23.738 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.738 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.738 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:23.738 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:11.937734+0000", 2024-02-05T04:53:23.738 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.5901000000000002e-05 2024-02-05T04:53:23.738 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.738 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.738 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:53:23.738 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:11.937745+0000", 2024-02-05T04:53:23.738 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1012000000000001e-05 2024-02-05T04:53:23.738 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.738 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.738 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-02-05T04:53:23.738 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:11.941858+0000", 2024-02-05T04:53:23.738 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0041132269999999997 2024-02-05T04:53:23.738 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.739 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.739 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:23.739 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:11.941896+0000", 2024-02-05T04:53:23.739 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.7642000000000003e-05 2024-02-05T04:53:23.739 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:23.739 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:23.739 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:23.739 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.739 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.739 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.4209.0:207 1.0 e26/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 26'153, mlcod=26'147)", 2024-02-05T04:53:23.739 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:11.938281+0000", 2024-02-05T04:53:23.739 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 11.72607041, 2024-02-05T04:53:23.739 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0036428110000000001, 2024-02-05T04:53:23.739 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:23.739 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:23.739 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:23.740 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.740 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:23.740 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:11.938281+0000", 2024-02-05T04:53:23.740 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:23.740 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.740 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.740 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:23.740 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:11.938281+0000", 2024-02-05T04:53:23.740 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:23.740 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.740 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.740 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:23.740 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:11.938281+0000", 2024-02-05T04:53:23.740 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.34e-07 2024-02-05T04:53:23.741 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.741 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.741 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:23.741 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:11.938284+0000", 2024-02-05T04:53:23.741 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.7180000000000001e-06 2024-02-05T04:53:23.741 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.741 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.741 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:23.741 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:11.938290+0000", 2024-02-05T04:53:23.741 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.1739999999999997e-06 2024-02-05T04:53:23.741 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.741 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.741 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:23.741 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:11.938294+0000", 2024-02-05T04:53:23.741 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.5590000000000001e-06 2024-02-05T04:53:23.741 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.742 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.742 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:23.742 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:11.938321+0000", 2024-02-05T04:53:23.742 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.7460000000000001e-05 2024-02-05T04:53:23.742 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.742 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.742 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:53:23.742 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:11.938332+0000", 2024-02-05T04:53:23.742 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0655e-05 2024-02-05T04:53:23.742 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.742 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.742 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-02-05T04:53:23.742 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:11.941898+0000", 2024-02-05T04:53:23.742 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0035665269999999999 2024-02-05T04:53:23.742 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.742 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.743 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:23.743 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:11.941924+0000", 2024-02-05T04:53:23.743 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.5384e-05 2024-02-05T04:53:23.743 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:23.743 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:23.743 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:23.743 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.743 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.743 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.4209.0:209 1.0 e26/23 1:30306672:devicehealth::main.db.0000000000000000:head v 26'155, mlcod=26'152)", 2024-02-05T04:53:23.743 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:11.943219+0000", 2024-02-05T04:53:23.743 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 11.721132661, 2024-02-05T04:53:23.743 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0038190519999999999, 2024-02-05T04:53:23.743 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:23.743 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:23.744 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:23.744 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.744 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:23.744 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:11.943219+0000", 2024-02-05T04:53:23.744 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:23.744 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.744 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.744 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:23.744 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:11.943219+0000", 2024-02-05T04:53:23.744 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:23.744 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.744 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.744 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:23.744 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:11.943219+0000", 2024-02-05T04:53:23.744 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.8799999999999998e-07 2024-02-05T04:53:23.745 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.745 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.745 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:23.745 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:11.943282+0000", 2024-02-05T04:53:23.745 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.2640999999999999e-05 2024-02-05T04:53:23.745 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.745 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.745 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:23.745 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:11.943288+0000", 2024-02-05T04:53:23.745 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.6050000000000003e-06 2024-02-05T04:53:23.745 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.745 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.745 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:23.745 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:11.943292+0000", 2024-02-05T04:53:23.745 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.9709999999999998e-06 2024-02-05T04:53:23.745 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.746 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.746 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:23.746 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:11.943329+0000", 2024-02-05T04:53:23.746 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.7129000000000003e-05 2024-02-05T04:53:23.746 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.746 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.746 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:53:23.746 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:11.943341+0000", 2024-02-05T04:53:23.746 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1999e-05 2024-02-05T04:53:23.746 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.746 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.746 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-02-05T04:53:23.746 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:11.947010+0000", 2024-02-05T04:53:23.746 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0036685590000000001 2024-02-05T04:53:23.746 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:23.747 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:23.747 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:23.747 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:11.947038+0000", 2024-02-05T04:53:23.747 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.7860000000000001e-05 2024-02-05T04:53:23.747 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:23.747 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:23.747 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:23.747 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:23.747 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:23.747 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-02-05T04:53:23.749 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-02-05T04:53:23.749 INFO:tasks.ceph.osd.6.smithi150.stderr:2024-02-05T04:53:23.747+0000 7f6bea9f0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34015) UID: 0 2024-02-05T04:53:23.849 INFO:tasks.ceph.osd.0.smithi062.stderr:2024-02-05T04:53:23.848+0000 7fece8e03640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34688) UID: 0 2024-02-05T04:53:23.895 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-02-05T04:53:23.895 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [], 2024-02-05T04:53:23.895 INFO:teuthology.orchestra.run.smithi150.stdout: "num_ops": 0 2024-02-05T04:53:23.895 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-02-05T04:53:23.904 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-02-05T04:53:23.950 INFO:tasks.ceph.osd.12.smithi062.stderr:2024-02-05T04:53:23.949+0000 7f5e5847e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 34695) UID: 0 2024-02-05T04:53:24.009 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-02-05T04:53:24.010 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [], 2024-02-05T04:53:24.010 INFO:teuthology.orchestra.run.smithi150.stdout: "complaint_time": 30, 2024-02-05T04:53:24.010 INFO:teuthology.orchestra.run.smithi150.stdout: "num_blocked_ops": 0 2024-02-05T04:53:24.010 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-02-05T04:53:24.018 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-02-05T04:53:24.052 INFO:tasks.ceph.osd.15.smithi196.stderr:2024-02-05T04:53:24.051+0000 7f1facc45640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 34611) UID: 0 2024-02-05T04:53:24.070 INFO:teuthology.orchestra.run.smithi062.stderr:nodeep-scrub is set 2024-02-05T04:53:24.070 INFO:teuthology.orchestra.run.smithi062.stderr:reweighted osd.3 to 1 (10000) 2024-02-05T04:53:24.072 INFO:teuthology.orchestra.run.smithi062.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-02-05T04:53:24.094 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-02-05T04:53:24.128 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-02-05T04:53:24.128 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 20, 2024-02-05T04:53:24.128 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 600, 2024-02-05T04:53:24.128 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [] 2024-02-05T04:53:24.128 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-02-05T04:53:24.137 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-02-05T04:53:24.153 INFO:tasks.ceph.osd.7.smithi196.stderr:2024-02-05T04:53:24.152+0000 7fb281419640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34606) UID: 0 2024-02-05T04:53:24.242 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-02-05T04:53:24.242 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [], 2024-02-05T04:53:24.242 INFO:teuthology.orchestra.run.smithi150.stdout: "num_ops": 0 2024-02-05T04:53:24.242 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-02-05T04:53:24.251 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-02-05T04:53:24.253 INFO:tasks.ceph.osd.1.smithi091.stderr:2024-02-05T04:53:24.252+0000 7ff9dc283640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34275) UID: 0 2024-02-05T04:53:24.354 INFO:tasks.ceph.osd.14.smithi150.stderr:2024-02-05T04:53:24.353+0000 7f5f81083640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 34020) UID: 0 2024-02-05T04:53:24.357 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-02-05T04:53:24.357 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [], 2024-02-05T04:53:24.357 INFO:teuthology.orchestra.run.smithi150.stdout: "complaint_time": 30, 2024-02-05T04:53:24.357 INFO:teuthology.orchestra.run.smithi150.stdout: "num_blocked_ops": 0 2024-02-05T04:53:24.357 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-02-05T04:53:24.366 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-02-05T04:53:24.455 INFO:tasks.ceph.osd.14.smithi150.stderr:2024-02-05T04:53:24.454+0000 7f5f81083640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 34020) UID: 0 2024-02-05T04:53:24.516 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-02-05T04:53:24.516 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 20, 2024-02-05T04:53:24.516 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 600, 2024-02-05T04:53:24.516 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [] 2024-02-05T04:53:24.516 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-02-05T04:53:24.525 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-02-05T04:53:24.566 INFO:tasks.ceph.osd.10.smithi150.stderr:2024-02-05T04:53:24.565+0000 7f3075a4d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 34018) UID: 0 2024-02-05T04:53:24.657 INFO:tasks.ceph.osd.7.smithi196.stderr:2024-02-05T04:53:24.656+0000 7fb281419640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34606) UID: 0 2024-02-05T04:53:24.671 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-02-05T04:53:24.671 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [], 2024-02-05T04:53:24.671 INFO:teuthology.orchestra.run.smithi150.stdout: "num_ops": 0 2024-02-05T04:53:24.672 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-02-05T04:53:24.680 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-02-05T04:53:24.758 INFO:tasks.ceph.osd.15.smithi196.stderr:2024-02-05T04:53:24.758+0000 7f1facc45640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 34611) UID: 0 2024-02-05T04:53:24.788 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-02-05T04:53:24.788 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [], 2024-02-05T04:53:24.788 INFO:teuthology.orchestra.run.smithi150.stdout: "complaint_time": 30, 2024-02-05T04:53:24.788 INFO:teuthology.orchestra.run.smithi150.stdout: "num_blocked_ops": 0 2024-02-05T04:53:24.788 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-02-05T04:53:24.796 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-02-05T04:53:24.859 INFO:tasks.ceph.osd.10.smithi150.stderr:2024-02-05T04:53:24.858+0000 7f3075a4d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 34018) UID: 0 2024-02-05T04:53:24.902 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-02-05T04:53:24.902 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 20, 2024-02-05T04:53:24.902 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 600, 2024-02-05T04:53:24.903 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [] 2024-02-05T04:53:24.903 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-02-05T04:53:24.911 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-02-05T04:53:24.960 INFO:tasks.ceph.osd.11.smithi196.stderr:2024-02-05T04:53:24.959+0000 7f85a8d86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 34609) UID: 0 2024-02-05T04:53:25.027 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T04:53:25.028 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-02-05T04:53:25.028 INFO:teuthology.orchestra.run.smithi196.stdout: "num_ops": 0 2024-02-05T04:53:25.028 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T04:53:25.036 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-02-05T04:53:25.060 INFO:tasks.ceph.osd.5.smithi091.stderr:2024-02-05T04:53:25.060+0000 7f63983b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34277) UID: 0 2024-02-05T04:53:25.097 DEBUG:teuthology.orchestra.run.smithi196:> CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --ec-pool --max-ops 400 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 0 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 100 --pool unique_pool_0 2024-02-05T04:53:25.097 INFO:teuthology.orchestra.run.smithi062.stderr:reweighted osd.4 to 1 (10000) 2024-02-05T04:53:25.113 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-02-05T04:53:25.146 INFO:tasks.rados.rados.0.smithi196.stdout:adding op weight read -> 100 2024-02-05T04:53:25.146 INFO:tasks.rados.rados.0.smithi196.stdout:adding op weight delete -> 50 2024-02-05T04:53:25.147 INFO:tasks.rados.rados.0.smithi196.stdout:adding op weight snap_create -> 50 2024-02-05T04:53:25.147 INFO:tasks.rados.rados.0.smithi196.stdout:adding op weight snap_remove -> 50 2024-02-05T04:53:25.147 INFO:tasks.rados.rados.0.smithi196.stdout:adding op weight rollback -> 50 2024-02-05T04:53:25.147 INFO:tasks.rados.rados.0.smithi196.stdout:adding op weight setattr -> 25 2024-02-05T04:53:25.147 INFO:tasks.rados.rados.0.smithi196.stdout:adding op weight rmattr -> 25 2024-02-05T04:53:25.147 INFO:tasks.rados.rados.0.smithi196.stdout:adding op weight copy_from -> 50 2024-02-05T04:53:25.147 INFO:tasks.rados.rados.0.smithi196.stdout:adding op weight append -> 100 2024-02-05T04:53:25.147 INFO:tasks.rados.rados.0.smithi196.stdout:ceph version 19.0.0-658-gd4ed20cf (d4ed20cfb238ef2c47d64ca4533eca00b8968b98) squid (dev) 2024-02-05T04:53:25.147 INFO:tasks.rados.rados.0.smithi196.stdout:Configuration: 2024-02-05T04:53:25.147 INFO:tasks.rados.rados.0.smithi196.stdout: Number of operations: 400 2024-02-05T04:53:25.147 INFO:tasks.rados.rados.0.smithi196.stdout: Number of objects: 50 2024-02-05T04:53:25.147 INFO:tasks.rados.rados.0.smithi196.stdout: Max in flight operations: 16 2024-02-05T04:53:25.147 INFO:tasks.rados.rados.0.smithi196.stdout: Object size (in bytes): 4000000 2024-02-05T04:53:25.147 INFO:tasks.rados.rados.0.smithi196.stdout: Write stride min: 400000 2024-02-05T04:53:25.147 INFO:tasks.rados.rados.0.smithi196.stdout: Write stride max: 800000 2024-02-05T04:53:25.161 INFO:tasks.ceph.osd.15.smithi196.stderr:2024-02-05T04:53:25.161+0000 7f1facc45640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 34611) UID: 0 2024-02-05T04:53:25.185 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T04:53:25.185 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-02-05T04:53:25.185 INFO:teuthology.orchestra.run.smithi196.stdout: "complaint_time": 30, 2024-02-05T04:53:25.185 INFO:teuthology.orchestra.run.smithi196.stdout: "num_blocked_ops": 0 2024-02-05T04:53:25.185 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T04:53:25.194 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-02-05T04:53:25.263 INFO:tasks.ceph.osd.6.smithi150.stderr:2024-02-05T04:53:25.262+0000 7f6bea9f0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34015) UID: 0 2024-02-05T04:53:25.340 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T04:53:25.340 INFO:teuthology.orchestra.run.smithi196.stdout: "size": 20, 2024-02-05T04:53:25.341 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 600, 2024-02-05T04:53:25.341 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [] 2024-02-05T04:53:25.341 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T04:53:25.349 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-02-05T04:53:25.393 INFO:tasks.ceph.osd.11.smithi196.stderr:2024-02-05T04:53:25.392+0000 7f85a8d86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 34609) UID: 0 2024-02-05T04:53:25.415 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T04:53:25.456 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T04:53:25.456 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-02-05T04:53:25.456 INFO:teuthology.orchestra.run.smithi196.stdout: "num_ops": 0 2024-02-05T04:53:25.456 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T04:53:25.465 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-02-05T04:53:25.466 INFO:tasks.ceph.osd.5.smithi091.stderr:2024-02-05T04:53:25.464+0000 7f63983b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34277) UID: 0 2024-02-05T04:53:25.540 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0: {} 2024-02-05T04:53:25.540 INFO:teuthology.orchestra.run.smithi062.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T04:53:25.558 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1: {} 2024-02-05T04:53:25.559 INFO:teuthology.orchestra.run.smithi062.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T04:53:25.565 INFO:tasks.ceph.osd.14.smithi150.stderr:2024-02-05T04:53:25.564+0000 7f5f81083640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 34020) UID: 0 2024-02-05T04:53:25.576 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2: {} 2024-02-05T04:53:25.576 INFO:teuthology.orchestra.run.smithi062.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T04:53:25.595 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3: {} 2024-02-05T04:53:25.595 INFO:teuthology.orchestra.run.smithi062.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T04:53:25.611 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4: {} 2024-02-05T04:53:25.611 INFO:teuthology.orchestra.run.smithi062.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T04:53:25.612 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T04:53:25.612 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-02-05T04:53:25.612 INFO:teuthology.orchestra.run.smithi196.stdout: "complaint_time": 30, 2024-02-05T04:53:25.612 INFO:teuthology.orchestra.run.smithi196.stdout: "num_blocked_ops": 0 2024-02-05T04:53:25.612 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T04:53:25.622 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-02-05T04:53:25.629 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5: {} 2024-02-05T04:53:25.629 INFO:teuthology.orchestra.run.smithi062.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T04:53:25.647 INFO:teuthology.orchestra.run.smithi062.stdout:osd.6: {} 2024-02-05T04:53:25.647 INFO:teuthology.orchestra.run.smithi062.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T04:53:25.665 INFO:teuthology.orchestra.run.smithi062.stdout:osd.7: {} 2024-02-05T04:53:25.665 INFO:teuthology.orchestra.run.smithi062.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T04:53:25.666 INFO:tasks.ceph.osd.4.smithi062.stderr:2024-02-05T04:53:25.664+0000 7f708c308640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34689) UID: 0 2024-02-05T04:53:25.686 INFO:teuthology.orchestra.run.smithi062.stdout:osd.8: {} 2024-02-05T04:53:25.686 INFO:teuthology.orchestra.run.smithi062.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-05T04:53:25.704 INFO:teuthology.orchestra.run.smithi062.stdout:osd.9: {} 2024-02-05T04:53:25.704 INFO:teuthology.orchestra.run.smithi062.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-05T04:53:25.722 INFO:teuthology.orchestra.run.smithi062.stdout:osd.10: {} 2024-02-05T04:53:25.723 INFO:teuthology.orchestra.run.smithi062.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-05T04:53:25.726 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:25.727 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:25.735 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-02-05T04:53:25.740 INFO:teuthology.orchestra.run.smithi062.stdout:osd.11: {} 2024-02-05T04:53:25.740 INFO:teuthology.orchestra.run.smithi062.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-05T04:53:25.756 INFO:teuthology.orchestra.run.smithi062.stdout:osd.12: {} 2024-02-05T04:53:25.757 INFO:teuthology.orchestra.run.smithi062.stderr:osd.12: osd_enable_op_tracker = '' 2024-02-05T04:53:25.774 INFO:teuthology.orchestra.run.smithi062.stdout:osd.13: {} 2024-02-05T04:53:25.774 INFO:teuthology.orchestra.run.smithi062.stderr:osd.13: osd_enable_op_tracker = '' 2024-02-05T04:53:25.779 INFO:tasks.ceph.osd.7.smithi196.stderr:2024-02-05T04:53:25.778+0000 7fb281419640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34606) UID: 0 2024-02-05T04:53:25.792 INFO:teuthology.orchestra.run.smithi062.stdout:osd.14: {} 2024-02-05T04:53:25.793 INFO:teuthology.orchestra.run.smithi062.stderr:osd.14: osd_enable_op_tracker = '' 2024-02-05T04:53:25.811 INFO:teuthology.orchestra.run.smithi062.stdout:osd.15: {} 2024-02-05T04:53:25.811 INFO:teuthology.orchestra.run.smithi062.stderr:osd.15: osd_enable_op_tracker = '' 2024-02-05T04:53:25.843 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:25.843 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:25.851 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-02-05T04:53:25.867 INFO:tasks.ceph.osd.2.smithi150.stderr:2024-02-05T04:53:25.866+0000 7f7337bf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 34014) UID: 0 2024-02-05T04:53:25.968 INFO:tasks.ceph.osd.7.smithi196.stderr:2024-02-05T04:53:25.967+0000 7fb281419640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34606) UID: 0 2024-02-05T04:53:25.997 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:25.997 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:26.006 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-02-05T04:53:26.069 INFO:tasks.ceph.osd.4.smithi062.stderr:2024-02-05T04:53:26.068+0000 7f708c308640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34689) UID: 0 2024-02-05T04:53:26.094 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-02-05T04:53:26.109 INFO:teuthology.orchestra.run.smithi062.stderr:reweighted osd.5 to 1 (10000) 2024-02-05T04:53:26.114 INFO:tasks.rados.rados.0.smithi196.stdout:1: write initial oid 1 2024-02-05T04:53:26.114 INFO:tasks.rados.rados.0.smithi196.stdout:1: seq_num 0 ranges {0=32768} 2024-02-05T04:53:26.114 INFO:tasks.rados.rados.0.smithi196.stdout:1: writing smithi19639362-1 from 0 to 32768 tid 1 2024-02-05T04:53:26.115 INFO:tasks.rados.rados.0.smithi196.stdout:2: write initial oid 2 2024-02-05T04:53:26.115 INFO:tasks.rados.rados.0.smithi196.stdout:2: seq_num 1 ranges {0=32768} 2024-02-05T04:53:26.115 INFO:tasks.rados.rados.0.smithi196.stdout:2: writing smithi19639362-2 from 0 to 32768 tid 1 2024-02-05T04:53:26.115 INFO:tasks.rados.rados.0.smithi196.stdout:3: write initial oid 3 2024-02-05T04:53:26.115 INFO:tasks.rados.rados.0.smithi196.stdout:3: seq_num 2 ranges {0=49152} 2024-02-05T04:53:26.116 INFO:tasks.rados.rados.0.smithi196.stdout:3: writing smithi19639362-3 from 0 to 49152 tid 1 2024-02-05T04:53:26.116 INFO:tasks.rados.rados.0.smithi196.stdout:4: write initial oid 4 2024-02-05T04:53:26.116 INFO:tasks.rados.rados.0.smithi196.stdout:4: seq_num 3 ranges {0=65536} 2024-02-05T04:53:26.116 INFO:tasks.rados.rados.0.smithi196.stdout:4: writing smithi19639362-4 from 0 to 65536 tid 1 2024-02-05T04:53:26.116 INFO:tasks.rados.rados.0.smithi196.stdout:5: write initial oid 5 2024-02-05T04:53:26.116 INFO:tasks.rados.rados.0.smithi196.stdout:5: seq_num 4 ranges {0=81920} 2024-02-05T04:53:26.116 INFO:tasks.rados.rados.0.smithi196.stdout:5: writing smithi19639362-5 from 0 to 81920 tid 1 2024-02-05T04:53:26.117 INFO:tasks.rados.rados.0.smithi196.stdout:6: write initial oid 6 2024-02-05T04:53:26.117 INFO:tasks.rados.rados.0.smithi196.stdout:6: seq_num 5 ranges {0=98304} 2024-02-05T04:53:26.117 INFO:tasks.rados.rados.0.smithi196.stdout:6: writing smithi19639362-6 from 0 to 98304 tid 1 2024-02-05T04:53:26.117 INFO:tasks.rados.rados.0.smithi196.stdout:7: write initial oid 7 2024-02-05T04:53:26.117 INFO:tasks.rados.rados.0.smithi196.stdout:7: seq_num 6 ranges {0=114688} 2024-02-05T04:53:26.118 INFO:tasks.rados.rados.0.smithi196.stdout:7: writing smithi19639362-7 from 0 to 114688 tid 1 2024-02-05T04:53:26.118 INFO:tasks.rados.rados.0.smithi196.stdout:8: write initial oid 8 2024-02-05T04:53:26.118 INFO:tasks.rados.rados.0.smithi196.stdout:8: seq_num 7 ranges {0=131072} 2024-02-05T04:53:26.119 INFO:tasks.rados.rados.0.smithi196.stdout:8: writing smithi19639362-8 from 0 to 131072 tid 1 2024-02-05T04:53:26.119 INFO:tasks.rados.rados.0.smithi196.stdout:1: finishing write tid 1 to smithi19639362-1 2024-02-05T04:53:26.119 INFO:tasks.rados.rados.0.smithi196.stdout:9: write initial oid 9 2024-02-05T04:53:26.119 INFO:tasks.rados.rados.0.smithi196.stdout:9: seq_num 8 ranges {0=147456} 2024-02-05T04:53:26.120 INFO:tasks.rados.rados.0.smithi196.stdout:9: writing smithi19639362-9 from 0 to 147456 tid 1 2024-02-05T04:53:26.120 INFO:tasks.rados.rados.0.smithi196.stdout:10: write initial oid 10 2024-02-05T04:53:26.120 INFO:tasks.rados.rados.0.smithi196.stdout:10: seq_num 9 ranges {0=163840} 2024-02-05T04:53:26.121 INFO:tasks.rados.rados.0.smithi196.stdout:10: writing smithi19639362-10 from 0 to 163840 tid 1 2024-02-05T04:53:26.121 INFO:tasks.rados.rados.0.smithi196.stdout:1: finishing write tid 2 to smithi19639362-1 2024-02-05T04:53:26.121 INFO:tasks.rados.rados.0.smithi196.stdout:11: write initial oid 11 2024-02-05T04:53:26.121 INFO:tasks.rados.rados.0.smithi196.stdout:11: seq_num 10 ranges {0=180224} 2024-02-05T04:53:26.123 INFO:tasks.rados.rados.0.smithi196.stdout:11: writing smithi19639362-11 from 0 to 180224 tid 1 2024-02-05T04:53:26.124 INFO:tasks.rados.rados.0.smithi196.stdout:12: write initial oid 12 2024-02-05T04:53:26.124 INFO:tasks.rados.rados.0.smithi196.stdout:12: seq_num 11 ranges {0=196608} 2024-02-05T04:53:26.124 INFO:tasks.rados.rados.0.smithi196.stdout:12: writing smithi19639362-12 from 0 to 196608 tid 1 2024-02-05T04:53:26.124 INFO:tasks.rados.rados.0.smithi196.stdout:13: write initial oid 13 2024-02-05T04:53:26.124 INFO:tasks.rados.rados.0.smithi196.stdout:13: seq_num 12 ranges {0=212992} 2024-02-05T04:53:26.125 INFO:tasks.rados.rados.0.smithi196.stdout:13: writing smithi19639362-13 from 0 to 212992 tid 1 2024-02-05T04:53:26.126 INFO:tasks.rados.rados.0.smithi196.stdout:2: finishing write tid 1 to smithi19639362-2 2024-02-05T04:53:26.126 INFO:tasks.rados.rados.0.smithi196.stdout:14: write initial oid 14 2024-02-05T04:53:26.126 INFO:tasks.rados.rados.0.smithi196.stdout:14: seq_num 13 ranges {0=229376} 2024-02-05T04:53:26.127 INFO:tasks.rados.rados.0.smithi196.stdout:14: writing smithi19639362-14 from 0 to 229376 tid 1 2024-02-05T04:53:26.128 INFO:tasks.rados.rados.0.smithi196.stdout:4: finishing write tid 1 to smithi19639362-4 2024-02-05T04:53:26.128 INFO:tasks.rados.rados.0.smithi196.stdout:4: finishing write tid 2 to smithi19639362-4 2024-02-05T04:53:26.128 INFO:tasks.rados.rados.0.smithi196.stdout:3: finishing write tid 1 to smithi19639362-3 2024-02-05T04:53:26.128 INFO:tasks.rados.rados.0.smithi196.stdout:5: finishing write tid 1 to smithi19639362-5 2024-02-05T04:53:26.128 INFO:tasks.rados.rados.0.smithi196.stdout:3: finishing write tid 2 to smithi19639362-3 2024-02-05T04:53:26.128 INFO:tasks.rados.rados.0.smithi196.stdout:15: write initial oid 15 2024-02-05T04:53:26.128 INFO:tasks.rados.rados.0.smithi196.stdout:15: seq_num 14 ranges {0=245760} 2024-02-05T04:53:26.129 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-02-05T04:53:26.130 INFO:tasks.rados.rados.0.smithi196.stdout:15: writing smithi19639362-15 from 0 to 245760 tid 1 2024-02-05T04:53:26.130 INFO:tasks.rados.rados.0.smithi196.stdout:16: write initial oid 16 2024-02-05T04:53:26.130 INFO:tasks.rados.rados.0.smithi196.stdout:16: seq_num 15 ranges {0=262144} 2024-02-05T04:53:26.132 INFO:tasks.rados.rados.0.smithi196.stdout:16: writing smithi19639362-16 from 0 to 262144 tid 1 2024-02-05T04:53:26.132 INFO:tasks.rados.rados.0.smithi196.stdout: waiting on 16 2024-02-05T04:53:26.132 INFO:tasks.rados.rados.0.smithi196.stdout:2: finishing write tid 2 to smithi19639362-2 2024-02-05T04:53:26.132 INFO:tasks.rados.rados.0.smithi196.stdout:1: finishing write tid 3 to smithi19639362-1 2024-02-05T04:53:26.132 INFO:tasks.rados.rados.0.smithi196.stdout:1: oid 1 updating version 0 to 1 2024-02-05T04:53:26.132 INFO:tasks.rados.rados.0.smithi196.stdout:1: oid 1 updating version 1 to 2 2024-02-05T04:53:26.132 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 1 v 2 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-02-05T04:53:26.132 INFO:tasks.rados.rados.0.smithi196.stdout:1: left oid 1 (ObjNum 0 snap 0 seq_num 0) 2024-02-05T04:53:26.132 INFO:tasks.rados.rados.0.smithi196.stdout:4: finishing write tid 3 to smithi19639362-4 2024-02-05T04:53:26.132 INFO:tasks.rados.rados.0.smithi196.stdout:4: oid 4 updating version 0 to 1 2024-02-05T04:53:26.132 INFO:tasks.rados.rados.0.smithi196.stdout:4: oid 4 updating version 1 to 2 2024-02-05T04:53:26.133 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 4 v 2 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-02-05T04:53:26.133 INFO:tasks.rados.rados.0.smithi196.stdout:4: left oid 4 (ObjNum 3 snap 0 seq_num 3) 2024-02-05T04:53:26.133 INFO:tasks.rados.rados.0.smithi196.stdout:5: finishing write tid 2 to smithi19639362-5 2024-02-05T04:53:26.133 INFO:tasks.rados.rados.0.smithi196.stdout:7: finishing write tid 1 to smithi19639362-7 2024-02-05T04:53:26.133 INFO:tasks.rados.rados.0.smithi196.stdout:6: finishing write tid 1 to smithi19639362-6 2024-02-05T04:53:26.133 INFO:tasks.rados.rados.0.smithi196.stdout:7: finishing write tid 2 to smithi19639362-7 2024-02-05T04:53:26.133 INFO:tasks.rados.rados.0.smithi196.stdout:6: finishing write tid 2 to smithi19639362-6 2024-02-05T04:53:26.133 INFO:tasks.rados.rados.0.smithi196.stdout:1: done (15 left) 2024-02-05T04:53:26.133 INFO:tasks.rados.rados.0.smithi196.stdout:4: done (14 left) 2024-02-05T04:53:26.133 INFO:tasks.rados.rados.0.smithi196.stdout:17: write initial oid 17 2024-02-05T04:53:26.133 INFO:tasks.rados.rados.0.smithi196.stdout:17: seq_num 16 ranges {0=278528} 2024-02-05T04:53:26.134 INFO:tasks.rados.rados.0.smithi196.stdout:17: writing smithi19639362-17 from 0 to 278528 tid 1 2024-02-05T04:53:26.134 INFO:tasks.rados.rados.0.smithi196.stdout:18: write initial oid 18 2024-02-05T04:53:26.134 INFO:tasks.rados.rados.0.smithi196.stdout:18: seq_num 17 ranges {0=294912} 2024-02-05T04:53:26.136 INFO:tasks.rados.rados.0.smithi196.stdout:18: writing smithi19639362-18 from 0 to 294912 tid 1 2024-02-05T04:53:26.136 INFO:tasks.rados.rados.0.smithi196.stdout: waiting on 16 2024-02-05T04:53:26.136 INFO:tasks.rados.rados.0.smithi196.stdout:10: finishing write tid 1 to smithi19639362-10 2024-02-05T04:53:26.137 INFO:tasks.rados.rados.0.smithi196.stdout:10: finishing write tid 2 to smithi19639362-10 2024-02-05T04:53:26.137 INFO:tasks.rados.rados.0.smithi196.stdout:11: finishing write tid 1 to smithi19639362-11 2024-02-05T04:53:26.137 INFO:tasks.rados.rados.0.smithi196.stdout:11: finishing write tid 2 to smithi19639362-11 2024-02-05T04:53:26.137 INFO:tasks.rados.rados.0.smithi196.stdout:5: finishing write tid 3 to smithi19639362-5 2024-02-05T04:53:26.137 INFO:tasks.rados.rados.0.smithi196.stdout:5: oid 5 updating version 0 to 1 2024-02-05T04:53:26.137 INFO:tasks.rados.rados.0.smithi196.stdout:5: oid 5 updating version 1 to 2 2024-02-05T04:53:26.137 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 5 v 2 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-02-05T04:53:26.137 INFO:tasks.rados.rados.0.smithi196.stdout:5: left oid 5 (ObjNum 4 snap 0 seq_num 4) 2024-02-05T04:53:26.137 INFO:tasks.rados.rados.0.smithi196.stdout:11: finishing write tid 3 to smithi19639362-11 2024-02-05T04:53:26.137 INFO:tasks.rados.rados.0.smithi196.stdout:11: oid 11 updating version 0 to 2 2024-02-05T04:53:26.137 INFO:tasks.rados.rados.0.smithi196.stdout:11: oid 11 version 2 is already newer than 1 2024-02-05T04:53:26.137 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 11 v 2 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-02-05T04:53:26.137 INFO:tasks.rados.rados.0.smithi196.stdout:11: left oid 11 (ObjNum 10 snap 0 seq_num 10) 2024-02-05T04:53:26.137 INFO:tasks.rados.rados.0.smithi196.stdout:5: done (15 left) 2024-02-05T04:53:26.137 INFO:tasks.rados.rados.0.smithi196.stdout:11: done (14 left) 2024-02-05T04:53:26.137 INFO:tasks.rados.rados.0.smithi196.stdout:19: write initial oid 19 2024-02-05T04:53:26.138 INFO:tasks.rados.rados.0.smithi196.stdout:19: seq_num 18 ranges {0=311296} 2024-02-05T04:53:26.138 INFO:tasks.rados.rados.0.smithi196.stdout:19: writing smithi19639362-19 from 0 to 311296 tid 1 2024-02-05T04:53:26.138 INFO:tasks.rados.rados.0.smithi196.stdout:20: write initial oid 20 2024-02-05T04:53:26.138 INFO:tasks.rados.rados.0.smithi196.stdout:20: seq_num 19 ranges {0=327680} 2024-02-05T04:53:26.141 INFO:tasks.rados.rados.0.smithi196.stdout:20: writing smithi19639362-20 from 0 to 327680 tid 1 2024-02-05T04:53:26.141 INFO:tasks.rados.rados.0.smithi196.stdout: waiting on 16 2024-02-05T04:53:26.141 INFO:tasks.rados.rados.0.smithi196.stdout:15: finishing write tid 1 to smithi19639362-15 2024-02-05T04:53:26.141 INFO:tasks.rados.rados.0.smithi196.stdout:15: finishing write tid 2 to smithi19639362-15 2024-02-05T04:53:26.141 INFO:tasks.rados.rados.0.smithi196.stdout:9: finishing write tid 1 to smithi19639362-9 2024-02-05T04:53:26.141 INFO:tasks.rados.rados.0.smithi196.stdout:16: finishing write tid 1 to smithi19639362-16 2024-02-05T04:53:26.142 INFO:tasks.rados.rados.0.smithi196.stdout:9: finishing write tid 2 to smithi19639362-9 2024-02-05T04:53:26.142 INFO:tasks.rados.rados.0.smithi196.stdout:16: finishing write tid 2 to smithi19639362-16 2024-02-05T04:53:26.142 INFO:tasks.rados.rados.0.smithi196.stdout:17: finishing write tid 1 to smithi19639362-17 2024-02-05T04:53:26.142 INFO:tasks.rados.rados.0.smithi196.stdout:17: finishing write tid 2 to smithi19639362-17 2024-02-05T04:53:26.142 INFO:tasks.rados.rados.0.smithi196.stdout:3: finishing write tid 3 to smithi19639362-3 2024-02-05T04:53:26.142 INFO:tasks.rados.rados.0.smithi196.stdout:3: oid 3 updating version 0 to 1 2024-02-05T04:53:26.142 INFO:tasks.rados.rados.0.smithi196.stdout:3: oid 3 updating version 1 to 2 2024-02-05T04:53:26.142 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 3 v 2 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-02-05T04:53:26.142 INFO:tasks.rados.rados.0.smithi196.stdout:3: left oid 3 (ObjNum 2 snap 0 seq_num 2) 2024-02-05T04:53:26.142 INFO:tasks.rados.rados.0.smithi196.stdout:7: finishing write tid 3 to smithi19639362-7 2024-02-05T04:53:26.142 INFO:tasks.rados.rados.0.smithi196.stdout:7: oid 7 updating version 0 to 1 2024-02-05T04:53:26.142 INFO:tasks.rados.rados.0.smithi196.stdout:7: oid 7 updating version 1 to 2 2024-02-05T04:53:26.142 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 7 v 2 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-02-05T04:53:26.142 INFO:tasks.rados.rados.0.smithi196.stdout:7: left oid 7 (ObjNum 6 snap 0 seq_num 6) 2024-02-05T04:53:26.142 INFO:tasks.rados.rados.0.smithi196.stdout:13: finishing write tid 1 to smithi19639362-13 2024-02-05T04:53:26.142 INFO:tasks.rados.rados.0.smithi196.stdout:12: finishing write tid 1 to smithi19639362-12 2024-02-05T04:53:26.142 INFO:tasks.rados.rados.0.smithi196.stdout:3: done (15 left) 2024-02-05T04:53:26.143 INFO:tasks.rados.rados.0.smithi196.stdout:7: done (14 left) 2024-02-05T04:53:26.143 INFO:tasks.rados.rados.0.smithi196.stdout:21: write initial oid 21 2024-02-05T04:53:26.143 INFO:tasks.rados.rados.0.smithi196.stdout:21: seq_num 20 ranges {0=344064} 2024-02-05T04:53:26.143 INFO:tasks.rados.rados.0.smithi196.stdout:21: writing smithi19639362-21 from 0 to 344064 tid 1 2024-02-05T04:53:26.143 INFO:tasks.rados.rados.0.smithi196.stdout:22: write initial oid 22 2024-02-05T04:53:26.143 INFO:tasks.rados.rados.0.smithi196.stdout:22: seq_num 21 ranges {0=360448} 2024-02-05T04:53:26.149 INFO:tasks.rados.rados.0.smithi196.stdout:22: writing smithi19639362-22 from 0 to 360448 tid 1 2024-02-05T04:53:26.149 INFO:tasks.rados.rados.0.smithi196.stdout: waiting on 16 2024-02-05T04:53:26.149 INFO:tasks.rados.rados.0.smithi196.stdout:12: finishing write tid 2 to smithi19639362-12 2024-02-05T04:53:26.149 INFO:tasks.rados.rados.0.smithi196.stdout:6: finishing write tid 3 to smithi19639362-6 2024-02-05T04:53:26.149 INFO:tasks.rados.rados.0.smithi196.stdout:6: oid 6 updating version 0 to 1 2024-02-05T04:53:26.149 INFO:tasks.rados.rados.0.smithi196.stdout:6: oid 6 updating version 1 to 2 2024-02-05T04:53:26.150 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 6 v 2 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-02-05T04:53:26.150 INFO:tasks.rados.rados.0.smithi196.stdout:6: left oid 6 (ObjNum 5 snap 0 seq_num 5) 2024-02-05T04:53:26.150 INFO:tasks.rados.rados.0.smithi196.stdout:8: finishing write tid 1 to smithi19639362-8 2024-02-05T04:53:26.150 INFO:tasks.rados.rados.0.smithi196.stdout:17: finishing write tid 3 to smithi19639362-17 2024-02-05T04:53:26.150 INFO:tasks.rados.rados.0.smithi196.stdout:17: oid 17 updating version 0 to 3 2024-02-05T04:53:26.150 INFO:tasks.rados.rados.0.smithi196.stdout:17: oid 17 updating version 3 to 4 2024-02-05T04:53:26.150 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 17 v 4 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-02-05T04:53:26.151 INFO:tasks.rados.rados.0.smithi196.stdout:17: left oid 17 (ObjNum 16 snap 0 seq_num 16) 2024-02-05T04:53:26.151 INFO:tasks.rados.rados.0.smithi196.stdout:8: finishing write tid 2 to smithi19639362-8 2024-02-05T04:53:26.151 INFO:tasks.rados.rados.0.smithi196.stdout:2: finishing write tid 3 to smithi19639362-2 2024-02-05T04:53:26.151 INFO:tasks.rados.rados.0.smithi196.stdout:2: oid 2 updating version 0 to 1 2024-02-05T04:53:26.152 INFO:tasks.rados.rados.0.smithi196.stdout:2: oid 2 updating version 1 to 2 2024-02-05T04:53:26.152 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 2 v 2 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-02-05T04:53:26.152 INFO:tasks.rados.rados.0.smithi196.stdout:2: left oid 2 (ObjNum 1 snap 0 seq_num 1) 2024-02-05T04:53:26.152 INFO:tasks.rados.rados.0.smithi196.stdout:15: finishing write tid 3 to smithi19639362-15 2024-02-05T04:53:26.152 INFO:tasks.rados.rados.0.smithi196.stdout:15: oid 15 updating version 0 to 2 2024-02-05T04:53:26.152 INFO:tasks.rados.rados.0.smithi196.stdout:15: oid 15 version 2 is already newer than 1 2024-02-05T04:53:26.153 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 15 v 2 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-02-05T04:53:26.153 INFO:tasks.rados.rados.0.smithi196.stdout:15: left oid 15 (ObjNum 14 snap 0 seq_num 14) 2024-02-05T04:53:26.153 INFO:tasks.rados.rados.0.smithi196.stdout:16: finishing write tid 3 to smithi19639362-16 2024-02-05T04:53:26.153 INFO:tasks.rados.rados.0.smithi196.stdout:16: oid 16 updating version 0 to 4 2024-02-05T04:53:26.153 INFO:tasks.rados.rados.0.smithi196.stdout:16: oid 16 version 4 is already newer than 3 2024-02-05T04:53:26.153 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 16 v 4 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-02-05T04:53:26.153 INFO:tasks.rados.rados.0.smithi196.stdout:16: left oid 16 (ObjNum 15 snap 0 seq_num 15) 2024-02-05T04:53:26.153 INFO:tasks.rados.rados.0.smithi196.stdout:10: finishing write tid 3 to smithi19639362-10 2024-02-05T04:53:26.154 INFO:tasks.rados.rados.0.smithi196.stdout:10: oid 10 updating version 0 to 2 2024-02-05T04:53:26.154 INFO:tasks.rados.rados.0.smithi196.stdout:10: oid 10 version 2 is already newer than 1 2024-02-05T04:53:26.154 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 10 v 2 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-02-05T04:53:26.154 INFO:tasks.rados.rados.0.smithi196.stdout:10: left oid 10 (ObjNum 9 snap 0 seq_num 9) 2024-02-05T04:53:26.154 INFO:tasks.rados.rados.0.smithi196.stdout:9: finishing write tid 3 to smithi19639362-9 2024-02-05T04:53:26.154 INFO:tasks.rados.rados.0.smithi196.stdout:9: oid 9 updating version 0 to 3 2024-02-05T04:53:26.154 INFO:tasks.rados.rados.0.smithi196.stdout:9: oid 9 updating version 3 to 4 2024-02-05T04:53:26.154 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 9 v 4 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-02-05T04:53:26.154 INFO:tasks.rados.rados.0.smithi196.stdout:9: left oid 9 (ObjNum 8 snap 0 seq_num 8) 2024-02-05T04:53:26.154 INFO:tasks.rados.rados.0.smithi196.stdout:12: finishing write tid 3 to smithi19639362-12 2024-02-05T04:53:26.154 INFO:tasks.rados.rados.0.smithi196.stdout:12: oid 12 updating version 0 to 4 2024-02-05T04:53:26.154 INFO:tasks.rados.rados.0.smithi196.stdout:12: oid 12 version 4 is already newer than 3 2024-02-05T04:53:26.154 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 12 v 4 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-02-05T04:53:26.154 INFO:tasks.rados.rados.0.smithi196.stdout:12: left oid 12 (ObjNum 11 snap 0 seq_num 11) 2024-02-05T04:53:26.155 INFO:tasks.rados.rados.0.smithi196.stdout:2: done (15 left) 2024-02-05T04:53:26.155 INFO:tasks.rados.rados.0.smithi196.stdout:6: done (14 left) 2024-02-05T04:53:26.155 INFO:tasks.rados.rados.0.smithi196.stdout:9: done (13 left) 2024-02-05T04:53:26.155 INFO:tasks.rados.rados.0.smithi196.stdout:10: done (12 left) 2024-02-05T04:53:26.155 INFO:tasks.rados.rados.0.smithi196.stdout:12: done (11 left) 2024-02-05T04:53:26.155 INFO:tasks.rados.rados.0.smithi196.stdout:15: done (10 left) 2024-02-05T04:53:26.155 INFO:tasks.rados.rados.0.smithi196.stdout:16: done (9 left) 2024-02-05T04:53:26.155 INFO:tasks.rados.rados.0.smithi196.stdout:17: done (8 left) 2024-02-05T04:53:26.156 INFO:tasks.rados.rados.0.smithi196.stdout:23: write initial oid 23 2024-02-05T04:53:26.156 INFO:tasks.rados.rados.0.smithi196.stdout:23: seq_num 22 ranges {0=376832} 2024-02-05T04:53:26.156 INFO:tasks.rados.rados.0.smithi196.stdout:23: writing smithi19639362-23 from 0 to 376832 tid 1 2024-02-05T04:53:26.156 INFO:tasks.rados.rados.0.smithi196.stdout:24: write initial oid 24 2024-02-05T04:53:26.156 INFO:tasks.rados.rados.0.smithi196.stdout:24: seq_num 23 ranges {0=393216} 2024-02-05T04:53:26.157 INFO:tasks.rados.rados.0.smithi196.stdout:24: writing smithi19639362-24 from 0 to 393216 tid 1 2024-02-05T04:53:26.157 INFO:tasks.rados.rados.0.smithi196.stdout:25: write initial oid 25 2024-02-05T04:53:26.157 INFO:tasks.rados.rados.0.smithi196.stdout:25: seq_num 24 ranges {0=409600} 2024-02-05T04:53:26.157 INFO:tasks.rados.rados.0.smithi196.stdout:25: writing smithi19639362-25 from 0 to 409600 tid 1 2024-02-05T04:53:26.157 INFO:tasks.rados.rados.0.smithi196.stdout:26: write initial oid 26 2024-02-05T04:53:26.157 INFO:tasks.rados.rados.0.smithi196.stdout:26: seq_num 25 ranges {0=425984} 2024-02-05T04:53:26.157 INFO:tasks.rados.rados.0.smithi196.stdout:26: writing smithi19639362-26 from 0 to 425984 tid 1 2024-02-05T04:53:26.158 INFO:tasks.rados.rados.0.smithi196.stdout:27: write initial oid 27 2024-02-05T04:53:26.158 INFO:tasks.rados.rados.0.smithi196.stdout:27: seq_num 26 ranges {0=442368} 2024-02-05T04:53:26.159 INFO:tasks.rados.rados.0.smithi196.stdout:27: writing smithi19639362-27 from 0 to 442368 tid 1 2024-02-05T04:53:26.159 INFO:tasks.rados.rados.0.smithi196.stdout:28: write initial oid 28 2024-02-05T04:53:26.159 INFO:tasks.rados.rados.0.smithi196.stdout:28: seq_num 27 ranges {0=458752} 2024-02-05T04:53:26.162 INFO:tasks.rados.rados.0.smithi196.stdout:28: writing smithi19639362-28 from 0 to 458752 tid 1 2024-02-05T04:53:26.162 INFO:tasks.rados.rados.0.smithi196.stdout:29: write initial oid 29 2024-02-05T04:53:26.162 INFO:tasks.rados.rados.0.smithi196.stdout:29: seq_num 28 ranges {0=475136} 2024-02-05T04:53:26.165 INFO:tasks.rados.rados.0.smithi196.stdout:29: writing smithi19639362-29 from 0 to 475136 tid 1 2024-02-05T04:53:26.165 INFO:tasks.rados.rados.0.smithi196.stdout:30: write initial oid 30 2024-02-05T04:53:26.165 INFO:tasks.rados.rados.0.smithi196.stdout:30: seq_num 29 ranges {0=491520} 2024-02-05T04:53:26.168 INFO:tasks.rados.rados.0.smithi196.stdout:30: writing smithi19639362-30 from 0 to 491520 tid 1 2024-02-05T04:53:26.168 INFO:tasks.rados.rados.0.smithi196.stdout: waiting on 16 2024-02-05T04:53:26.168 INFO:tasks.rados.rados.0.smithi196.stdout:8: finishing write tid 3 to smithi19639362-8 2024-02-05T04:53:26.168 INFO:tasks.rados.rados.0.smithi196.stdout:8: oid 8 updating version 0 to 1 2024-02-05T04:53:26.168 INFO:tasks.rados.rados.0.smithi196.stdout:8: oid 8 updating version 1 to 2 2024-02-05T04:53:26.169 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 8 v 2 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-02-05T04:53:26.169 INFO:tasks.rados.rados.0.smithi196.stdout:8: left oid 8 (ObjNum 7 snap 0 seq_num 7) 2024-02-05T04:53:26.169 INFO:tasks.rados.rados.0.smithi196.stdout:14: finishing write tid 1 to smithi19639362-14 2024-02-05T04:53:26.169 INFO:tasks.rados.rados.0.smithi196.stdout:18: finishing write tid 1 to smithi19639362-18 2024-02-05T04:53:26.169 INFO:tasks.rados.rados.0.smithi196.stdout:8: done (15 left) 2024-02-05T04:53:26.169 INFO:tasks.rados.rados.0.smithi196.stdout:31: write initial oid 31 2024-02-05T04:53:26.169 INFO:tasks.rados.rados.0.smithi196.stdout:31: seq_num 30 ranges {0=507904} 2024-02-05T04:53:26.170 INFO:tasks.ceph.osd.2.smithi150.stderr:2024-02-05T04:53:26.169+0000 7f7337bf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 34014) UID: 0 2024-02-05T04:53:26.173 INFO:tasks.rados.rados.0.smithi196.stdout:31: writing smithi19639362-31 from 0 to 507904 tid 1 2024-02-05T04:53:26.174 INFO:tasks.rados.rados.0.smithi196.stdout: waiting on 16 2024-02-05T04:53:26.174 INFO:tasks.rados.rados.0.smithi196.stdout:13: finishing write tid 2 to smithi19639362-13 2024-02-05T04:53:26.174 INFO:tasks.rados.rados.0.smithi196.stdout:19: finishing write tid 1 to smithi19639362-19 2024-02-05T04:53:26.174 INFO:tasks.rados.rados.0.smithi196.stdout:14: finishing write tid 2 to smithi19639362-14 2024-02-05T04:53:26.174 INFO:tasks.rados.rados.0.smithi196.stdout:19: finishing write tid 2 to smithi19639362-19 2024-02-05T04:53:26.174 INFO:tasks.rados.rados.0.smithi196.stdout:18: finishing write tid 2 to smithi19639362-18 2024-02-05T04:53:26.174 INFO:tasks.rados.rados.0.smithi196.stdout:20: finishing write tid 1 to smithi19639362-20 2024-02-05T04:53:26.174 INFO:tasks.rados.rados.0.smithi196.stdout:20: finishing write tid 2 to smithi19639362-20 2024-02-05T04:53:26.175 INFO:tasks.rados.rados.0.smithi196.stdout:22: finishing write tid 1 to smithi19639362-22 2024-02-05T04:53:26.175 INFO:tasks.rados.rados.0.smithi196.stdout:21: finishing write tid 1 to smithi19639362-21 2024-02-05T04:53:26.175 INFO:tasks.rados.rados.0.smithi196.stdout:21: finishing write tid 2 to smithi19639362-21 2024-02-05T04:53:26.175 INFO:tasks.rados.rados.0.smithi196.stdout:13: finishing write tid 3 to smithi19639362-13 2024-02-05T04:53:26.175 INFO:tasks.rados.rados.0.smithi196.stdout:13: oid 13 updating version 0 to 4 2024-02-05T04:53:26.175 INFO:tasks.rados.rados.0.smithi196.stdout:13: oid 13 version 4 is already newer than 3 2024-02-05T04:53:26.175 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 13 v 4 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-02-05T04:53:26.175 INFO:tasks.rados.rados.0.smithi196.stdout:13: left oid 13 (ObjNum 12 snap 0 seq_num 12) 2024-02-05T04:53:26.175 INFO:tasks.rados.rados.0.smithi196.stdout:18: finishing write tid 3 to smithi19639362-18 2024-02-05T04:53:26.175 INFO:tasks.rados.rados.0.smithi196.stdout:18: oid 18 updating version 0 to 5 2024-02-05T04:53:26.175 INFO:tasks.rados.rados.0.smithi196.stdout:18: oid 18 updating version 5 to 6 2024-02-05T04:53:26.176 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 18 v 6 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-02-05T04:53:26.176 INFO:tasks.rados.rados.0.smithi196.stdout:18: left oid 18 (ObjNum 17 snap 0 seq_num 17) 2024-02-05T04:53:26.176 INFO:tasks.rados.rados.0.smithi196.stdout:20: finishing write tid 3 to smithi19639362-20 2024-02-05T04:53:26.176 INFO:tasks.rados.rados.0.smithi196.stdout:20: oid 20 updating version 0 to 4 2024-02-05T04:53:26.176 INFO:tasks.rados.rados.0.smithi196.stdout:20: oid 20 version 4 is already newer than 3 2024-02-05T04:53:26.176 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 20 v 4 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-02-05T04:53:26.176 INFO:tasks.rados.rados.0.smithi196.stdout:20: left oid 20 (ObjNum 19 snap 0 seq_num 19) 2024-02-05T04:53:26.176 INFO:tasks.rados.rados.0.smithi196.stdout:22: finishing write tid 2 to smithi19639362-22 2024-02-05T04:53:26.176 INFO:tasks.rados.rados.0.smithi196.stdout:19: finishing write tid 3 to smithi19639362-19 2024-02-05T04:53:26.176 INFO:tasks.rados.rados.0.smithi196.stdout:19: oid 19 updating version 0 to 2 2024-02-05T04:53:26.176 INFO:tasks.rados.rados.0.smithi196.stdout:19: oid 19 version 2 is already newer than 1 2024-02-05T04:53:26.176 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 19 v 2 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-02-05T04:53:26.177 INFO:tasks.rados.rados.0.smithi196.stdout:19: left oid 19 (ObjNum 18 snap 0 seq_num 18) 2024-02-05T04:53:26.177 INFO:tasks.rados.rados.0.smithi196.stdout:14: finishing write tid 3 to smithi19639362-14 2024-02-05T04:53:26.177 INFO:tasks.rados.rados.0.smithi196.stdout:14: oid 14 updating version 0 to 3 2024-02-05T04:53:26.177 INFO:tasks.rados.rados.0.smithi196.stdout:14: oid 14 updating version 3 to 4 2024-02-05T04:53:26.177 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 14 v 4 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-02-05T04:53:26.177 INFO:tasks.rados.rados.0.smithi196.stdout:14: left oid 14 (ObjNum 13 snap 0 seq_num 13) 2024-02-05T04:53:26.177 INFO:tasks.rados.rados.0.smithi196.stdout:23: finishing write tid 1 to smithi19639362-23 2024-02-05T04:53:26.177 INFO:tasks.rados.rados.0.smithi196.stdout:23: finishing write tid 2 to smithi19639362-23 2024-02-05T04:53:26.177 INFO:tasks.rados.rados.0.smithi196.stdout:22: finishing write tid 3 to smithi19639362-22 2024-02-05T04:53:26.177 INFO:tasks.rados.rados.0.smithi196.stdout:22: oid 22 updating version 0 to 5 2024-02-05T04:53:26.177 INFO:tasks.rados.rados.0.smithi196.stdout:22: oid 22 updating version 5 to 6 2024-02-05T04:53:26.177 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 22 v 6 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-02-05T04:53:26.178 INFO:tasks.rados.rados.0.smithi196.stdout:22: left oid 22 (ObjNum 21 snap 0 seq_num 21) 2024-02-05T04:53:26.178 INFO:tasks.rados.rados.0.smithi196.stdout:24: finishing write tid 1 to smithi19639362-24 2024-02-05T04:53:26.178 INFO:tasks.rados.rados.0.smithi196.stdout:21: finishing write tid 3 to smithi19639362-21 2024-02-05T04:53:26.178 INFO:tasks.rados.rados.0.smithi196.stdout:21: oid 21 updating version 0 to 6 2024-02-05T04:53:26.178 INFO:tasks.rados.rados.0.smithi196.stdout:21: oid 21 version 6 is already newer than 5 2024-02-05T04:53:26.178 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 21 v 6 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-02-05T04:53:26.178 INFO:tasks.rados.rados.0.smithi196.stdout:21: left oid 21 (ObjNum 20 snap 0 seq_num 20) 2024-02-05T04:53:26.178 INFO:tasks.rados.rados.0.smithi196.stdout:24: finishing write tid 2 to smithi19639362-24 2024-02-05T04:53:26.178 INFO:tasks.rados.rados.0.smithi196.stdout:23: finishing write tid 3 to smithi19639362-23 2024-02-05T04:53:26.178 INFO:tasks.rados.rados.0.smithi196.stdout:23: oid 23 updating version 0 to 5 2024-02-05T04:53:26.178 INFO:tasks.rados.rados.0.smithi196.stdout:23: oid 23 updating version 5 to 6 2024-02-05T04:53:26.178 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 23 v 6 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-02-05T04:53:26.178 INFO:tasks.rados.rados.0.smithi196.stdout:23: left oid 23 (ObjNum 22 snap 0 seq_num 22) 2024-02-05T04:53:26.179 INFO:tasks.rados.rados.0.smithi196.stdout:24: finishing write tid 3 to smithi19639362-24 2024-02-05T04:53:26.179 INFO:tasks.rados.rados.0.smithi196.stdout:24: oid 24 updating version 0 to 5 2024-02-05T04:53:26.179 INFO:tasks.rados.rados.0.smithi196.stdout:24: oid 24 updating version 5 to 6 2024-02-05T04:53:26.179 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 24 v 6 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-02-05T04:53:26.179 INFO:tasks.rados.rados.0.smithi196.stdout:24: left oid 24 (ObjNum 23 snap 0 seq_num 23) 2024-02-05T04:53:26.179 INFO:tasks.rados.rados.0.smithi196.stdout:26: finishing write tid 1 to smithi19639362-26 2024-02-05T04:53:26.179 INFO:tasks.rados.rados.0.smithi196.stdout:25: finishing write tid 1 to smithi19639362-25 2024-02-05T04:53:26.179 INFO:tasks.rados.rados.0.smithi196.stdout:25: finishing write tid 2 to smithi19639362-25 2024-02-05T04:53:26.179 INFO:tasks.rados.rados.0.smithi196.stdout:26: finishing write tid 2 to smithi19639362-26 2024-02-05T04:53:26.179 INFO:tasks.rados.rados.0.smithi196.stdout:27: finishing write tid 1 to smithi19639362-27 2024-02-05T04:53:26.179 INFO:tasks.rados.rados.0.smithi196.stdout:27: finishing write tid 2 to smithi19639362-27 2024-02-05T04:53:26.179 INFO:tasks.rados.rados.0.smithi196.stdout:25: finishing write tid 3 to smithi19639362-25 2024-02-05T04:53:26.179 INFO:tasks.rados.rados.0.smithi196.stdout:25: oid 25 updating version 0 to 2 2024-02-05T04:53:26.179 INFO:tasks.rados.rados.0.smithi196.stdout:25: oid 25 version 2 is already newer than 1 2024-02-05T04:53:26.180 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 25 v 2 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-02-05T04:53:26.180 INFO:tasks.rados.rados.0.smithi196.stdout:25: left oid 25 (ObjNum 24 snap 0 seq_num 24) 2024-02-05T04:53:26.180 INFO:tasks.rados.rados.0.smithi196.stdout:28: finishing write tid 1 to smithi19639362-28 2024-02-05T04:53:26.180 INFO:tasks.rados.rados.0.smithi196.stdout:26: finishing write tid 3 to smithi19639362-26 2024-02-05T04:53:26.180 INFO:tasks.rados.rados.0.smithi196.stdout:26: oid 26 updating version 0 to 3 2024-02-05T04:53:26.180 INFO:tasks.rados.rados.0.smithi196.stdout:26: oid 26 updating version 3 to 4 2024-02-05T04:53:26.180 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 26 v 4 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-02-05T04:53:26.180 INFO:tasks.rados.rados.0.smithi196.stdout:26: left oid 26 (ObjNum 25 snap 0 seq_num 25) 2024-02-05T04:53:26.180 INFO:tasks.rados.rados.0.smithi196.stdout:28: finishing write tid 2 to smithi19639362-28 2024-02-05T04:53:26.180 INFO:tasks.rados.rados.0.smithi196.stdout:29: finishing write tid 1 to smithi19639362-29 2024-02-05T04:53:26.180 INFO:tasks.rados.rados.0.smithi196.stdout:29: finishing write tid 2 to smithi19639362-29 2024-02-05T04:53:26.180 INFO:tasks.rados.rados.0.smithi196.stdout:13: done (15 left) 2024-02-05T04:53:26.180 INFO:tasks.rados.rados.0.smithi196.stdout:14: done (14 left) 2024-02-05T04:53:26.180 INFO:tasks.rados.rados.0.smithi196.stdout:18: done (13 left) 2024-02-05T04:53:26.181 INFO:tasks.rados.rados.0.smithi196.stdout:19: done (12 left) 2024-02-05T04:53:26.181 INFO:tasks.rados.rados.0.smithi196.stdout:20: done (11 left) 2024-02-05T04:53:26.181 INFO:tasks.rados.rados.0.smithi196.stdout:21: done (10 left) 2024-02-05T04:53:26.181 INFO:tasks.rados.rados.0.smithi196.stdout:22: done (9 left) 2024-02-05T04:53:26.181 INFO:tasks.rados.rados.0.smithi196.stdout:23: done (8 left) 2024-02-05T04:53:26.181 INFO:tasks.rados.rados.0.smithi196.stdout:24: done (7 left) 2024-02-05T04:53:26.181 INFO:tasks.rados.rados.0.smithi196.stdout:25: done (6 left) 2024-02-05T04:53:26.181 INFO:tasks.rados.rados.0.smithi196.stdout:26: done (5 left) 2024-02-05T04:53:26.181 INFO:tasks.rados.rados.0.smithi196.stdout:32: write initial oid 32 2024-02-05T04:53:26.181 INFO:tasks.rados.rados.0.smithi196.stdout:32: seq_num 31 ranges {0=524288} 2024-02-05T04:53:26.181 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:26.181 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:26.184 INFO:tasks.rados.rados.0.smithi196.stdout:32: writing smithi19639362-32 from 0 to 524288 tid 1 2024-02-05T04:53:26.184 INFO:tasks.rados.rados.0.smithi196.stdout:33: write initial oid 33 2024-02-05T04:53:26.184 INFO:tasks.rados.rados.0.smithi196.stdout:33: seq_num 32 ranges {0=540672} 2024-02-05T04:53:26.184 INFO:tasks.rados.rados.0.smithi196.stdout:33: writing smithi19639362-33 from 0 to 540672 tid 1 2024-02-05T04:53:26.184 INFO:tasks.rados.rados.0.smithi196.stdout:34: write initial oid 34 2024-02-05T04:53:26.184 INFO:tasks.rados.rados.0.smithi196.stdout:34: seq_num 33 ranges {0=557056} 2024-02-05T04:53:26.184 INFO:tasks.rados.rados.0.smithi196.stdout:34: writing smithi19639362-34 from 0 to 557056 tid 1 2024-02-05T04:53:26.184 INFO:tasks.rados.rados.0.smithi196.stdout:35: write initial oid 35 2024-02-05T04:53:26.184 INFO:tasks.rados.rados.0.smithi196.stdout:27: finishing write tid 3 to smithi19639362-27 2024-02-05T04:53:26.185 INFO:tasks.rados.rados.0.smithi196.stdout:27: oid 27 updating version 0 to 1 2024-02-05T04:53:26.185 INFO:tasks.rados.rados.0.smithi196.stdout:27: oid 27 updating version 1 to 2 2024-02-05T04:53:26.185 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 27 v 2 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-02-05T04:53:26.185 INFO:tasks.rados.rados.0.smithi196.stdout:27: left oid 27 (ObjNum 26 snap 0 seq_num 26) 2024-02-05T04:53:26.185 INFO:tasks.rados.rados.0.smithi196.stdout:28: finishing write tid 3 to smithi19639362-28 2024-02-05T04:53:26.185 INFO:tasks.rados.rados.0.smithi196.stdout:28: oid 28 updating version 0 to 3 2024-02-05T04:53:26.185 INFO:tasks.rados.rados.0.smithi196.stdout:28: oid 28 updating version 3 to 4 2024-02-05T04:53:26.185 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 28 v 4 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-02-05T04:53:26.185 INFO:tasks.rados.rados.0.smithi196.stdout:28: left oid 28 (ObjNum 27 snap 0 seq_num 27) 2024-02-05T04:53:26.185 INFO:tasks.rados.rados.0.smithi196.stdout:29: finishing write tid 3 to smithi19639362-29 2024-02-05T04:53:26.185 INFO:tasks.rados.rados.0.smithi196.stdout:29: oid 29 updating version 0 to 3 2024-02-05T04:53:26.185 INFO:tasks.rados.rados.0.smithi196.stdout:29: oid 29 updating version 3 to 4 2024-02-05T04:53:26.185 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 29 v 4 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-02-05T04:53:26.185 INFO:tasks.rados.rados.0.smithi196.stdout:29: left oid 29 (ObjNum 28 snap 0 seq_num 28) 2024-02-05T04:53:26.186 INFO:tasks.rados.rados.0.smithi196.stdout:30: finishing write tid 1 to smithi19639362-30 2024-02-05T04:53:26.186 INFO:tasks.rados.rados.0.smithi196.stdout:30: finishing write tid 2 to smithi19639362-30 2024-02-05T04:53:26.186 INFO:tasks.rados.rados.0.smithi196.stdout:31: finishing write tid 1 to smithi19639362-31 2024-02-05T04:53:26.186 INFO:tasks.rados.rados.0.smithi196.stdout:30: finishing write tid 3 to smithi19639362-30 2024-02-05T04:53:26.186 INFO:tasks.rados.rados.0.smithi196.stdout:30: oid 30 updating version 0 to 4 2024-02-05T04:53:26.186 INFO:tasks.rados.rados.0.smithi196.stdout:30: oid 30 version 4 is already newer than 3 2024-02-05T04:53:26.186 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 30 v 4 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-02-05T04:53:26.186 INFO:tasks.rados.rados.0.smithi196.stdout:30: left oid 30 (ObjNum 29 snap 0 seq_num 29) 2024-02-05T04:53:26.186 INFO:tasks.rados.rados.0.smithi196.stdout:31: finishing write tid 2 to smithi19639362-31 2024-02-05T04:53:26.186 INFO:tasks.rados.rados.0.smithi196.stdout:32: finishing write tid 1 to smithi19639362-32 2024-02-05T04:53:26.186 INFO:tasks.rados.rados.0.smithi196.stdout:32: finishing write tid 2 to smithi19639362-32 2024-02-05T04:53:26.186 INFO:tasks.rados.rados.0.smithi196.stdout:35: seq_num 34 ranges {0=573440} 2024-02-05T04:53:26.187 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-02-05T04:53:26.189 INFO:tasks.rados.rados.0.smithi196.stdout:35: writing smithi19639362-35 from 0 to 573440 tid 1 2024-02-05T04:53:26.189 INFO:tasks.rados.rados.0.smithi196.stdout:27: done (8 left) 2024-02-05T04:53:26.189 INFO:tasks.rados.rados.0.smithi196.stdout:28: done (7 left) 2024-02-05T04:53:26.189 INFO:tasks.rados.rados.0.smithi196.stdout:29: done (6 left) 2024-02-05T04:53:26.189 INFO:tasks.rados.rados.0.smithi196.stdout:30: done (5 left) 2024-02-05T04:53:26.189 INFO:tasks.rados.rados.0.smithi196.stdout:36: write initial oid 36 2024-02-05T04:53:26.189 INFO:tasks.rados.rados.0.smithi196.stdout:36: seq_num 35 ranges {0=589824} 2024-02-05T04:53:26.189 INFO:tasks.rados.rados.0.smithi196.stdout:36: writing smithi19639362-36 from 0 to 589824 tid 1 2024-02-05T04:53:26.189 INFO:tasks.rados.rados.0.smithi196.stdout:37: write initial oid 37 2024-02-05T04:53:26.189 INFO:tasks.rados.rados.0.smithi196.stdout:37: seq_num 36 ranges {0=606208} 2024-02-05T04:53:26.193 INFO:tasks.rados.rados.0.smithi196.stdout:37: writing smithi19639362-37 from 0 to 606208 tid 1 2024-02-05T04:53:26.193 INFO:tasks.rados.rados.0.smithi196.stdout:38: write initial oid 38 2024-02-05T04:53:26.193 INFO:tasks.rados.rados.0.smithi196.stdout:38: seq_num 37 ranges {0=622592} 2024-02-05T04:53:26.197 INFO:tasks.rados.rados.0.smithi196.stdout:38: writing smithi19639362-38 from 0 to 622592 tid 1 2024-02-05T04:53:26.197 INFO:tasks.rados.rados.0.smithi196.stdout:39: write initial oid 39 2024-02-05T04:53:26.197 INFO:tasks.rados.rados.0.smithi196.stdout:39: seq_num 38 ranges {0=638976} 2024-02-05T04:53:26.201 INFO:tasks.rados.rados.0.smithi196.stdout:39: writing smithi19639362-39 from 0 to 638976 tid 1 2024-02-05T04:53:26.201 INFO:tasks.rados.rados.0.smithi196.stdout:40: write initial oid 40 2024-02-05T04:53:26.201 INFO:tasks.rados.rados.0.smithi196.stdout:40: seq_num 39 ranges {0=671744} 2024-02-05T04:53:26.205 INFO:tasks.rados.rados.0.smithi196.stdout:40: writing smithi19639362-40 from 0 to 671744 tid 1 2024-02-05T04:53:26.205 INFO:tasks.rados.rados.0.smithi196.stdout:41: write initial oid 41 2024-02-05T04:53:26.205 INFO:tasks.rados.rados.0.smithi196.stdout:41: seq_num 40 ranges {0=688128} 2024-02-05T04:53:26.210 INFO:tasks.rados.rados.0.smithi196.stdout:41: writing smithi19639362-41 from 0 to 688128 tid 1 2024-02-05T04:53:26.210 INFO:tasks.rados.rados.0.smithi196.stdout:42: write initial oid 42 2024-02-05T04:53:26.210 INFO:tasks.rados.rados.0.smithi196.stdout:42: seq_num 41 ranges {0=704512} 2024-02-05T04:53:26.214 INFO:tasks.rados.rados.0.smithi196.stdout:42: writing smithi19639362-42 from 0 to 704512 tid 1 2024-02-05T04:53:26.214 INFO:tasks.rados.rados.0.smithi196.stdout:43: write initial oid 43 2024-02-05T04:53:26.214 INFO:tasks.rados.rados.0.smithi196.stdout:43: seq_num 42 ranges {0=720896} 2024-02-05T04:53:26.219 INFO:tasks.rados.rados.0.smithi196.stdout:43: writing smithi19639362-43 from 0 to 720896 tid 1 2024-02-05T04:53:26.219 INFO:tasks.rados.rados.0.smithi196.stdout:44: write initial oid 44 2024-02-05T04:53:26.219 INFO:tasks.rados.rados.0.smithi196.stdout:44: seq_num 43 ranges {0=737280} 2024-02-05T04:53:26.223 INFO:tasks.rados.rados.0.smithi196.stdout:44: writing smithi19639362-44 from 0 to 737280 tid 1 2024-02-05T04:53:26.223 INFO:tasks.rados.rados.0.smithi196.stdout:45: write initial oid 45 2024-02-05T04:53:26.223 INFO:tasks.rados.rados.0.smithi196.stdout:45: seq_num 44 ranges {0=753664} 2024-02-05T04:53:26.228 INFO:tasks.rados.rados.0.smithi196.stdout:45: writing smithi19639362-45 from 0 to 753664 tid 1 2024-02-05T04:53:26.228 INFO:tasks.rados.rados.0.smithi196.stdout:46: write initial oid 46 2024-02-05T04:53:26.228 INFO:tasks.rados.rados.0.smithi196.stdout:46: seq_num 45 ranges {0=770048} 2024-02-05T04:53:26.233 INFO:tasks.rados.rados.0.smithi196.stdout:46: writing smithi19639362-46 from 0 to 770048 tid 1 2024-02-05T04:53:26.233 INFO:tasks.rados.rados.0.smithi196.stdout: waiting on 16 2024-02-05T04:53:26.233 INFO:tasks.rados.rados.0.smithi196.stdout:31: finishing write tid 3 to smithi19639362-31 2024-02-05T04:53:26.233 INFO:tasks.rados.rados.0.smithi196.stdout:31: oid 31 updating version 0 to 8 2024-02-05T04:53:26.233 INFO:tasks.rados.rados.0.smithi196.stdout:31: oid 31 version 8 is already newer than 7 2024-02-05T04:53:26.233 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 31 v 8 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-02-05T04:53:26.233 INFO:tasks.rados.rados.0.smithi196.stdout:31: left oid 31 (ObjNum 30 snap 0 seq_num 30) 2024-02-05T04:53:26.234 INFO:tasks.rados.rados.0.smithi196.stdout:31: done (15 left) 2024-02-05T04:53:26.234 INFO:tasks.rados.rados.0.smithi196.stdout:47: write initial oid 47 2024-02-05T04:53:26.234 INFO:tasks.rados.rados.0.smithi196.stdout:47: seq_num 46 ranges {0=786432} 2024-02-05T04:53:26.240 INFO:tasks.rados.rados.0.smithi196.stdout:47: writing smithi19639362-47 from 0 to 786432 tid 1 2024-02-05T04:53:26.240 INFO:tasks.rados.rados.0.smithi196.stdout: waiting on 16 2024-02-05T04:53:26.240 INFO:tasks.rados.rados.0.smithi196.stdout:32: finishing write tid 3 to smithi19639362-32 2024-02-05T04:53:26.240 INFO:tasks.rados.rados.0.smithi196.stdout:32: oid 32 updating version 0 to 4 2024-02-05T04:53:26.241 INFO:tasks.rados.rados.0.smithi196.stdout:32: oid 32 version 4 is already newer than 3 2024-02-05T04:53:26.241 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 32 v 4 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-02-05T04:53:26.241 INFO:tasks.rados.rados.0.smithi196.stdout:32: left oid 32 (ObjNum 31 snap 0 seq_num 31) 2024-02-05T04:53:26.241 INFO:tasks.rados.rados.0.smithi196.stdout:33: finishing write tid 1 to smithi19639362-33 2024-02-05T04:53:26.241 INFO:tasks.rados.rados.0.smithi196.stdout:33: finishing write tid 2 to smithi19639362-33 2024-02-05T04:53:26.241 INFO:tasks.rados.rados.0.smithi196.stdout:33: finishing write tid 3 to smithi19639362-33 2024-02-05T04:53:26.241 INFO:tasks.rados.rados.0.smithi196.stdout:33: oid 33 updating version 0 to 8 2024-02-05T04:53:26.241 INFO:tasks.rados.rados.0.smithi196.stdout:33: oid 33 version 8 is already newer than 7 2024-02-05T04:53:26.241 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 33 v 8 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-02-05T04:53:26.241 INFO:tasks.rados.rados.0.smithi196.stdout:33: left oid 33 (ObjNum 32 snap 0 seq_num 32) 2024-02-05T04:53:26.241 INFO:tasks.rados.rados.0.smithi196.stdout:34: finishing write tid 1 to smithi19639362-34 2024-02-05T04:53:26.241 INFO:tasks.rados.rados.0.smithi196.stdout:34: finishing write tid 2 to smithi19639362-34 2024-02-05T04:53:26.241 INFO:tasks.rados.rados.0.smithi196.stdout:35: finishing write tid 1 to smithi19639362-35 2024-02-05T04:53:26.241 INFO:tasks.rados.rados.0.smithi196.stdout:34: finishing write tid 3 to smithi19639362-34 2024-02-05T04:53:26.242 INFO:tasks.rados.rados.0.smithi196.stdout:34: oid 34 updating version 0 to 8 2024-02-05T04:53:26.242 INFO:tasks.rados.rados.0.smithi196.stdout:34: oid 34 version 8 is already newer than 7 2024-02-05T04:53:26.242 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 34 v 8 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-02-05T04:53:26.242 INFO:tasks.rados.rados.0.smithi196.stdout:34: left oid 34 (ObjNum 33 snap 0 seq_num 33) 2024-02-05T04:53:26.242 INFO:tasks.rados.rados.0.smithi196.stdout:35: finishing write tid 2 to smithi19639362-35 2024-02-05T04:53:26.242 INFO:tasks.rados.rados.0.smithi196.stdout:35: finishing write tid 3 to smithi19639362-35 2024-02-05T04:53:26.242 INFO:tasks.rados.rados.0.smithi196.stdout:35: oid 35 updating version 0 to 8 2024-02-05T04:53:26.242 INFO:tasks.rados.rados.0.smithi196.stdout:35: oid 35 version 8 is already newer than 7 2024-02-05T04:53:26.242 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 35 v 8 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-02-05T04:53:26.242 INFO:tasks.rados.rados.0.smithi196.stdout:35: left oid 35 (ObjNum 34 snap 0 seq_num 34) 2024-02-05T04:53:26.242 INFO:tasks.rados.rados.0.smithi196.stdout:36: finishing write tid 1 to smithi19639362-36 2024-02-05T04:53:26.242 INFO:tasks.rados.rados.0.smithi196.stdout:36: finishing write tid 2 to smithi19639362-36 2024-02-05T04:53:26.242 INFO:tasks.rados.rados.0.smithi196.stdout:36: finishing write tid 3 to smithi19639362-36 2024-02-05T04:53:26.242 INFO:tasks.rados.rados.0.smithi196.stdout:36: oid 36 updating version 0 to 9 2024-02-05T04:53:26.243 INFO:tasks.rados.rados.0.smithi196.stdout:36: oid 36 updating version 9 to 10 2024-02-05T04:53:26.243 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 36 v 10 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-02-05T04:53:26.243 INFO:tasks.rados.rados.0.smithi196.stdout:36: left oid 36 (ObjNum 35 snap 0 seq_num 35) 2024-02-05T04:53:26.243 INFO:tasks.rados.rados.0.smithi196.stdout:37: finishing write tid 1 to smithi19639362-37 2024-02-05T04:53:26.243 INFO:tasks.rados.rados.0.smithi196.stdout:37: finishing write tid 2 to smithi19639362-37 2024-02-05T04:53:26.243 INFO:tasks.rados.rados.0.smithi196.stdout:39: finishing write tid 1 to smithi19639362-39 2024-02-05T04:53:26.243 INFO:tasks.rados.rados.0.smithi196.stdout:38: finishing write tid 1 to smithi19639362-38 2024-02-05T04:53:26.243 INFO:tasks.rados.rados.0.smithi196.stdout:38: finishing write tid 2 to smithi19639362-38 2024-02-05T04:53:26.243 INFO:tasks.rados.rados.0.smithi196.stdout:39: finishing write tid 2 to smithi19639362-39 2024-02-05T04:53:26.243 INFO:tasks.rados.rados.0.smithi196.stdout:37: finishing write tid 3 to smithi19639362-37 2024-02-05T04:53:26.243 INFO:tasks.rados.rados.0.smithi196.stdout:37: oid 37 updating version 0 to 2 2024-02-05T04:53:26.243 INFO:tasks.rados.rados.0.smithi196.stdout:37: oid 37 version 2 is already newer than 1 2024-02-05T04:53:26.243 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 37 v 2 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-02-05T04:53:26.243 INFO:tasks.rados.rados.0.smithi196.stdout:37: left oid 37 (ObjNum 36 snap 0 seq_num 36) 2024-02-05T04:53:26.244 INFO:tasks.rados.rados.0.smithi196.stdout:38: finishing write tid 3 to smithi19639362-38 2024-02-05T04:53:26.244 INFO:tasks.rados.rados.0.smithi196.stdout:38: oid 38 updating version 0 to 3 2024-02-05T04:53:26.244 INFO:tasks.rados.rados.0.smithi196.stdout:38: oid 38 updating version 3 to 4 2024-02-05T04:53:26.244 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 38 v 4 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-02-05T04:53:26.244 INFO:tasks.rados.rados.0.smithi196.stdout:38: left oid 38 (ObjNum 37 snap 0 seq_num 37) 2024-02-05T04:53:26.244 INFO:tasks.rados.rados.0.smithi196.stdout:39: finishing write tid 3 to smithi19639362-39 2024-02-05T04:53:26.244 INFO:tasks.rados.rados.0.smithi196.stdout:39: oid 39 updating version 0 to 6 2024-02-05T04:53:26.244 INFO:tasks.rados.rados.0.smithi196.stdout:39: oid 39 version 6 is already newer than 5 2024-02-05T04:53:26.244 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 39 v 6 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-02-05T04:53:26.244 INFO:tasks.rados.rados.0.smithi196.stdout:39: left oid 39 (ObjNum 38 snap 0 seq_num 38) 2024-02-05T04:53:26.244 INFO:tasks.rados.rados.0.smithi196.stdout:40: finishing write tid 1 to smithi19639362-40 2024-02-05T04:53:26.244 INFO:tasks.rados.rados.0.smithi196.stdout:40: finishing write tid 2 to smithi19639362-40 2024-02-05T04:53:26.245 INFO:tasks.rados.rados.0.smithi196.stdout:41: finishing write tid 1 to smithi19639362-41 2024-02-05T04:53:26.245 INFO:tasks.rados.rados.0.smithi196.stdout:40: finishing write tid 3 to smithi19639362-40 2024-02-05T04:53:26.245 INFO:tasks.rados.rados.0.smithi196.stdout:40: oid 40 updating version 0 to 3 2024-02-05T04:53:26.245 INFO:tasks.rados.rados.0.smithi196.stdout:40: oid 40 updating version 3 to 4 2024-02-05T04:53:26.245 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 40 v 4 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-02-05T04:53:26.245 INFO:tasks.rados.rados.0.smithi196.stdout:40: left oid 40 (ObjNum 39 snap 0 seq_num 39) 2024-02-05T04:53:26.245 INFO:tasks.rados.rados.0.smithi196.stdout:41: finishing write tid 2 to smithi19639362-41 2024-02-05T04:53:26.245 INFO:tasks.rados.rados.0.smithi196.stdout:41: finishing write tid 3 to smithi19639362-41 2024-02-05T04:53:26.245 INFO:tasks.rados.rados.0.smithi196.stdout:41: oid 41 updating version 0 to 12 2024-02-05T04:53:26.245 INFO:tasks.rados.rados.0.smithi196.stdout:41: oid 41 version 12 is already newer than 11 2024-02-05T04:53:26.245 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 41 v 12 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-02-05T04:53:26.245 INFO:tasks.rados.rados.0.smithi196.stdout:41: left oid 41 (ObjNum 40 snap 0 seq_num 40) 2024-02-05T04:53:26.245 INFO:tasks.rados.rados.0.smithi196.stdout:42: finishing write tid 1 to smithi19639362-42 2024-02-05T04:53:26.245 INFO:tasks.rados.rados.0.smithi196.stdout:42: finishing write tid 2 to smithi19639362-42 2024-02-05T04:53:26.245 INFO:tasks.rados.rados.0.smithi196.stdout:43: finishing write tid 1 to smithi19639362-43 2024-02-05T04:53:26.246 INFO:tasks.rados.rados.0.smithi196.stdout:43: finishing write tid 2 to smithi19639362-43 2024-02-05T04:53:26.246 INFO:tasks.rados.rados.0.smithi196.stdout:43: finishing write tid 3 to smithi19639362-43 2024-02-05T04:53:26.246 INFO:tasks.rados.rados.0.smithi196.stdout:43: oid 43 updating version 0 to 10 2024-02-05T04:53:26.246 INFO:tasks.rados.rados.0.smithi196.stdout:43: oid 43 version 10 is already newer than 9 2024-02-05T04:53:26.246 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 43 v 10 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-02-05T04:53:26.246 INFO:tasks.rados.rados.0.smithi196.stdout:43: left oid 43 (ObjNum 42 snap 0 seq_num 42) 2024-02-05T04:53:26.246 INFO:tasks.rados.rados.0.smithi196.stdout:42: finishing write tid 3 to smithi19639362-42 2024-02-05T04:53:26.246 INFO:tasks.rados.rados.0.smithi196.stdout:42: oid 42 updating version 0 to 5 2024-02-05T04:53:26.246 INFO:tasks.rados.rados.0.smithi196.stdout:42: oid 42 updating version 5 to 6 2024-02-05T04:53:26.246 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 42 v 6 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-02-05T04:53:26.246 INFO:tasks.rados.rados.0.smithi196.stdout:42: left oid 42 (ObjNum 41 snap 0 seq_num 41) 2024-02-05T04:53:26.246 INFO:tasks.rados.rados.0.smithi196.stdout:44: finishing write tid 1 to smithi19639362-44 2024-02-05T04:53:26.246 INFO:tasks.rados.rados.0.smithi196.stdout:44: finishing write tid 2 to smithi19639362-44 2024-02-05T04:53:26.246 INFO:tasks.rados.rados.0.smithi196.stdout:44: finishing write tid 3 to smithi19639362-44 2024-02-05T04:53:26.246 INFO:tasks.rados.rados.0.smithi196.stdout:44: oid 44 updating version 0 to 6 2024-02-05T04:53:26.246 INFO:tasks.rados.rados.0.smithi196.stdout:44: oid 44 version 6 is already newer than 5 2024-02-05T04:53:26.247 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 44 v 6 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-02-05T04:53:26.247 INFO:tasks.rados.rados.0.smithi196.stdout:44: left oid 44 (ObjNum 43 snap 0 seq_num 43) 2024-02-05T04:53:26.247 INFO:tasks.rados.rados.0.smithi196.stdout:45: finishing write tid 1 to smithi19639362-45 2024-02-05T04:53:26.247 INFO:tasks.rados.rados.0.smithi196.stdout:45: finishing write tid 2 to smithi19639362-45 2024-02-05T04:53:26.247 INFO:tasks.rados.rados.0.smithi196.stdout:46: finishing write tid 1 to smithi19639362-46 2024-02-05T04:53:26.247 INFO:tasks.rados.rados.0.smithi196.stdout:46: finishing write tid 2 to smithi19639362-46 2024-02-05T04:53:26.247 INFO:tasks.rados.rados.0.smithi196.stdout:32: done (15 left) 2024-02-05T04:53:26.247 INFO:tasks.rados.rados.0.smithi196.stdout:33: done (14 left) 2024-02-05T04:53:26.247 INFO:tasks.rados.rados.0.smithi196.stdout:34: done (13 left) 2024-02-05T04:53:26.247 INFO:tasks.rados.rados.0.smithi196.stdout:35: done (12 left) 2024-02-05T04:53:26.247 INFO:tasks.rados.rados.0.smithi196.stdout:36: done (11 left) 2024-02-05T04:53:26.247 INFO:tasks.rados.rados.0.smithi196.stdout:37: done (10 left) 2024-02-05T04:53:26.247 INFO:tasks.rados.rados.0.smithi196.stdout:38: done (9 left) 2024-02-05T04:53:26.247 INFO:tasks.rados.rados.0.smithi196.stdout:39: done (8 left) 2024-02-05T04:53:26.247 INFO:tasks.rados.rados.0.smithi196.stdout:40: done (7 left) 2024-02-05T04:53:26.247 INFO:tasks.rados.rados.0.smithi196.stdout:41: done (6 left) 2024-02-05T04:53:26.247 INFO:tasks.rados.rados.0.smithi196.stdout:42: done (5 left) 2024-02-05T04:53:26.248 INFO:tasks.rados.rados.0.smithi196.stdout:43: done (4 left) 2024-02-05T04:53:26.248 INFO:tasks.rados.rados.0.smithi196.stdout:44: done (3 left) 2024-02-05T04:53:26.248 INFO:tasks.rados.rados.0.smithi196.stdout:48: write initial oid 48 2024-02-05T04:53:26.248 INFO:tasks.rados.rados.0.smithi196.stdout:48: seq_num 47 ranges {0=425984,425984=376832} 2024-02-05T04:53:26.248 INFO:tasks.rados.rados.0.smithi196.stdout:48: writing smithi19639362-48 from 0 to 425984 tid 1 2024-02-05T04:53:26.249 INFO:tasks.rados.rados.0.smithi196.stdout:48: writing smithi19639362-48 from 425984 to 802816 tid 2 2024-02-05T04:53:26.249 INFO:tasks.rados.rados.0.smithi196.stdout:49: write initial oid 49 2024-02-05T04:53:26.249 INFO:tasks.rados.rados.0.smithi196.stdout:49: seq_num 48 ranges {0=442368,442368=376832} 2024-02-05T04:53:26.249 INFO:tasks.rados.rados.0.smithi196.stdout:49: writing smithi19639362-49 from 0 to 442368 tid 1 2024-02-05T04:53:26.249 INFO:tasks.rados.rados.0.smithi196.stdout:49: writing smithi19639362-49 from 442368 to 819200 tid 2 2024-02-05T04:53:26.249 INFO:tasks.rados.rados.0.smithi196.stdout:50: write initial oid 50 2024-02-05T04:53:26.249 INFO:tasks.rados.rados.0.smithi196.stdout:50: seq_num 49 ranges {0=458752,458752=376832} 2024-02-05T04:53:26.250 INFO:tasks.rados.rados.0.smithi196.stdout:50: writing smithi19639362-50 from 0 to 458752 tid 1 2024-02-05T04:53:26.254 INFO:tasks.rados.rados.0.smithi196.stdout:50: writing smithi19639362-50 from 458752 to 835584 tid 2 2024-02-05T04:53:26.254 INFO:tasks.rados.rados.0.smithi196.stdout:51: setattr oid 29 current snap is 0 2024-02-05T04:53:26.254 INFO:tasks.rados.rados.0.smithi196.stdout:45: finishing write tid 3 to smithi19639362-45 2024-02-05T04:53:26.254 INFO:tasks.rados.rados.0.smithi196.stdout:45: oid 45 updating version 0 to 10 2024-02-05T04:53:26.254 INFO:tasks.rados.rados.0.smithi196.stdout:45: oid 45 version 10 is already newer than 9 2024-02-05T04:53:26.254 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 45 v 10 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-02-05T04:53:26.254 INFO:tasks.rados.rados.0.smithi196.stdout:45: left oid 45 (ObjNum 44 snap 0 seq_num 44) 2024-02-05T04:53:26.254 INFO:tasks.rados.rados.0.smithi196.stdout:46: finishing write tid 3 to smithi19639362-46 2024-02-05T04:53:26.254 INFO:tasks.rados.rados.0.smithi196.stdout:46: oid 46 updating version 0 to 5 2024-02-05T04:53:26.254 INFO:tasks.rados.rados.0.smithi196.stdout:46: oid 46 updating version 5 to 6 2024-02-05T04:53:26.254 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 46 v 6 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-02-05T04:53:26.254 INFO:tasks.rados.rados.0.smithi196.stdout:46: left oid 46 (ObjNum 45 snap 0 seq_num 45) 2024-02-05T04:53:26.254 INFO:tasks.rados.rados.0.smithi196.stdout:47: finishing write tid 1 to smithi19639362-47 2024-02-05T04:53:26.254 INFO:tasks.rados.rados.0.smithi196.stdout:47: finishing write tid 2 to smithi19639362-47 2024-02-05T04:53:26.254 INFO:tasks.rados.rados.0.smithi196.stdout:48: finishing write tid 1 to smithi19639362-48 2024-02-05T04:53:26.255 INFO:tasks.rados.rados.0.smithi196.stdout:47: finishing write tid 3 to smithi19639362-47 2024-02-05T04:53:26.255 INFO:tasks.rados.rados.0.smithi196.stdout:47: oid 47 updating version 0 to 11 2024-02-05T04:53:26.255 INFO:tasks.rados.rados.0.smithi196.stdout:47: oid 47 updating version 11 to 12 2024-02-05T04:53:26.255 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 47 v 12 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-02-05T04:53:26.255 INFO:tasks.rados.rados.0.smithi196.stdout:47: left oid 47 (ObjNum 46 snap 0 seq_num 46) 2024-02-05T04:53:26.255 INFO:tasks.rados.rados.0.smithi196.stdout:48: finishing write tid 2 to smithi19639362-48 2024-02-05T04:53:26.255 INFO:tasks.rados.rados.0.smithi196.stdout:48: finishing write tid 3 to smithi19639362-48 2024-02-05T04:53:26.255 INFO:tasks.rados.rados.0.smithi196.stdout:49: finishing write tid 1 to smithi19639362-49 2024-02-05T04:53:26.255 INFO:tasks.rados.rados.0.smithi196.stdout:48: finishing write tid 4 to smithi19639362-48 2024-02-05T04:53:26.255 INFO:tasks.rados.rados.0.smithi196.stdout:48: oid 48 updating version 0 to 7 2024-02-05T04:53:26.255 INFO:tasks.rados.rados.0.smithi196.stdout:48: oid 48 version 7 is already newer than 6 2024-02-05T04:53:26.255 INFO:tasks.rados.rados.0.smithi196.stdout:48: oid 48 version 7 is already newer than 5 2024-02-05T04:53:26.255 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 48 v 7 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-02-05T04:53:26.255 INFO:tasks.rados.rados.0.smithi196.stdout:48: left oid 48 (ObjNum 47 snap 0 seq_num 47) 2024-02-05T04:53:26.255 INFO:tasks.rados.rados.0.smithi196.stdout:49: finishing write tid 2 to smithi19639362-49 2024-02-05T04:53:26.255 INFO:tasks.rados.rados.0.smithi196.stdout:49: finishing write tid 3 to smithi19639362-49 2024-02-05T04:53:26.256 INFO:tasks.rados.rados.0.smithi196.stdout:49: finishing write tid 4 to smithi19639362-49 2024-02-05T04:53:26.256 INFO:tasks.rados.rados.0.smithi196.stdout:49: oid 49 updating version 0 to 14 2024-02-05T04:53:26.256 INFO:tasks.rados.rados.0.smithi196.stdout:49: oid 49 version 14 is already newer than 13 2024-02-05T04:53:26.256 INFO:tasks.rados.rados.0.smithi196.stdout:49: oid 49 updating version 14 to 15 2024-02-05T04:53:26.256 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 49 v 15 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-02-05T04:53:26.256 INFO:tasks.rados.rados.0.smithi196.stdout:49: left oid 49 (ObjNum 48 snap 0 seq_num 48) 2024-02-05T04:53:26.257 INFO:tasks.rados.rados.0.smithi196.stdout:50: finishing write tid 1 to smithi19639362-50 2024-02-05T04:53:26.257 INFO:tasks.rados.rados.0.smithi196.stdout:45: done (6 left) 2024-02-05T04:53:26.257 INFO:tasks.rados.rados.0.smithi196.stdout:46: done (5 left) 2024-02-05T04:53:26.257 INFO:tasks.rados.rados.0.smithi196.stdout:47: done (4 left) 2024-02-05T04:53:26.257 INFO:tasks.rados.rados.0.smithi196.stdout:48: done (3 left) 2024-02-05T04:53:26.257 INFO:tasks.rados.rados.0.smithi196.stdout:49: done (2 left) 2024-02-05T04:53:26.257 INFO:tasks.rados.rados.0.smithi196.stdout:append oid 21 current snap is 0 2024-02-05T04:53:26.257 INFO:tasks.rados.rados.0.smithi196.stdout:52: seq_num 50 ranges {344064=475136,819200=376832} 2024-02-05T04:53:26.260 INFO:tasks.rados.rados.0.smithi196.stdout:52: writing smithi19639362-21 from 344064 to 819200 tid 1 2024-02-05T04:53:26.263 INFO:tasks.rados.rados.0.smithi196.stdout:52: writing smithi19639362-21 from 819200 to 1196032 tid 2 2024-02-05T04:53:26.263 INFO:tasks.rados.rados.0.smithi196.stdout:53: read oid 13 snap -1 2024-02-05T04:53:26.263 INFO:tasks.rados.rados.0.smithi196.stdout:53: expect (ObjNum 12 snap 0 seq_num 12) 2024-02-05T04:53:26.264 INFO:tasks.rados.rados.0.smithi196.stdout:54: read oid 43 snap -1 2024-02-05T04:53:26.264 INFO:tasks.rados.rados.0.smithi196.stdout:54: expect (ObjNum 42 snap 0 seq_num 42) 2024-02-05T04:53:26.264 INFO:tasks.rados.rados.0.smithi196.stdout:55: copy_from oid 31 from oid 7 current snap is 0 2024-02-05T04:53:26.264 INFO:tasks.rados.rados.0.smithi196.stdout:50: finishing write tid 2 to smithi19639362-50 2024-02-05T04:53:26.264 INFO:tasks.rados.rados.0.smithi196.stdout:50: finishing write tid 3 to smithi19639362-50 2024-02-05T04:53:26.264 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 29 v 7 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-02-05T04:53:26.264 INFO:tasks.rados.rados.0.smithi196.stdout:50: finishing write tid 4 to smithi19639362-50 2024-02-05T04:53:26.264 INFO:tasks.rados.rados.0.smithi196.stdout:50: oid 50 updating version 0 to 7 2024-02-05T04:53:26.264 INFO:tasks.rados.rados.0.smithi196.stdout:50: oid 50 updating version 7 to 8 2024-02-05T04:53:26.264 INFO:tasks.rados.rados.0.smithi196.stdout:50: oid 50 updating version 8 to 9 2024-02-05T04:53:26.264 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 50 v 9 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-02-05T04:53:26.264 INFO:tasks.rados.rados.0.smithi196.stdout:50: left oid 50 (ObjNum 49 snap 0 seq_num 49) 2024-02-05T04:53:26.264 INFO:tasks.rados.rados.0.smithi196.stdout:50: done (5 left) 2024-02-05T04:53:26.264 INFO:tasks.rados.rados.0.smithi196.stdout:51: done (4 left) 2024-02-05T04:53:26.265 INFO:tasks.rados.rados.0.smithi196.stdout:append oid 12 current snap is 0 2024-02-05T04:53:26.265 INFO:tasks.rados.rados.0.smithi196.stdout:56: seq_num 51 ranges {196608=491520,688128=376832} 2024-02-05T04:53:26.267 INFO:tasks.rados.rados.0.smithi196.stdout:56: writing smithi19639362-12 from 196608 to 688128 tid 1 2024-02-05T04:53:26.270 INFO:tasks.rados.rados.0.smithi196.stdout:56: writing smithi19639362-12 from 688128 to 1064960 tid 2 2024-02-05T04:53:26.270 INFO:tasks.rados.rados.0.smithi196.stdout:57: rmattr oid 17 current snap is 0 2024-02-05T04:53:26.271 INFO:tasks.rados.rados.0.smithi196.stdout:52: finishing write tid 1 to smithi19639362-21 2024-02-05T04:53:26.271 INFO:tasks.rados.rados.0.smithi196.stdout:58: delete oid 20 current snap is 0 2024-02-05T04:53:26.271 INFO:tasks.rados.rados.0.smithi196.stdout:55: finishing copy_from racing read to smithi19639362-31 2024-02-05T04:53:26.271 INFO:tasks.rados.rados.0.smithi196.stdout:53: expect (ObjNum 12 snap 0 seq_num 12) 2024-02-05T04:53:26.272 INFO:tasks.ceph.osd.12.smithi062.stderr:2024-02-05T04:53:26.271+0000 7f5e5847e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 34695) UID: 0 2024-02-05T04:53:26.276 INFO:tasks.rados.rados.0.smithi196.stdout:54: expect (ObjNum 42 snap 0 seq_num 42) 2024-02-05T04:53:26.296 INFO:tasks.rados.rados.0.smithi196.stdout:52: finishing write tid 2 to smithi19639362-21 2024-02-05T04:53:26.296 INFO:tasks.rados.rados.0.smithi196.stdout:52: finishing write tid 3 to smithi19639362-21 2024-02-05T04:53:26.296 INFO:tasks.rados.rados.0.smithi196.stdout:53: done (6 left) 2024-02-05T04:53:26.296 INFO:tasks.rados.rados.0.smithi196.stdout:54: done (5 left) 2024-02-05T04:53:26.297 INFO:tasks.rados.rados.0.smithi196.stdout:58: done (4 left) 2024-02-05T04:53:26.297 INFO:tasks.rados.rados.0.smithi196.stdout:59: read oid 38 snap -1 2024-02-05T04:53:26.297 INFO:tasks.rados.rados.0.smithi196.stdout:59: expect (ObjNum 37 snap 0 seq_num 37) 2024-02-05T04:53:26.297 INFO:tasks.rados.rados.0.smithi196.stdout:60: read oid 33 snap -1 2024-02-05T04:53:26.297 INFO:tasks.rados.rados.0.smithi196.stdout:60: expect (ObjNum 32 snap 0 seq_num 32) 2024-02-05T04:53:26.297 INFO:tasks.rados.rados.0.smithi196.stdout:append oid 10 current snap is 0 2024-02-05T04:53:26.297 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 17 v 0 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-02-05T04:53:26.297 INFO:tasks.rados.rados.0.smithi196.stdout:52: finishing write tid 4 to smithi19639362-21 2024-02-05T04:53:26.297 INFO:tasks.rados.rados.0.smithi196.stdout:52: oid 21 updating version 0 to 9 2024-02-05T04:53:26.297 INFO:tasks.rados.rados.0.smithi196.stdout:52: oid 21 version 9 is already newer than 7 2024-02-05T04:53:26.297 INFO:tasks.rados.rados.0.smithi196.stdout:52: oid 21 version 9 is already newer than 8 2024-02-05T04:53:26.297 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 21 v 9 (ObjNum 50 snap 0 seq_num 50) dirty exists 2024-02-05T04:53:26.297 INFO:tasks.rados.rados.0.smithi196.stdout:52: left oid 21 (ObjNum 50 snap 0 seq_num 50) 2024-02-05T04:53:26.297 INFO:tasks.rados.rados.0.smithi196.stdout:55: finishing copy_from to smithi19639362-31 2024-02-05T04:53:26.297 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 31 v 2 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-02-05T04:53:26.298 INFO:tasks.rados.rados.0.smithi196.stdout:56: finishing write tid 1 to smithi19639362-12 2024-02-05T04:53:26.298 INFO:tasks.rados.rados.0.smithi196.stdout:56: finishing write tid 2 to smithi19639362-12 2024-02-05T04:53:26.298 INFO:tasks.rados.rados.0.smithi196.stdout:56: finishing write tid 3 to smithi19639362-12 2024-02-05T04:53:26.298 INFO:tasks.rados.rados.0.smithi196.stdout:56: finishing write tid 4 to smithi19639362-12 2024-02-05T04:53:26.298 INFO:tasks.rados.rados.0.smithi196.stdout:56: oid 12 updating version 0 to 14 2024-02-05T04:53:26.298 INFO:tasks.rados.rados.0.smithi196.stdout:56: oid 12 updating version 14 to 15 2024-02-05T04:53:26.298 INFO:tasks.rados.rados.0.smithi196.stdout:56: oid 12 version 15 is already newer than 13 2024-02-05T04:53:26.298 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 12 v 15 (ObjNum 51 snap 0 seq_num 51) dirty exists 2024-02-05T04:53:26.298 INFO:tasks.rados.rados.0.smithi196.stdout:56: left oid 12 (ObjNum 51 snap 0 seq_num 51) 2024-02-05T04:53:26.298 INFO:tasks.rados.rados.0.smithi196.stdout:61: seq_num 53 ranges {163840=524288,688128=376832} 2024-02-05T04:53:26.299 INFO:tasks.rados.rados.0.smithi196.stdout:61: writing smithi19639362-10 from 163840 to 688128 tid 1 2024-02-05T04:53:26.302 INFO:tasks.rados.rados.0.smithi196.stdout:61: writing smithi19639362-10 from 688128 to 1064960 tid 2 2024-02-05T04:53:26.302 INFO:tasks.rados.rados.0.smithi196.stdout:52: done (6 left) 2024-02-05T04:53:26.302 INFO:tasks.rados.rados.0.smithi196.stdout:55: done (5 left) 2024-02-05T04:53:26.302 INFO:tasks.rados.rados.0.smithi196.stdout:56: done (4 left) 2024-02-05T04:53:26.302 INFO:tasks.rados.rados.0.smithi196.stdout:57: done (3 left) 2024-02-05T04:53:26.302 INFO:tasks.rados.rados.0.smithi196.stdout:62: copy_from oid 48 from oid 50 current snap is 0 2024-02-05T04:53:26.302 INFO:tasks.rados.rados.0.smithi196.stdout:append oid 40 current snap is 0 2024-02-05T04:53:26.302 INFO:tasks.rados.rados.0.smithi196.stdout:63: seq_num 54 ranges {671744=540672,1212416=376832} 2024-02-05T04:53:26.309 INFO:tasks.rados.rados.0.smithi196.stdout:63: writing smithi19639362-40 from 671744 to 1212416 tid 1 2024-02-05T04:53:26.311 INFO:tasks.rados.rados.0.smithi196.stdout:63: writing smithi19639362-40 from 1212416 to 1589248 tid 2 2024-02-05T04:53:26.311 INFO:tasks.rados.rados.0.smithi196.stdout:64: delete oid 14 current snap is 0 2024-02-05T04:53:26.311 INFO:tasks.rados.rados.0.smithi196.stdout:59: expect (ObjNum 37 snap 0 seq_num 37) 2024-02-05T04:53:26.321 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:26.321 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:26.331 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-02-05T04:53:26.331 INFO:tasks.rados.rados.0.smithi196.stdout:60: expect (ObjNum 32 snap 0 seq_num 32) 2024-02-05T04:53:26.345 INFO:tasks.rados.rados.0.smithi196.stdout:61: finishing write tid 1 to smithi19639362-10 2024-02-05T04:53:26.345 INFO:tasks.rados.rados.0.smithi196.stdout:61: finishing write tid 2 to smithi19639362-10 2024-02-05T04:53:26.346 INFO:tasks.rados.rados.0.smithi196.stdout:61: finishing write tid 3 to smithi19639362-10 2024-02-05T04:53:26.346 INFO:tasks.rados.rados.0.smithi196.stdout:61: finishing write tid 4 to smithi19639362-10 2024-02-05T04:53:26.346 INFO:tasks.rados.rados.0.smithi196.stdout:61: oid 10 updating version 0 to 10 2024-02-05T04:53:26.346 INFO:tasks.rados.rados.0.smithi196.stdout:61: oid 10 version 10 is already newer than 8 2024-02-05T04:53:26.346 INFO:tasks.rados.rados.0.smithi196.stdout:61: oid 10 version 10 is already newer than 9 2024-02-05T04:53:26.346 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 10 v 10 (ObjNum 53 snap 0 seq_num 53) dirty exists 2024-02-05T04:53:26.346 INFO:tasks.rados.rados.0.smithi196.stdout:61: left oid 10 (ObjNum 53 snap 0 seq_num 53) 2024-02-05T04:53:26.346 INFO:tasks.rados.rados.0.smithi196.stdout:59: done (5 left) 2024-02-05T04:53:26.346 INFO:tasks.rados.rados.0.smithi196.stdout:60: done (4 left) 2024-02-05T04:53:26.346 INFO:tasks.rados.rados.0.smithi196.stdout:61: done (3 left) 2024-02-05T04:53:26.346 INFO:tasks.rados.rados.0.smithi196.stdout:64: done (2 left) 2024-02-05T04:53:26.346 INFO:tasks.rados.rados.0.smithi196.stdout:65: read oid 10 snap -1 2024-02-05T04:53:26.346 INFO:tasks.rados.rados.0.smithi196.stdout:65: expect (ObjNum 53 snap 0 seq_num 53) 2024-02-05T04:53:26.346 INFO:tasks.rados.rados.0.smithi196.stdout:66: rmattr oid 1 current snap is 0 2024-02-05T04:53:26.347 INFO:tasks.rados.rados.0.smithi196.stdout:66: done (3 left) 2024-02-05T04:53:26.347 INFO:tasks.rados.rados.0.smithi196.stdout:67: read oid 11 snap -1 2024-02-05T04:53:26.347 INFO:tasks.rados.rados.0.smithi196.stdout:67: expect (ObjNum 10 snap 0 seq_num 10) 2024-02-05T04:53:26.347 INFO:tasks.rados.rados.0.smithi196.stdout:append oid 29 current snap is 0 2024-02-05T04:53:26.347 INFO:tasks.rados.rados.0.smithi196.stdout:62: finishing copy_from racing read to smithi19639362-48 2024-02-05T04:53:26.347 INFO:tasks.rados.rados.0.smithi196.stdout:63: finishing write tid 1 to smithi19639362-40 2024-02-05T04:53:26.347 INFO:tasks.rados.rados.0.smithi196.stdout:62: finishing copy_from to smithi19639362-48 2024-02-05T04:53:26.347 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 48 v 9 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-02-05T04:53:26.347 INFO:tasks.rados.rados.0.smithi196.stdout:63: finishing write tid 2 to smithi19639362-40 2024-02-05T04:53:26.347 INFO:tasks.rados.rados.0.smithi196.stdout:68: seq_num 56 ranges {475136=573440,1048576=376832} 2024-02-05T04:53:26.351 INFO:tasks.rados.rados.0.smithi196.stdout:68: writing smithi19639362-29 from 475136 to 1048576 tid 1 2024-02-05T04:53:26.354 INFO:tasks.rados.rados.0.smithi196.stdout:68: writing smithi19639362-29 from 1048576 to 1425408 tid 2 2024-02-05T04:53:26.354 INFO:tasks.rados.rados.0.smithi196.stdout:62: done (4 left) 2024-02-05T04:53:26.354 INFO:tasks.rados.rados.0.smithi196.stdout:69: snap_create 2024-02-05T04:53:26.354 INFO:tasks.rados.rados.0.smithi196.stdout:63: finishing write tid 3 to smithi19639362-40 2024-02-05T04:53:26.354 INFO:tasks.rados.rados.0.smithi196.stdout:63: finishing write tid 4 to smithi19639362-40 2024-02-05T04:53:26.354 INFO:tasks.rados.rados.0.smithi196.stdout:63: oid 40 updating version 0 to 11 2024-02-05T04:53:26.355 INFO:tasks.rados.rados.0.smithi196.stdout:63: oid 40 updating version 11 to 12 2024-02-05T04:53:26.355 INFO:tasks.rados.rados.0.smithi196.stdout:63: oid 40 version 12 is already newer than 10 2024-02-05T04:53:26.355 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 40 v 12 (ObjNum 54 snap 0 seq_num 54) dirty exists 2024-02-05T04:53:26.355 INFO:tasks.rados.rados.0.smithi196.stdout:63: left oid 40 (ObjNum 54 snap 0 seq_num 54) 2024-02-05T04:53:26.355 INFO:tasks.rados.rados.0.smithi196.stdout:67: expect (ObjNum 10 snap 0 seq_num 10) 2024-02-05T04:53:26.358 INFO:tasks.rados.rados.0.smithi196.stdout:65: expect (ObjNum 53 snap 0 seq_num 53) 2024-02-05T04:53:26.373 INFO:tasks.ceph.osd.0.smithi062.stderr:2024-02-05T04:53:26.372+0000 7fece8e03640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34688) UID: 0 2024-02-05T04:53:26.389 INFO:tasks.rados.rados.0.smithi196.stdout:68: finishing write tid 1 to smithi19639362-29 2024-02-05T04:53:26.390 INFO:tasks.rados.rados.0.smithi196.stdout:68: finishing write tid 2 to smithi19639362-29 2024-02-05T04:53:26.390 INFO:tasks.rados.rados.0.smithi196.stdout:68: finishing write tid 3 to smithi19639362-29 2024-02-05T04:53:26.390 INFO:tasks.rados.rados.0.smithi196.stdout:68: finishing write tid 4 to smithi19639362-29 2024-02-05T04:53:26.391 INFO:tasks.rados.rados.0.smithi196.stdout:68: oid 29 updating version 0 to 9 2024-02-05T04:53:26.391 INFO:tasks.rados.rados.0.smithi196.stdout:68: oid 29 updating version 9 to 10 2024-02-05T04:53:26.391 INFO:tasks.rados.rados.0.smithi196.stdout:68: oid 29 version 10 is already newer than 8 2024-02-05T04:53:26.391 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 29 v 10 (ObjNum 56 snap 0 seq_num 56) dirty exists 2024-02-05T04:53:26.391 INFO:tasks.rados.rados.0.smithi196.stdout:68: left oid 29 (ObjNum 56 snap 0 seq_num 56) 2024-02-05T04:53:26.451 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:26.451 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:26.460 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-02-05T04:53:26.474 INFO:tasks.ceph.osd.2.smithi150.stderr:2024-02-05T04:53:26.473+0000 7f7337bf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 34014) UID: 0 2024-02-05T04:53:26.575 INFO:tasks.ceph.osd.12.smithi062.stderr:2024-02-05T04:53:26.574+0000 7f5e5847e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 34695) UID: 0 2024-02-05T04:53:26.606 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:26.606 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:26.614 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-02-05T04:53:26.676 INFO:tasks.ceph.osd.3.smithi196.stderr:2024-02-05T04:53:26.676+0000 7fc8817c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34605) UID: 0 2024-02-05T04:53:26.723 INFO:teuthology.orchestra.run.smithi062.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:26.723 INFO:teuthology.orchestra.run.smithi062.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:26.731 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-02-05T04:53:26.778 INFO:tasks.ceph.osd.3.smithi196.stderr:2024-02-05T04:53:26.777+0000 7fc8817c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34605) UID: 0 2024-02-05T04:53:26.838 INFO:teuthology.orchestra.run.smithi062.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:26.838 INFO:teuthology.orchestra.run.smithi062.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:26.846 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-02-05T04:53:26.879 INFO:tasks.ceph.osd.5.smithi091.stderr:2024-02-05T04:53:26.879+0000 7f63983b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34277) UID: 0 2024-02-05T04:53:26.953 INFO:teuthology.orchestra.run.smithi062.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:26.954 INFO:teuthology.orchestra.run.smithi062.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:26.962 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-02-05T04:53:26.980 INFO:tasks.ceph.osd.2.smithi150.stderr:2024-02-05T04:53:26.979+0000 7f7337bf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 34014) UID: 0 2024-02-05T04:53:27.069 INFO:teuthology.orchestra.run.smithi062.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:27.070 INFO:teuthology.orchestra.run.smithi062.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:27.078 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-02-05T04:53:27.080 INFO:tasks.ceph.osd.9.smithi091.stderr:2024-02-05T04:53:27.080+0000 7efc18827640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 34279) UID: 0 2024-02-05T04:53:27.114 INFO:teuthology.orchestra.run.smithi062.stderr:noscrub is unset 2024-02-05T04:53:27.115 INFO:teuthology.orchestra.run.smithi062.stderr:reweighted osd.6 to 1 (10000) 2024-02-05T04:53:27.117 INFO:tasks.rados.rados.0.smithi196.stdout:63: done (4 left) 2024-02-05T04:53:27.117 INFO:tasks.rados.rados.0.smithi196.stdout:65: done (3 left) 2024-02-05T04:53:27.118 INFO:tasks.rados.rados.0.smithi196.stdout:67: done (2 left) 2024-02-05T04:53:27.118 INFO:tasks.rados.rados.0.smithi196.stdout:68: done (1 left) 2024-02-05T04:53:27.118 INFO:tasks.rados.rados.0.smithi196.stdout:69: done (0 left) 2024-02-05T04:53:27.118 INFO:tasks.rados.rados.0.smithi196.stdout:70: delete oid 15 current snap is 1 2024-02-05T04:53:27.122 INFO:tasks.rados.rados.0.smithi196.stdout:70: done (0 left) 2024-02-05T04:53:27.122 INFO:tasks.rados.rados.0.smithi196.stdout:append oid 42 current snap is 1 2024-02-05T04:53:27.122 INFO:tasks.rados.rados.0.smithi196.stdout:71: seq_num 58 ranges {704512=606208,1310720=376832} 2024-02-05T04:53:27.129 INFO:tasks.rados.rados.0.smithi196.stdout:71: writing smithi19639362-42 from 704512 to 1310720 tid 1 2024-02-05T04:53:27.132 INFO:tasks.rados.rados.0.smithi196.stdout:71: writing smithi19639362-42 from 1310720 to 1687552 tid 2 2024-02-05T04:53:27.132 INFO:tasks.rados.rados.0.smithi196.stdout:72: rollback oid 30 current snap is 1 2024-02-05T04:53:27.132 INFO:tasks.rados.rados.0.smithi196.stdout:rollback oid 30 to 0 2024-02-05T04:53:27.132 INFO:tasks.rados.rados.0.smithi196.stdout:append oid 14 current snap is 1 2024-02-05T04:53:27.132 INFO:tasks.rados.rados.0.smithi196.stdout:73: seq_num 59 ranges {0=622592,622592=376832} 2024-02-05T04:53:27.133 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-02-05T04:53:27.135 INFO:tasks.rados.rados.0.smithi196.stdout:73: writing smithi19639362-14 from 0 to 622592 tid 1 2024-02-05T04:53:27.139 INFO:tasks.rados.rados.0.smithi196.stdout:73: writing smithi19639362-14 from 622592 to 999424 tid 2 2024-02-05T04:53:27.139 INFO:tasks.rados.rados.0.smithi196.stdout:74: snap_create 2024-02-05T04:53:27.139 INFO:tasks.rados.rados.0.smithi196.stdout:71: finishing write tid 1 to smithi19639362-42 2024-02-05T04:53:27.140 INFO:tasks.rados.rados.0.smithi196.stdout:71: finishing write tid 2 to smithi19639362-42 2024-02-05T04:53:27.140 INFO:tasks.rados.rados.0.smithi196.stdout:71: finishing write tid 3 to smithi19639362-42 2024-02-05T04:53:27.140 INFO:tasks.rados.rados.0.smithi196.stdout:72: finishing rollback tid 0 to smithi19639362-30 2024-02-05T04:53:27.140 INFO:tasks.rados.rados.0.smithi196.stdout:71: finishing write tid 4 to smithi19639362-42 2024-02-05T04:53:27.140 INFO:tasks.rados.rados.0.smithi196.stdout:71: oid 42 updating version 0 to 8 2024-02-05T04:53:27.140 INFO:tasks.rados.rados.0.smithi196.stdout:71: oid 42 updating version 8 to 9 2024-02-05T04:53:27.140 INFO:tasks.rados.rados.0.smithi196.stdout:71: oid 42 updating version 9 to 10 2024-02-05T04:53:27.140 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 42 v 10 (ObjNum 58 snap 1 seq_num 58) dirty exists 2024-02-05T04:53:27.141 INFO:tasks.rados.rados.0.smithi196.stdout:71: left oid 42 (ObjNum 58 snap 1 seq_num 58) 2024-02-05T04:53:27.141 INFO:tasks.rados.rados.0.smithi196.stdout:72: finishing rollback tid 1 to smithi19639362-30 2024-02-05T04:53:27.141 INFO:tasks.rados.rados.0.smithi196.stdout:72: finishing rollback tid 2 to smithi19639362-30 2024-02-05T04:53:27.141 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 30 v 8 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-02-05T04:53:27.141 INFO:tasks.rados.rados.0.smithi196.stdout:73: finishing write tid 1 to smithi19639362-14 2024-02-05T04:53:27.142 INFO:tasks.rados.rados.0.smithi196.stdout:73: finishing write tid 2 to smithi19639362-14 2024-02-05T04:53:27.145 INFO:tasks.rados.rados.0.smithi196.stdout:73: finishing write tid 3 to smithi19639362-14 2024-02-05T04:53:27.147 INFO:tasks.rados.rados.0.smithi196.stdout:73: finishing write tid 4 to smithi19639362-14 2024-02-05T04:53:27.147 INFO:tasks.rados.rados.0.smithi196.stdout:73: oid 14 updating version 0 to 13 2024-02-05T04:53:27.147 INFO:tasks.rados.rados.0.smithi196.stdout:73: oid 14 version 13 is already newer than 12 2024-02-05T04:53:27.147 INFO:tasks.rados.rados.0.smithi196.stdout:73: oid 14 version 13 is already newer than 11 2024-02-05T04:53:27.147 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 14 v 13 (ObjNum 59 snap 1 seq_num 59) dirty exists 2024-02-05T04:53:27.147 INFO:tasks.rados.rados.0.smithi196.stdout:73: left oid 14 (ObjNum 59 snap 1 seq_num 59) 2024-02-05T04:53:27.181 INFO:tasks.ceph.osd.15.smithi196.stderr:2024-02-05T04:53:27.180+0000 7f1facc45640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 34611) UID: 0 2024-02-05T04:53:27.199 INFO:teuthology.orchestra.run.smithi062.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:27.199 INFO:teuthology.orchestra.run.smithi062.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:27.209 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-02-05T04:53:27.282 INFO:tasks.ceph.osd.9.smithi091.stderr:2024-02-05T04:53:27.281+0000 7efc18827640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 34279) UID: 0 2024-02-05T04:53:27.359 INFO:teuthology.orchestra.run.smithi062.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:27.359 INFO:teuthology.orchestra.run.smithi062.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:27.368 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-02-05T04:53:27.412 INFO:tasks.ceph.osd.4.smithi062.stderr:2024-02-05T04:53:27.411+0000 7f708c308640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34689) UID: 0 2024-02-05T04:53:27.476 INFO:teuthology.orchestra.run.smithi062.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:27.476 INFO:teuthology.orchestra.run.smithi062.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:27.484 INFO:tasks.ceph.osd.7.smithi196.stderr:2024-02-05T04:53:27.483+0000 7fb281419640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34606) UID: 0 2024-02-05T04:53:27.485 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-02-05T04:53:27.585 INFO:tasks.ceph.osd.15.smithi196.stderr:2024-02-05T04:53:27.585+0000 7f1facc45640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 34611) UID: 0 2024-02-05T04:53:27.632 INFO:teuthology.orchestra.run.smithi062.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:27.632 INFO:teuthology.orchestra.run.smithi062.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:27.642 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-02-05T04:53:27.686 INFO:tasks.ceph.osd.11.smithi196.stderr:2024-02-05T04:53:27.685+0000 7f85a8d86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 34609) UID: 0 2024-02-05T04:53:27.749 INFO:teuthology.orchestra.run.smithi062.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:27.749 INFO:teuthology.orchestra.run.smithi062.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:27.758 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-02-05T04:53:27.802 INFO:tasks.ceph.osd.0.smithi062.stderr:2024-02-05T04:53:27.801+0000 7fece8e03640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34688) UID: 0 2024-02-05T04:53:27.825 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T04:53:27.868 INFO:teuthology.orchestra.run.smithi062.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:27.868 INFO:teuthology.orchestra.run.smithi062.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:27.875 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-02-05T04:53:27.888 INFO:tasks.ceph.osd.14.smithi150.stderr:2024-02-05T04:53:27.887+0000 7f5f81083640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 34020) UID: 0 2024-02-05T04:53:27.950 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0: {} 2024-02-05T04:53:27.951 INFO:teuthology.orchestra.run.smithi062.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T04:53:27.969 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1: {} 2024-02-05T04:53:27.970 INFO:teuthology.orchestra.run.smithi062.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T04:53:27.988 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2: {} 2024-02-05T04:53:27.988 INFO:teuthology.orchestra.run.smithi062.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T04:53:27.989 INFO:tasks.ceph.osd.5.smithi091.stderr:2024-02-05T04:53:27.988+0000 7f63983b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34277) UID: 0 2024-02-05T04:53:28.006 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3: {} 2024-02-05T04:53:28.006 INFO:teuthology.orchestra.run.smithi062.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T04:53:28.024 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4: {} 2024-02-05T04:53:28.025 INFO:teuthology.orchestra.run.smithi062.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T04:53:28.025 INFO:teuthology.orchestra.run.smithi062.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:28.026 INFO:teuthology.orchestra.run.smithi062.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:28.035 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-02-05T04:53:28.042 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5: {} 2024-02-05T04:53:28.043 INFO:teuthology.orchestra.run.smithi062.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T04:53:28.061 INFO:teuthology.orchestra.run.smithi062.stdout:osd.6: {} 2024-02-05T04:53:28.061 INFO:teuthology.orchestra.run.smithi062.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T04:53:28.079 INFO:teuthology.orchestra.run.smithi062.stdout:osd.7: {} 2024-02-05T04:53:28.079 INFO:teuthology.orchestra.run.smithi062.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T04:53:28.089 INFO:tasks.ceph.osd.14.smithi150.stderr:2024-02-05T04:53:28.089+0000 7f5f81083640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 34020) UID: 0 2024-02-05T04:53:28.096 INFO:teuthology.orchestra.run.smithi062.stdout:osd.8: {} 2024-02-05T04:53:28.096 INFO:teuthology.orchestra.run.smithi062.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-05T04:53:28.114 INFO:teuthology.orchestra.run.smithi062.stdout:osd.9: {} 2024-02-05T04:53:28.114 INFO:teuthology.orchestra.run.smithi062.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-05T04:53:28.118 INFO:teuthology.orchestra.run.smithi062.stderr:reweighted osd.7 to 1 (10000) 2024-02-05T04:53:28.122 INFO:tasks.rados.rados.0.smithi196.stdout:71: done (3 left) 2024-02-05T04:53:28.122 INFO:tasks.rados.rados.0.smithi196.stdout:72: done (2 left) 2024-02-05T04:53:28.122 INFO:tasks.rados.rados.0.smithi196.stdout:73: done (1 left) 2024-02-05T04:53:28.122 INFO:tasks.rados.rados.0.smithi196.stdout:74: done (0 left) 2024-02-05T04:53:28.123 INFO:tasks.rados.rados.0.smithi196.stdout:75: read oid 18 snap -1 2024-02-05T04:53:28.123 INFO:tasks.rados.rados.0.smithi196.stdout:75: expect (ObjNum 17 snap 0 seq_num 17) 2024-02-05T04:53:28.123 INFO:tasks.rados.rados.0.smithi196.stdout:76: snap_remove snap 0 2024-02-05T04:53:28.132 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 8 1 2024-02-05T04:53:28.134 INFO:teuthology.orchestra.run.smithi062.stdout:osd.10: {} 2024-02-05T04:53:28.135 INFO:teuthology.orchestra.run.smithi062.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-05T04:53:28.153 INFO:teuthology.orchestra.run.smithi062.stdout:osd.11: {} 2024-02-05T04:53:28.154 INFO:teuthology.orchestra.run.smithi062.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-05T04:53:28.155 INFO:teuthology.orchestra.run.smithi062.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:28.155 INFO:teuthology.orchestra.run.smithi062.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:28.165 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-02-05T04:53:28.172 INFO:teuthology.orchestra.run.smithi062.stdout:osd.12: {} 2024-02-05T04:53:28.172 INFO:teuthology.orchestra.run.smithi062.stderr:osd.12: osd_enable_op_tracker = '' 2024-02-05T04:53:28.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-02-05T04:53:28.171+0000 7f98a2ffd640 0 -- 172.21.15.62:0/2850797475 >> [v2:172.21.15.91:6809/3656255795,v1:172.21.15.91:6812/3656255795] conn(0x7f98a42156d0 msgr2=0x7f98a4215b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-02-05T04:53:28.190 INFO:tasks.ceph.osd.11.smithi196.stderr:2024-02-05T04:53:28.190+0000 7f85a8d86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 34609) UID: 0 2024-02-05T04:53:28.283 INFO:teuthology.orchestra.run.smithi091.stdout:{ 2024-02-05T04:53:28.283 INFO:teuthology.orchestra.run.smithi091.stdout: "ops": [], 2024-02-05T04:53:28.283 INFO:teuthology.orchestra.run.smithi091.stdout: "num_ops": 0 2024-02-05T04:53:28.283 INFO:teuthology.orchestra.run.smithi091.stdout:} 2024-02-05T04:53:28.291 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-02-05T04:53:28.294 INFO:tasks.ceph.osd.3.smithi196.stderr:2024-02-05T04:53:28.293+0000 7fc8817c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34605) UID: 0 2024-02-05T04:53:28.390 INFO:teuthology.orchestra.run.smithi062.stdout:osd.13: {} 2024-02-05T04:53:28.390 INFO:teuthology.orchestra.run.smithi062.stderr:osd.13: osd_enable_op_tracker = '' 2024-02-05T04:53:28.394 INFO:tasks.ceph.osd.9.smithi091.stderr:2024-02-05T04:53:28.394+0000 7efc18827640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 34279) UID: 0 2024-02-05T04:53:28.399 INFO:teuthology.orchestra.run.smithi091.stdout:{ 2024-02-05T04:53:28.399 INFO:teuthology.orchestra.run.smithi091.stdout: "ops": [], 2024-02-05T04:53:28.399 INFO:teuthology.orchestra.run.smithi091.stdout: "complaint_time": 30, 2024-02-05T04:53:28.399 INFO:teuthology.orchestra.run.smithi091.stdout: "num_blocked_ops": 0 2024-02-05T04:53:28.399 INFO:teuthology.orchestra.run.smithi091.stdout:} 2024-02-05T04:53:28.408 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-02-05T04:53:28.408 INFO:teuthology.orchestra.run.smithi062.stdout:osd.14: {} 2024-02-05T04:53:28.409 INFO:teuthology.orchestra.run.smithi062.stderr:osd.14: osd_enable_op_tracker = '' 2024-02-05T04:53:28.425 INFO:teuthology.orchestra.run.smithi062.stdout:osd.15: {} 2024-02-05T04:53:28.425 INFO:teuthology.orchestra.run.smithi062.stderr:osd.15: osd_enable_op_tracker = '' 2024-02-05T04:53:28.496 INFO:tasks.ceph.osd.13.smithi091.stderr:2024-02-05T04:53:28.495+0000 7f526fd4d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 34281) UID: 0 2024-02-05T04:53:28.556 INFO:teuthology.orchestra.run.smithi091.stdout:{ 2024-02-05T04:53:28.556 INFO:teuthology.orchestra.run.smithi091.stdout: "size": 20, 2024-02-05T04:53:28.556 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 600, 2024-02-05T04:53:28.556 INFO:teuthology.orchestra.run.smithi091.stdout: "ops": [ 2024-02-05T04:53:28.556 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.556 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:214 3.4s0 3:25248d28:::smithi19639362-18:head [read 0~294912,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-02-05T04:53:28.556 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:28.122299+0000", 2024-02-05T04:53:28.556 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 0.43222362800000003, 2024-02-05T04:53:28.557 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0042913819999999998, 2024-02-05T04:53:28.557 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:28.557 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "started", 2024-02-05T04:53:28.557 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:53:28.557 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:53:28.557 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:28.557 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 214 2024-02-05T04:53:28.557 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.557 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:28.557 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.557 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:28.557 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.122299+0000", 2024-02-05T04:53:28.557 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:28.557 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.558 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.558 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:28.558 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.122299+0000", 2024-02-05T04:53:28.558 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:28.558 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.558 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.558 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:28.558 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.122300+0000", 2024-02-05T04:53:28.558 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.7680000000000001e-06 2024-02-05T04:53:28.558 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.558 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.558 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:28.558 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.122310+0000", 2024-02-05T04:53:28.558 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.2310000000000002e-06 2024-02-05T04:53:28.559 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.559 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.559 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:28.559 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.122326+0000", 2024-02-05T04:53:28.559 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.6886000000000001e-05 2024-02-05T04:53:28.559 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.559 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.559 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:28.559 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.122335+0000", 2024-02-05T04:53:28.559 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.0119999999999994e-06 2024-02-05T04:53:28.559 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.559 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.559 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:28.559 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.122417+0000", 2024-02-05T04:53:28.560 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.1366999999999995e-05 2024-02-05T04:53:28.560 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.560 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.560 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "op must wait for map", 2024-02-05T04:53:28.560 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.122426+0000", 2024-02-05T04:53:28.560 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.2210000000000003e-06 2024-02-05T04:53:28.560 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.560 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.560 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:28.560 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.124382+0000", 2024-02-05T04:53:28.560 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.001955891 2024-02-05T04:53:28.560 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.560 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.560 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:53:28.561 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.124446+0000", 2024-02-05T04:53:28.561 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.4525999999999999e-05 2024-02-05T04:53:28.561 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.561 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.561 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:28.561 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.126590+0000", 2024-02-05T04:53:28.561 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0021434800000000001 2024-02-05T04:53:28.561 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:28.561 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:28.561 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:28.561 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.561 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.561 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:215 3.4s0 3:25248d28:::smithi19639362-18:head [read 0~294912,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-02-05T04:53:28.561 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:28.122352+0000", 2024-02-05T04:53:28.561 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 0.43216973199999997, 2024-02-05T04:53:28.562 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0046661749999999998, 2024-02-05T04:53:28.562 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:28.562 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "started", 2024-02-05T04:53:28.562 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:53:28.562 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:53:28.562 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:28.562 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 215 2024-02-05T04:53:28.562 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.562 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:28.562 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.562 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:28.562 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.122352+0000", 2024-02-05T04:53:28.562 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:28.562 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.563 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.563 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:28.563 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.122352+0000", 2024-02-05T04:53:28.563 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:28.563 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.563 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.563 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:28.563 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.122353+0000", 2024-02-05T04:53:28.563 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.4900000000000001e-07 2024-02-05T04:53:28.563 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.563 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.563 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:28.563 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.122355+0000", 2024-02-05T04:53:28.563 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.4949999999999998e-06 2024-02-05T04:53:28.563 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.564 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.564 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:28.564 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.122360+0000", 2024-02-05T04:53:28.564 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.7380000000000002e-06 2024-02-05T04:53:28.564 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.564 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.564 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:28.564 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.122371+0000", 2024-02-05T04:53:28.564 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.0458e-05 2024-02-05T04:53:28.564 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.564 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.564 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:28.564 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.122505+0000", 2024-02-05T04:53:28.564 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00013468599999999999 2024-02-05T04:53:28.565 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.565 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.565 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "waiting_for_map not empty", 2024-02-05T04:53:28.565 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.122509+0000", 2024-02-05T04:53:28.565 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.968e-06 2024-02-05T04:53:28.565 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.565 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.565 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:28.565 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.124887+0000", 2024-02-05T04:53:28.565 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.002377966 2024-02-05T04:53:28.565 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.565 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.565 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:53:28.565 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.124978+0000", 2024-02-05T04:53:28.565 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.0427999999999995e-05 2024-02-05T04:53:28.565 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.566 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.566 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:28.566 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.127019+0000", 2024-02-05T04:53:28.566 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.002041087 2024-02-05T04:53:28.566 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:28.566 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:28.566 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:28.566 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.566 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.566 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:216 3.4s0 3:25248d28:::smithi19639362-18:head [read 0~294912,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-02-05T04:53:28.566 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:28.122406+0000", 2024-02-05T04:53:28.566 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 0.43211660099999999, 2024-02-05T04:53:28.566 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0056680589999999996, 2024-02-05T04:53:28.566 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:28.567 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "started", 2024-02-05T04:53:28.567 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:53:28.567 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:53:28.567 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:28.567 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 216 2024-02-05T04:53:28.567 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.567 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:28.567 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.567 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:28.567 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.122406+0000", 2024-02-05T04:53:28.567 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:28.567 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.567 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.567 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:28.568 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.122406+0000", 2024-02-05T04:53:28.568 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:28.568 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.568 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.568 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:28.568 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.122406+0000", 2024-02-05T04:53:28.568 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.8000000000000001e-07 2024-02-05T04:53:28.568 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.568 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.568 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:28.568 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.122408+0000", 2024-02-05T04:53:28.568 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.4600000000000002e-06 2024-02-05T04:53:28.568 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.568 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.569 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:28.569 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.122413+0000", 2024-02-05T04:53:28.569 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.0209999999999999e-06 2024-02-05T04:53:28.569 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.569 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.569 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:28.569 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.122417+0000", 2024-02-05T04:53:28.569 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.6239999999999999e-06 2024-02-05T04:53:28.569 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.569 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.569 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:28.569 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.122541+0000", 2024-02-05T04:53:28.569 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00012391100000000001 2024-02-05T04:53:28.569 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.570 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.570 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "waiting_for_map not empty", 2024-02-05T04:53:28.570 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.122547+0000", 2024-02-05T04:53:28.570 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.682e-06 2024-02-05T04:53:28.570 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.570 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.570 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:28.570 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.127053+0000", 2024-02-05T04:53:28.570 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0045067170000000004 2024-02-05T04:53:28.570 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.570 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.570 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:53:28.570 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.127105+0000", 2024-02-05T04:53:28.570 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.1458000000000001e-05 2024-02-05T04:53:28.570 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.571 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.571 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:28.571 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.128074+0000", 2024-02-05T04:53:28.571 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00096880600000000001 2024-02-05T04:53:28.571 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:28.571 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:28.571 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:28.571 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.571 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.571 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpRead(3.4s0 36/31 ECSubRead(tid=19, to_read={3:25248d28:::smithi19639362-18:head=0,73728,0}, subchunks={3:25248d28:::smithi19639362-18:head=[0,1]}, attrs_to_read=))", 2024-02-05T04:53:28.571 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:28.124729+0000", 2024-02-05T04:53:28.571 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 0.42979281899999999, 2024-02-05T04:53:28.571 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0010492699999999999, 2024-02-05T04:53:28.571 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:28.572 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:28.572 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:28.572 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.572 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:28.572 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.124729+0000", 2024-02-05T04:53:28.572 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:28.572 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.572 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.572 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:28.572 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.124729+0000", 2024-02-05T04:53:28.572 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:28.572 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.572 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.573 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:28.573 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.124729+0000", 2024-02-05T04:53:28.573 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:28.573 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.573 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.573 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:28.573 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.124729+0000", 2024-02-05T04:53:28.573 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:28.573 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.573 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.573 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:28.573 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.124751+0000", 2024-02-05T04:53:28.573 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.1438e-05 2024-02-05T04:53:28.573 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.573 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.573 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:28.574 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.124766+0000", 2024-02-05T04:53:28.574 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.5481e-05 2024-02-05T04:53:28.574 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.574 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.574 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:28.574 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.125301+0000", 2024-02-05T04:53:28.574 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00053510799999999996 2024-02-05T04:53:28.574 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.574 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.574 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:28.574 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.125779+0000", 2024-02-05T04:53:28.574 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00047724300000000002 2024-02-05T04:53:28.574 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:28.574 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:28.575 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:28.575 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.575 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.575 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpRead(3.4s0 36/31 ECSubRead(tid=20, to_read={3:25248d28:::smithi19639362-18:head=0,73728,0}, subchunks={3:25248d28:::smithi19639362-18:head=[0,1]}, attrs_to_read=))", 2024-02-05T04:53:28.575 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:28.125151+0000", 2024-02-05T04:53:28.575 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 0.42937133999999999, 2024-02-05T04:53:28.575 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00076732799999999996, 2024-02-05T04:53:28.575 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:28.575 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:28.575 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:28.575 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.575 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:28.575 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.125151+0000", 2024-02-05T04:53:28.575 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:28.575 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.576 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.576 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:28.576 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.125151+0000", 2024-02-05T04:53:28.576 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:28.576 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.576 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.576 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:28.576 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.125151+0000", 2024-02-05T04:53:28.576 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:28.576 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.576 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.576 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:28.577 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.125151+0000", 2024-02-05T04:53:28.577 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:28.577 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.577 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.577 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:28.577 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.125172+0000", 2024-02-05T04:53:28.577 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.1466000000000001e-05 2024-02-05T04:53:28.577 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.577 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.577 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:28.577 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.125185+0000", 2024-02-05T04:53:28.577 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.2524000000000001e-05 2024-02-05T04:53:28.577 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.578 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.578 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:28.578 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.125822+0000", 2024-02-05T04:53:28.578 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00063664500000000003 2024-02-05T04:53:28.578 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.578 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.578 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:28.578 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.125918+0000", 2024-02-05T04:53:28.578 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.6693000000000004e-05 2024-02-05T04:53:28.578 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:28.578 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:28.578 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:28.578 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.578 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.578 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpReadReply(3.4s0 35/31 ECSubReadReply(tid=19, attrs_read=0))", 2024-02-05T04:53:28.579 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:28.125487+0000", 2024-02-05T04:53:28.579 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 0.42903562000000001, 2024-02-05T04:53:28.579 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00052389500000000002, 2024-02-05T04:53:28.579 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:28.579 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:28.579 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:28.579 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.579 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:28.579 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.125487+0000", 2024-02-05T04:53:28.579 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:28.579 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.579 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.579 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:28.579 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.125487+0000", 2024-02-05T04:53:28.579 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:28.580 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.580 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.580 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:28.580 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.125488+0000", 2024-02-05T04:53:28.580 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.102e-06 2024-02-05T04:53:28.580 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.580 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.580 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:28.580 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.125637+0000", 2024-02-05T04:53:28.580 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.000148944 2024-02-05T04:53:28.580 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.580 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.580 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:28.580 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.125653+0000", 2024-02-05T04:53:28.581 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.6453e-05 2024-02-05T04:53:28.581 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.581 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.581 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:28.581 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.125661+0000", 2024-02-05T04:53:28.581 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.2570000000000007e-06 2024-02-05T04:53:28.581 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.581 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.581 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:28.581 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.125964+0000", 2024-02-05T04:53:28.581 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00030310200000000003 2024-02-05T04:53:28.581 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.581 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.581 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:28.581 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.126010+0000", 2024-02-05T04:53:28.582 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.6037e-05 2024-02-05T04:53:28.582 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:28.582 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:28.582 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:28.582 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.582 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.582 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpReadReply(3.4s0 36/31 ECSubReadReply(tid=19, attrs_read=0))", 2024-02-05T04:53:28.582 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:28.125578+0000", 2024-02-05T04:53:28.582 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 0.42894432199999999, 2024-02-05T04:53:28.582 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0010275950000000001, 2024-02-05T04:53:28.582 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:28.582 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:28.582 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:28.582 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.582 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:28.582 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.125578+0000", 2024-02-05T04:53:28.583 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:28.583 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.583 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.583 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:28.583 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.125578+0000", 2024-02-05T04:53:28.583 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:28.583 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.583 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.583 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:28.583 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.125579+0000", 2024-02-05T04:53:28.583 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.8599999999999996e-07 2024-02-05T04:53:28.583 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.583 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.584 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:28.584 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.126005+0000", 2024-02-05T04:53:28.584 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00042628000000000002 2024-02-05T04:53:28.584 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.584 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.584 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:28.584 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.126013+0000", 2024-02-05T04:53:28.584 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.3159999999999997e-06 2024-02-05T04:53:28.584 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.584 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.584 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:28.584 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.126019+0000", 2024-02-05T04:53:28.584 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.4890000000000003e-06 2024-02-05T04:53:28.584 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.584 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.585 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:28.585 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.126279+0000", 2024-02-05T04:53:28.585 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00025983799999999997 2024-02-05T04:53:28.585 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.585 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.585 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:28.585 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.126605+0000", 2024-02-05T04:53:28.585 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00032668600000000002 2024-02-05T04:53:28.585 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:28.585 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:28.585 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:28.585 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.585 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.585 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpReadReply(3.4s0 35/31 ECSubReadReply(tid=19, attrs_read=0))", 2024-02-05T04:53:28.586 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:28.125608+0000", 2024-02-05T04:53:28.586 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 0.42891426500000002, 2024-02-05T04:53:28.586 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00063411700000000001, 2024-02-05T04:53:28.586 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:28.586 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:28.586 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:28.586 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.586 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:28.586 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.125608+0000", 2024-02-05T04:53:28.586 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:28.586 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.586 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.586 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:28.586 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.125608+0000", 2024-02-05T04:53:28.586 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:28.587 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.587 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.587 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:28.587 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.125608+0000", 2024-02-05T04:53:28.587 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.39e-07 2024-02-05T04:53:28.587 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.587 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.587 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:28.587 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.125757+0000", 2024-02-05T04:53:28.587 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00014903799999999999 2024-02-05T04:53:28.587 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.587 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.587 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:28.587 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.125769+0000", 2024-02-05T04:53:28.588 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.1184e-05 2024-02-05T04:53:28.588 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.588 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.588 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:28.588 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.125775+0000", 2024-02-05T04:53:28.588 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.9100000000000002e-06 2024-02-05T04:53:28.588 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.588 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.588 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:28.588 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.126209+0000", 2024-02-05T04:53:28.588 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.000434184 2024-02-05T04:53:28.588 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.588 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.588 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:28.588 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.126242+0000", 2024-02-05T04:53:28.589 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.3361999999999997e-05 2024-02-05T04:53:28.589 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:28.589 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:28.589 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:28.589 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.589 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.589 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpReadReply(3.4s0 35/31 ECSubReadReply(tid=20, attrs_read=0))", 2024-02-05T04:53:28.589 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:28.125693+0000", 2024-02-05T04:53:28.589 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 0.42882901600000001, 2024-02-05T04:53:28.589 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00039345, 2024-02-05T04:53:28.589 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:28.589 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:28.589 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:28.589 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.589 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:28.590 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.125693+0000", 2024-02-05T04:53:28.590 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:28.590 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.590 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.590 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:28.590 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.125693+0000", 2024-02-05T04:53:28.590 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:28.590 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.590 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.590 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:28.590 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.125694+0000", 2024-02-05T04:53:28.590 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.5799999999999999e-07 2024-02-05T04:53:28.590 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.590 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.590 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:28.591 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.125774+0000", 2024-02-05T04:53:28.591 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.0383999999999995e-05 2024-02-05T04:53:28.591 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.591 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.591 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:28.591 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.125788+0000", 2024-02-05T04:53:28.591 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.4104000000000001e-05 2024-02-05T04:53:28.591 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.591 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.591 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:28.591 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.125795+0000", 2024-02-05T04:53:28.591 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.754e-06 2024-02-05T04:53:28.591 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.591 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.591 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:28.592 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.126049+0000", 2024-02-05T04:53:28.592 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00025434500000000001 2024-02-05T04:53:28.592 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.592 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.592 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:28.592 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.126087+0000", 2024-02-05T04:53:28.592 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.7305000000000003e-05 2024-02-05T04:53:28.592 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:28.592 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:28.592 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:28.592 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.592 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.592 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpReadReply(3.4s0 36/31 ECSubReadReply(tid=19, attrs_read=0))", 2024-02-05T04:53:28.592 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:28.125768+0000", 2024-02-05T04:53:28.592 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 0.42875449799999998, 2024-02-05T04:53:28.593 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00041003799999999999, 2024-02-05T04:53:28.593 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:28.593 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:28.593 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:28.593 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.593 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:28.593 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.125768+0000", 2024-02-05T04:53:28.593 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:28.593 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.593 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.593 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:28.593 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.125768+0000", 2024-02-05T04:53:28.593 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:28.593 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.593 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.594 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:28.594 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.125768+0000", 2024-02-05T04:53:28.594 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:28.594 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.594 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.594 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:28.594 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.125768+0000", 2024-02-05T04:53:28.594 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:28.594 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.594 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.594 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:28.594 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.125792+0000", 2024-02-05T04:53:28.594 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.4070000000000002e-05 2024-02-05T04:53:28.594 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.594 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.594 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:28.595 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.125800+0000", 2024-02-05T04:53:28.595 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.2910000000000007e-06 2024-02-05T04:53:28.595 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.595 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.595 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:28.595 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.126146+0000", 2024-02-05T04:53:28.595 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00034561200000000002 2024-02-05T04:53:28.595 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.595 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.595 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:28.595 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.126178+0000", 2024-02-05T04:53:28.595 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.2064999999999999e-05 2024-02-05T04:53:28.595 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:28.595 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:28.595 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:28.596 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.596 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.596 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpReadReply(3.4s0 35/31 ECSubReadReply(tid=20, attrs_read=0))", 2024-02-05T04:53:28.596 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:28.125824+0000", 2024-02-05T04:53:28.596 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 0.42869794900000002, 2024-02-05T04:53:28.596 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00088597800000000005, 2024-02-05T04:53:28.596 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:28.596 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:28.596 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:28.596 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.596 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:28.596 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.125824+0000", 2024-02-05T04:53:28.596 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:28.596 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.596 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.597 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:28.597 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.125824+0000", 2024-02-05T04:53:28.597 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:28.597 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.597 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.597 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:28.597 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.125825+0000", 2024-02-05T04:53:28.597 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.3000000000000001e-07 2024-02-05T04:53:28.597 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.597 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.597 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:28.597 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.125905+0000", 2024-02-05T04:53:28.597 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.0588999999999997e-05 2024-02-05T04:53:28.597 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.597 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.598 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:28.598 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.125914+0000", 2024-02-05T04:53:28.598 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.3890000000000005e-06 2024-02-05T04:53:28.598 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.598 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.598 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:28.598 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.125919+0000", 2024-02-05T04:53:28.598 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.1189999999999996e-06 2024-02-05T04:53:28.598 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.598 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.598 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:28.598 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.126689+0000", 2024-02-05T04:53:28.598 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00076995999999999998 2024-02-05T04:53:28.598 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.598 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.598 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:28.599 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.126710+0000", 2024-02-05T04:53:28.599 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.1491000000000001e-05 2024-02-05T04:53:28.599 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:28.599 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:28.599 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:28.599 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.599 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.599 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpReadReply(3.4s0 36/31 ECSubReadReply(tid=20, attrs_read=0))", 2024-02-05T04:53:28.599 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:28.125910+0000", 2024-02-05T04:53:28.599 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 0.42861191799999998, 2024-02-05T04:53:28.599 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00075263700000000001, 2024-02-05T04:53:28.599 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:28.599 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:28.599 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:28.599 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.600 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:28.600 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.125910+0000", 2024-02-05T04:53:28.600 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:28.600 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.600 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.600 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:28.600 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.125910+0000", 2024-02-05T04:53:28.600 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:28.600 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.600 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.600 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:28.600 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.125910+0000", 2024-02-05T04:53:28.600 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:28.600 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.600 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.601 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:28.601 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.125910+0000", 2024-02-05T04:53:28.601 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:28.601 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.601 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.601 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:28.601 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.125924+0000", 2024-02-05T04:53:28.601 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.4028e-05 2024-02-05T04:53:28.601 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.601 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.601 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:28.601 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.125930+0000", 2024-02-05T04:53:28.601 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.7300000000000002e-06 2024-02-05T04:53:28.601 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.601 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.601 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:28.602 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.126639+0000", 2024-02-05T04:53:28.602 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00070904300000000002 2024-02-05T04:53:28.602 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.602 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.602 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:28.602 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.126663+0000", 2024-02-05T04:53:28.602 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.3836e-05 2024-02-05T04:53:28.602 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:28.602 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:28.602 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:28.602 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.602 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.602 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpReadReply(3.4s0 36/31 ECSubReadReply(tid=20, attrs_read=0))", 2024-02-05T04:53:28.602 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:28.126044+0000", 2024-02-05T04:53:28.602 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 0.42847854099999999, 2024-02-05T04:53:28.602 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0009915709999999999, 2024-02-05T04:53:28.603 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:28.603 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:28.603 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:28.603 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.603 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:28.603 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.126044+0000", 2024-02-05T04:53:28.603 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:28.603 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.603 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.603 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:28.603 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.126044+0000", 2024-02-05T04:53:28.603 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:28.603 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.603 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.603 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:28.604 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.126044+0000", 2024-02-05T04:53:28.604 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.2599999999999998e-07 2024-02-05T04:53:28.604 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.604 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.604 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:28.604 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.126106+0000", 2024-02-05T04:53:28.604 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.1798999999999996e-05 2024-02-05T04:53:28.604 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.604 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.604 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:28.604 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.126130+0000", 2024-02-05T04:53:28.604 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.4011000000000001e-05 2024-02-05T04:53:28.604 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.604 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.605 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:28.605 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.126136+0000", 2024-02-05T04:53:28.605 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.0650000000000004e-06 2024-02-05T04:53:28.605 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.605 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.605 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:28.605 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.126734+0000", 2024-02-05T04:53:28.605 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00059794100000000005 2024-02-05T04:53:28.605 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.605 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.605 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:28.605 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.127035+0000", 2024-02-05T04:53:28.605 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00030132899999999997 2024-02-05T04:53:28.606 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:28.606 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:28.606 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:28.606 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.606 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.606 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpRead(3.4s0 36/31 ECSubRead(tid=21, to_read={3:25248d28:::smithi19639362-18:head=0,73728,0}, subchunks={3:25248d28:::smithi19639362-18:head=[0,1]}, attrs_to_read=))", 2024-02-05T04:53:28.606 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:28.127212+0000", 2024-02-05T04:53:28.606 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 0.42731040300000001, 2024-02-05T04:53:28.606 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00014758300000000001, 2024-02-05T04:53:28.606 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:28.606 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:28.606 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:28.606 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.606 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:28.606 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.127212+0000", 2024-02-05T04:53:28.607 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:28.607 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.607 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.607 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:28.607 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.127212+0000", 2024-02-05T04:53:28.607 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:28.607 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.607 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.607 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:28.607 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.127212+0000", 2024-02-05T04:53:28.607 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:28.607 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.607 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.607 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:28.607 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.127212+0000", 2024-02-05T04:53:28.608 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:28.608 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.608 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.608 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:28.608 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.127226+0000", 2024-02-05T04:53:28.608 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.4572e-05 2024-02-05T04:53:28.608 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.608 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.608 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:28.608 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.127232+0000", 2024-02-05T04:53:28.608 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.93e-06 2024-02-05T04:53:28.608 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.608 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.609 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:28.609 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.127293+0000", 2024-02-05T04:53:28.609 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.0365000000000002e-05 2024-02-05T04:53:28.609 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.609 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.609 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:28.609 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.127359+0000", 2024-02-05T04:53:28.609 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.6716000000000003e-05 2024-02-05T04:53:28.609 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:28.609 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:28.609 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:28.609 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.609 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.609 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpReadReply(3.4s0 36/31 ECSubReadReply(tid=21, attrs_read=0))", 2024-02-05T04:53:28.610 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:28.127352+0000", 2024-02-05T04:53:28.610 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 0.42717002900000001, 2024-02-05T04:53:28.610 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.8602000000000003e-05, 2024-02-05T04:53:28.610 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:28.610 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:28.610 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:28.610 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.610 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:28.610 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.127352+0000", 2024-02-05T04:53:28.610 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:28.610 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.610 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.610 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:28.610 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.127352+0000", 2024-02-05T04:53:28.610 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:28.611 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.611 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.611 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:28.611 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.127352+0000", 2024-02-05T04:53:28.611 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:28.611 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.611 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.611 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:28.611 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.127352+0000", 2024-02-05T04:53:28.611 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:28.611 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.611 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.611 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:28.611 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.127374+0000", 2024-02-05T04:53:28.612 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.1560999999999999e-05 2024-02-05T04:53:28.612 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.612 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.612 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:28.612 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.127381+0000", 2024-02-05T04:53:28.612 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.0040000000000002e-06 2024-02-05T04:53:28.612 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.612 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.612 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:28.612 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.127422+0000", 2024-02-05T04:53:28.612 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.1028000000000001e-05 2024-02-05T04:53:28.612 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.612 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.612 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:28.612 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.127441+0000", 2024-02-05T04:53:28.613 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.9009e-05 2024-02-05T04:53:28.613 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:28.613 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:28.613 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:28.613 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.613 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.613 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpReadReply(3.4s0 36/31 ECSubReadReply(tid=21, attrs_read=0))", 2024-02-05T04:53:28.613 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:28.127526+0000", 2024-02-05T04:53:28.613 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 0.42699631100000002, 2024-02-05T04:53:28.613 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.000110939, 2024-02-05T04:53:28.613 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:28.613 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:28.613 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:28.613 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.614 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:28.614 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.127526+0000", 2024-02-05T04:53:28.614 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:28.614 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.614 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.614 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:28.614 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.127526+0000", 2024-02-05T04:53:28.614 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:28.614 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.614 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.614 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:28.614 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.127526+0000", 2024-02-05T04:53:28.614 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.7000000000000005e-07 2024-02-05T04:53:28.614 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.614 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.615 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:28.615 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.127555+0000", 2024-02-05T04:53:28.615 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.8050000000000001e-05 2024-02-05T04:53:28.615 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.615 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.615 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:28.615 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.127567+0000", 2024-02-05T04:53:28.615 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.2339e-05 2024-02-05T04:53:28.615 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.615 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.615 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:28.615 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.127573+0000", 2024-02-05T04:53:28.615 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.6710000000000004e-06 2024-02-05T04:53:28.615 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.615 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.615 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:28.616 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.127616+0000", 2024-02-05T04:53:28.616 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.3245999999999999e-05 2024-02-05T04:53:28.616 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.616 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.616 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:28.616 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.127637+0000", 2024-02-05T04:53:28.616 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.1063000000000001e-05 2024-02-05T04:53:28.616 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:28.616 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:28.616 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:28.616 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.616 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.616 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpReadReply(3.4s0 36/31 ECSubReadReply(tid=21, attrs_read=0))", 2024-02-05T04:53:28.616 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:28.127649+0000", 2024-02-05T04:53:28.616 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 0.42687317499999999, 2024-02-05T04:53:28.616 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.000156599, 2024-02-05T04:53:28.617 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:28.617 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:28.617 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:28.617 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.617 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:28.617 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.127649+0000", 2024-02-05T04:53:28.617 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:28.617 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.617 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.617 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:28.617 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.127649+0000", 2024-02-05T04:53:28.617 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:28.617 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.617 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.617 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:28.618 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.127650+0000", 2024-02-05T04:53:28.618 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.3099999999999996e-07 2024-02-05T04:53:28.618 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.618 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.618 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:28.618 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.127723+0000", 2024-02-05T04:53:28.618 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.3300000000000006e-05 2024-02-05T04:53:28.618 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.618 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.618 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:28.618 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.127737+0000", 2024-02-05T04:53:28.618 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.3648e-05 2024-02-05T04:53:28.618 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.618 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.618 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:28.619 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.127744+0000", 2024-02-05T04:53:28.619 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.6260000000000003e-06 2024-02-05T04:53:28.619 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.619 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.619 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:28.619 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.127788+0000", 2024-02-05T04:53:28.619 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.3273999999999999e-05 2024-02-05T04:53:28.619 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.619 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.619 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:28.619 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.127806+0000", 2024-02-05T04:53:28.619 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.7920000000000001e-05 2024-02-05T04:53:28.619 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:28.619 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:28.619 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:28.620 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.620 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.620 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpReadReply(3.4s0 36/31 ECSubReadReply(tid=21, attrs_read=0))", 2024-02-05T04:53:28.620 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:28.127787+0000", 2024-02-05T04:53:28.620 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 0.42673478799999998, 2024-02-05T04:53:28.620 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.000297103, 2024-02-05T04:53:28.620 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:28.620 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:28.620 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:28.620 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.620 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:28.620 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.127787+0000", 2024-02-05T04:53:28.620 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:28.620 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.621 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.621 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:28.621 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.127787+0000", 2024-02-05T04:53:28.621 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:28.621 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.621 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.621 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:28.621 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.127788+0000", 2024-02-05T04:53:28.621 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.4700000000000001e-07 2024-02-05T04:53:28.621 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.621 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.621 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:28.621 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.127826+0000", 2024-02-05T04:53:28.621 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.7644000000000001e-05 2024-02-05T04:53:28.622 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.622 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.622 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:28.622 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.127834+0000", 2024-02-05T04:53:28.622 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.1710000000000002e-06 2024-02-05T04:53:28.622 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.622 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.622 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:28.622 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.127839+0000", 2024-02-05T04:53:28.622 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.66e-06 2024-02-05T04:53:28.622 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.622 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.622 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:28.622 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.127874+0000", 2024-02-05T04:53:28.623 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.4725999999999999e-05 2024-02-05T04:53:28.623 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:28.623 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:28.623 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:28.623 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.128085+0000", 2024-02-05T04:53:28.623 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00021035499999999999 2024-02-05T04:53:28.623 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:28.623 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:28.623 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:28.623 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:28.623 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:28.623 INFO:teuthology.orchestra.run.smithi091.stdout:} 2024-02-05T04:53:28.625 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-02-05T04:53:28.625 INFO:tasks.ceph.osd.6.smithi150.stderr:2024-02-05T04:53:28.624+0000 7f6bea9f0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34015) UID: 0 2024-02-05T04:53:28.726 INFO:tasks.ceph.osd.8.smithi062.stderr:2024-02-05T04:53:28.725+0000 7f9385bf2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 34693) UID: 0 2024-02-05T04:53:28.771 INFO:teuthology.orchestra.run.smithi091.stdout:{ 2024-02-05T04:53:28.771 INFO:teuthology.orchestra.run.smithi091.stdout: "ops": [], 2024-02-05T04:53:28.771 INFO:teuthology.orchestra.run.smithi091.stdout: "num_ops": 0 2024-02-05T04:53:28.771 INFO:teuthology.orchestra.run.smithi091.stdout:} 2024-02-05T04:53:28.779 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-02-05T04:53:28.826 INFO:tasks.ceph.osd.7.smithi196.stderr:2024-02-05T04:53:28.826+0000 7fb281419640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34606) UID: 0 2024-02-05T04:53:28.884 INFO:teuthology.orchestra.run.smithi091.stdout:{ 2024-02-05T04:53:28.884 INFO:teuthology.orchestra.run.smithi091.stdout: "ops": [], 2024-02-05T04:53:28.884 INFO:teuthology.orchestra.run.smithi091.stdout: "complaint_time": 30, 2024-02-05T04:53:28.884 INFO:teuthology.orchestra.run.smithi091.stdout: "num_blocked_ops": 0 2024-02-05T04:53:28.884 INFO:teuthology.orchestra.run.smithi091.stdout:} 2024-02-05T04:53:28.893 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-02-05T04:53:28.936 INFO:tasks.ceph.osd.9.smithi091.stderr:2024-02-05T04:53:28.936+0000 7efc18827640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 34279) UID: 0 2024-02-05T04:53:28.999 INFO:teuthology.orchestra.run.smithi091.stdout:{ 2024-02-05T04:53:28.999 INFO:teuthology.orchestra.run.smithi091.stdout: "size": 20, 2024-02-05T04:53:28.999 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 600, 2024-02-05T04:53:28.999 INFO:teuthology.orchestra.run.smithi091.stdout: "ops": [ 2024-02-05T04:53:29.000 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.000 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpRead(3.4s2 36/31 ECSubRead(tid=19, to_read={3:25248d28:::smithi19639362-18:head=0,73728,0}, subchunks={3:25248d28:::smithi19639362-18:head=[0,1]}, attrs_to_read=))", 2024-02-05T04:53:29.000 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:28.124883+0000", 2024-02-05T04:53:29.000 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 0.87358265499999999, 2024-02-05T04:53:29.000 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00056562500000000005, 2024-02-05T04:53:29.000 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:29.000 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:29.001 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:29.001 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.001 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:29.001 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.124883+0000", 2024-02-05T04:53:29.001 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:29.001 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.001 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.001 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:29.002 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.124883+0000", 2024-02-05T04:53:29.002 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:29.002 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.002 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.002 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:29.002 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.124883+0000", 2024-02-05T04:53:29.002 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.05e-07 2024-02-05T04:53:29.002 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.002 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.003 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:29.003 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.124890+0000", 2024-02-05T04:53:29.003 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.6699999999999997e-06 2024-02-05T04:53:29.003 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.003 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.003 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:29.003 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.124916+0000", 2024-02-05T04:53:29.003 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.6534e-05 2024-02-05T04:53:29.004 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.004 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.004 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:29.004 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.124925+0000", 2024-02-05T04:53:29.004 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.5879999999999998e-06 2024-02-05T04:53:29.004 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.004 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.005 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:29.005 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.125014+0000", 2024-02-05T04:53:29.005 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.9133000000000002e-05 2024-02-05T04:53:29.005 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.005 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.005 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:29.005 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.125448+0000", 2024-02-05T04:53:29.005 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00043449500000000002 2024-02-05T04:53:29.005 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:29.005 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:29.005 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:29.005 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.005 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.005 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpRead(3.4s2 36/31 ECSubRead(tid=20, to_read={3:25248d28:::smithi19639362-18:head=0,73728,0}, subchunks={3:25248d28:::smithi19639362-18:head=[0,1]}, attrs_to_read=))", 2024-02-05T04:53:29.006 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:28.125246+0000", 2024-02-05T04:53:29.006 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 0.87321967599999994, 2024-02-05T04:53:29.006 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00035423699999999998, 2024-02-05T04:53:29.006 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:29.006 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:29.006 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:29.006 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.006 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:29.006 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.125246+0000", 2024-02-05T04:53:29.007 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:29.007 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.007 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.007 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:29.007 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.125246+0000", 2024-02-05T04:53:29.007 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:29.007 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.007 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.007 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:29.007 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.125246+0000", 2024-02-05T04:53:29.007 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.6899999999999998e-07 2024-02-05T04:53:29.007 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.007 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.007 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:29.007 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.125252+0000", 2024-02-05T04:53:29.007 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.7180000000000001e-06 2024-02-05T04:53:29.008 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.008 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.008 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:29.008 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.125262+0000", 2024-02-05T04:53:29.008 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.0022e-05 2024-02-05T04:53:29.008 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.008 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.008 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:29.008 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.125270+0000", 2024-02-05T04:53:29.008 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.5709999999999998e-06 2024-02-05T04:53:29.008 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.008 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.008 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:29.008 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.125494+0000", 2024-02-05T04:53:29.009 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.000223935 2024-02-05T04:53:29.009 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.009 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.009 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:29.009 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.125600+0000", 2024-02-05T04:53:29.009 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.000105522 2024-02-05T04:53:29.009 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:29.009 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:29.009 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:29.009 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.009 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.009 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpRead(3.4s2 36/31 ECSubRead(tid=21, to_read={3:25248d28:::smithi19639362-18:head=0,73728,0}, subchunks={3:25248d28:::smithi19639362-18:head=[0,1]}, attrs_to_read=))", 2024-02-05T04:53:29.009 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:28.127285+0000", 2024-02-05T04:53:29.009 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 0.87118065, 2024-02-05T04:53:29.009 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.000200628, 2024-02-05T04:53:29.010 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:29.010 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:29.010 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:29.010 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.010 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:29.010 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.127285+0000", 2024-02-05T04:53:29.010 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:29.010 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.010 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.010 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:29.010 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.127285+0000", 2024-02-05T04:53:29.010 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:29.010 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.010 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.010 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:29.011 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.127286+0000", 2024-02-05T04:53:29.011 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.2699999999999998e-07 2024-02-05T04:53:29.011 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.011 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.011 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:29.011 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.127295+0000", 2024-02-05T04:53:29.011 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.0040000000000005e-06 2024-02-05T04:53:29.011 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.011 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.011 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:29.011 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.127308+0000", 2024-02-05T04:53:29.011 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.3188e-05 2024-02-05T04:53:29.011 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.011 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.011 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:29.012 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.127316+0000", 2024-02-05T04:53:29.012 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.6219999999999998e-06 2024-02-05T04:53:29.012 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.012 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.012 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:29.012 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.127386+0000", 2024-02-05T04:53:29.012 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.9845000000000006e-05 2024-02-05T04:53:29.012 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.012 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.012 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:29.012 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.127485+0000", 2024-02-05T04:53:29.012 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.9141999999999994e-05 2024-02-05T04:53:29.012 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:29.013 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:29.013 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:29.013 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:29.013 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:29.013 INFO:teuthology.orchestra.run.smithi091.stdout:} 2024-02-05T04:53:29.014 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-02-05T04:53:29.028 INFO:tasks.ceph.osd.4.smithi062.stderr:2024-02-05T04:53:29.027+0000 7f708c308640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34689) UID: 0 2024-02-05T04:53:29.123 INFO:teuthology.orchestra.run.smithi062.stderr:reweighted osd.8 to 1 (10000) 2024-02-05T04:53:29.126 INFO:tasks.rados.rados.0.smithi196.stdout:76: done (1 left) 2024-02-05T04:53:29.126 INFO:tasks.rados.rados.0.smithi196.stdout:append oid 16 current snap is 2 2024-02-05T04:53:29.126 INFO:tasks.rados.rados.0.smithi196.stdout:77: seq_num 60 ranges {262144=638976,901120=376832} 2024-02-05T04:53:29.129 INFO:tasks.ceph.osd.15.smithi196.stderr:2024-02-05T04:53:29.128+0000 7f1facc45640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 34611) UID: 0 2024-02-05T04:53:29.131 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-02-05T04:53:29.132 INFO:tasks.rados.rados.0.smithi196.stdout:77: writing smithi19639362-16 from 262144 to 901120 tid 1 2024-02-05T04:53:29.135 INFO:tasks.rados.rados.0.smithi196.stdout:77: writing smithi19639362-16 from 901120 to 1277952 tid 2 2024-02-05T04:53:29.135 INFO:tasks.rados.rados.0.smithi196.stdout:78: read oid 4 snap 1 2024-02-05T04:53:29.135 INFO:tasks.rados.rados.0.smithi196.stdout:78: expect (ObjNum 3 snap 0 seq_num 3) 2024-02-05T04:53:29.135 INFO:tasks.rados.rados.0.smithi196.stdout:79: copy_from oid 25 from oid 39 current snap is 2 2024-02-05T04:53:29.135 INFO:tasks.rados.rados.0.smithi196.stdout:75: expect (ObjNum 17 snap 0 seq_num 17) 2024-02-05T04:53:29.146 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 9 1 2024-02-05T04:53:29.147 INFO:tasks.rados.rados.0.smithi196.stdout:75: done (3 left) 2024-02-05T04:53:29.147 INFO:tasks.rados.rados.0.smithi196.stdout:80: rollback oid 11 current snap is 2 2024-02-05T04:53:29.147 INFO:tasks.rados.rados.0.smithi196.stdout:rollback oid 11 to 1 2024-02-05T04:53:29.147 INFO:tasks.rados.rados.0.smithi196.stdout:77: finishing write tid 1 to smithi19639362-16 2024-02-05T04:53:29.147 INFO:tasks.rados.rados.0.smithi196.stdout:77: finishing write tid 2 to smithi19639362-16 2024-02-05T04:53:29.147 INFO:tasks.rados.rados.0.smithi196.stdout:77: finishing write tid 3 to smithi19639362-16 2024-02-05T04:53:29.147 INFO:tasks.rados.rados.0.smithi196.stdout:78: expect (ObjNum 3 snap 0 seq_num 3) 2024-02-05T04:53:29.150 INFO:tasks.rados.rados.0.smithi196.stdout:77: finishing write tid 4 to smithi19639362-16 2024-02-05T04:53:29.150 INFO:tasks.rados.rados.0.smithi196.stdout:77: oid 16 updating version 0 to 14 2024-02-05T04:53:29.150 INFO:tasks.rados.rados.0.smithi196.stdout:77: oid 16 version 14 is already newer than 12 2024-02-05T04:53:29.150 INFO:tasks.rados.rados.0.smithi196.stdout:77: oid 16 version 14 is already newer than 13 2024-02-05T04:53:29.150 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 16 v 14 (ObjNum 60 snap 2 seq_num 60) dirty exists 2024-02-05T04:53:29.150 INFO:tasks.rados.rados.0.smithi196.stdout:77: left oid 16 (ObjNum 60 snap 2 seq_num 60) 2024-02-05T04:53:29.150 INFO:tasks.rados.rados.0.smithi196.stdout:79: finishing copy_from racing read to smithi19639362-25 2024-02-05T04:53:29.150 INFO:tasks.rados.rados.0.smithi196.stdout:77: done (3 left) 2024-02-05T04:53:29.150 INFO:tasks.rados.rados.0.smithi196.stdout:78: done (2 left) 2024-02-05T04:53:29.151 INFO:tasks.rados.rados.0.smithi196.stdout:81: read oid 41 snap -1 2024-02-05T04:53:29.151 INFO:tasks.rados.rados.0.smithi196.stdout:81: expect (ObjNum 40 snap 0 seq_num 40) 2024-02-05T04:53:29.151 INFO:tasks.rados.rados.0.smithi196.stdout:82: read oid 24 snap 1 2024-02-05T04:53:29.151 INFO:tasks.rados.rados.0.smithi196.stdout:82: expect (ObjNum 23 snap 0 seq_num 23) 2024-02-05T04:53:29.151 INFO:tasks.rados.rados.0.smithi196.stdout:83: read oid 2 snap -1 2024-02-05T04:53:29.151 INFO:tasks.rados.rados.0.smithi196.stdout:83: expect (ObjNum 1 snap 0 seq_num 1) 2024-02-05T04:53:29.151 INFO:tasks.rados.rados.0.smithi196.stdout:append oid 9 current snap is 2 2024-02-05T04:53:29.151 INFO:tasks.rados.rados.0.smithi196.stdout:84: seq_num 61 ranges {147456=655360,802816=376832} 2024-02-05T04:53:29.154 INFO:tasks.rados.rados.0.smithi196.stdout:84: writing smithi19639362-9 from 147456 to 802816 tid 1 2024-02-05T04:53:29.157 INFO:tasks.rados.rados.0.smithi196.stdout:84: writing smithi19639362-9 from 802816 to 1179648 tid 2 2024-02-05T04:53:29.157 INFO:tasks.rados.rados.0.smithi196.stdout:85: setattr oid 28 current snap is 2 2024-02-05T04:53:29.157 INFO:tasks.rados.rados.0.smithi196.stdout:79: finishing copy_from to smithi19639362-25 2024-02-05T04:53:29.158 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 25 v 6 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-02-05T04:53:29.158 INFO:tasks.rados.rados.0.smithi196.stdout:80: finishing rollback tid 0 to smithi19639362-11 2024-02-05T04:53:29.158 INFO:tasks.rados.rados.0.smithi196.stdout:82: expect (ObjNum 23 snap 0 seq_num 23) 2024-02-05T04:53:29.168 INFO:tasks.rados.rados.0.smithi196.stdout:81: expect (ObjNum 40 snap 0 seq_num 40) 2024-02-05T04:53:29.173 INFO:teuthology.orchestra.run.smithi091.stdout:{ 2024-02-05T04:53:29.173 INFO:teuthology.orchestra.run.smithi091.stdout: "ops": [], 2024-02-05T04:53:29.173 INFO:teuthology.orchestra.run.smithi091.stdout: "num_ops": 0 2024-02-05T04:53:29.173 INFO:teuthology.orchestra.run.smithi091.stdout:} 2024-02-05T04:53:29.182 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-02-05T04:53:29.185 INFO:tasks.rados.rados.0.smithi196.stdout:83: expect (ObjNum 1 snap 0 seq_num 1) 2024-02-05T04:53:29.187 INFO:tasks.rados.rados.0.smithi196.stdout:80: finishing rollback tid 1 to smithi19639362-11 2024-02-05T04:53:29.187 INFO:tasks.rados.rados.0.smithi196.stdout:80: finishing rollback tid 2 to smithi19639362-11 2024-02-05T04:53:29.187 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 11 v 8 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-02-05T04:53:29.187 INFO:tasks.rados.rados.0.smithi196.stdout:84: finishing write tid 1 to smithi19639362-9 2024-02-05T04:53:29.187 INFO:tasks.rados.rados.0.smithi196.stdout:84: finishing write tid 2 to smithi19639362-9 2024-02-05T04:53:29.187 INFO:tasks.rados.rados.0.smithi196.stdout:84: finishing write tid 3 to smithi19639362-9 2024-02-05T04:53:29.187 INFO:tasks.rados.rados.0.smithi196.stdout:84: finishing write tid 4 to smithi19639362-9 2024-02-05T04:53:29.187 INFO:tasks.rados.rados.0.smithi196.stdout:84: oid 9 updating version 0 to 12 2024-02-05T04:53:29.187 INFO:tasks.rados.rados.0.smithi196.stdout:84: oid 9 updating version 12 to 14 2024-02-05T04:53:29.187 INFO:tasks.rados.rados.0.smithi196.stdout:84: oid 9 version 14 is already newer than 13 2024-02-05T04:53:29.187 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 9 v 14 (ObjNum 61 snap 2 seq_num 61) dirty exists 2024-02-05T04:53:29.188 INFO:tasks.rados.rados.0.smithi196.stdout:84: left oid 9 (ObjNum 61 snap 2 seq_num 61) 2024-02-05T04:53:29.188 INFO:tasks.rados.rados.0.smithi196.stdout:79: done (6 left) 2024-02-05T04:53:29.188 INFO:tasks.rados.rados.0.smithi196.stdout:80: done (5 left) 2024-02-05T04:53:29.188 INFO:tasks.rados.rados.0.smithi196.stdout:81: done (4 left) 2024-02-05T04:53:29.188 INFO:tasks.rados.rados.0.smithi196.stdout:82: done (3 left) 2024-02-05T04:53:29.188 INFO:tasks.rados.rados.0.smithi196.stdout:83: done (2 left) 2024-02-05T04:53:29.188 INFO:tasks.rados.rados.0.smithi196.stdout:84: done (1 left) 2024-02-05T04:53:29.188 INFO:tasks.rados.rados.0.smithi196.stdout:86: rollback oid 29 current snap is 2 2024-02-05T04:53:29.188 INFO:tasks.rados.rados.0.smithi196.stdout:rollback oid 29 to 1 2024-02-05T04:53:29.188 INFO:tasks.rados.rados.0.smithi196.stdout:87: copy_from oid 8 from oid 44 current snap is 2 2024-02-05T04:53:29.188 INFO:tasks.rados.rados.0.smithi196.stdout:88: delete oid 46 current snap is 2 2024-02-05T04:53:29.191 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 28 v 6 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-02-05T04:53:29.191 INFO:tasks.rados.rados.0.smithi196.stdout:86: finishing rollback tid 0 to smithi19639362-29 2024-02-05T04:53:29.193 INFO:tasks.rados.rados.0.smithi196.stdout:86: finishing rollback tid 1 to smithi19639362-29 2024-02-05T04:53:29.194 INFO:tasks.rados.rados.0.smithi196.stdout:87: finishing copy_from racing read to smithi19639362-8 2024-02-05T04:53:29.195 INFO:tasks.rados.rados.0.smithi196.stdout:86: finishing rollback tid 2 to smithi19639362-29 2024-02-05T04:53:29.195 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 29 v 14 (ObjNum 56 snap 0 seq_num 56) dirty exists 2024-02-05T04:53:29.198 INFO:tasks.rados.rados.0.smithi196.stdout:85: done (3 left) 2024-02-05T04:53:29.199 INFO:tasks.rados.rados.0.smithi196.stdout:86: done (2 left) 2024-02-05T04:53:29.199 INFO:tasks.rados.rados.0.smithi196.stdout:88: done (1 left) 2024-02-05T04:53:29.199 INFO:tasks.rados.rados.0.smithi196.stdout:89: rollback oid 13 current snap is 2 2024-02-05T04:53:29.199 INFO:tasks.rados.rados.0.smithi196.stdout:rollback oid 13 to 1 2024-02-05T04:53:29.199 INFO:tasks.rados.rados.0.smithi196.stdout:87: finishing copy_from to smithi19639362-8 2024-02-05T04:53:29.199 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 8 v 6 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-02-05T04:53:29.199 INFO:tasks.rados.rados.0.smithi196.stdout:87: done (1 left) 2024-02-05T04:53:29.199 INFO:tasks.rados.rados.0.smithi196.stdout:append oid 23 current snap is 2 2024-02-05T04:53:29.199 INFO:tasks.rados.rados.0.smithi196.stdout:90: seq_num 63 ranges {376832=688128,1064960=376832} 2024-02-05T04:53:29.204 INFO:tasks.rados.rados.0.smithi196.stdout:90: writing smithi19639362-23 from 376832 to 1064960 tid 1 2024-02-05T04:53:29.207 INFO:tasks.rados.rados.0.smithi196.stdout:90: writing smithi19639362-23 from 1064960 to 1441792 tid 2 2024-02-05T04:53:29.207 INFO:tasks.rados.rados.0.smithi196.stdout:91: delete oid 3 current snap is 2 2024-02-05T04:53:29.207 INFO:tasks.rados.rados.0.smithi196.stdout:89: finishing rollback tid 0 to smithi19639362-13 2024-02-05T04:53:29.209 INFO:tasks.rados.rados.0.smithi196.stdout:89: finishing rollback tid 1 to smithi19639362-13 2024-02-05T04:53:29.210 INFO:tasks.rados.rados.0.smithi196.stdout:89: finishing rollback tid 2 to smithi19639362-13 2024-02-05T04:53:29.210 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 13 v 12 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-02-05T04:53:29.212 INFO:tasks.rados.rados.0.smithi196.stdout:90: finishing write tid 1 to smithi19639362-23 2024-02-05T04:53:29.213 INFO:tasks.rados.rados.0.smithi196.stdout:90: finishing write tid 2 to smithi19639362-23 2024-02-05T04:53:29.213 INFO:tasks.rados.rados.0.smithi196.stdout:90: finishing write tid 3 to smithi19639362-23 2024-02-05T04:53:29.214 INFO:tasks.rados.rados.0.smithi196.stdout:90: finishing write tid 4 to smithi19639362-23 2024-02-05T04:53:29.214 INFO:tasks.rados.rados.0.smithi196.stdout:90: oid 23 updating version 0 to 14 2024-02-05T04:53:29.214 INFO:tasks.rados.rados.0.smithi196.stdout:90: oid 23 updating version 14 to 15 2024-02-05T04:53:29.214 INFO:tasks.rados.rados.0.smithi196.stdout:90: oid 23 updating version 15 to 16 2024-02-05T04:53:29.215 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 23 v 16 (ObjNum 63 snap 2 seq_num 63) dirty exists 2024-02-05T04:53:29.215 INFO:tasks.rados.rados.0.smithi196.stdout:90: left oid 23 (ObjNum 63 snap 2 seq_num 63) 2024-02-05T04:53:29.216 INFO:tasks.rados.rados.0.smithi196.stdout:89: done (2 left) 2024-02-05T04:53:29.216 INFO:tasks.rados.rados.0.smithi196.stdout:90: done (1 left) 2024-02-05T04:53:29.216 INFO:tasks.rados.rados.0.smithi196.stdout:91: done (0 left) 2024-02-05T04:53:29.216 INFO:tasks.rados.rados.0.smithi196.stdout:92: snap_remove snap 1 2024-02-05T04:53:29.230 INFO:tasks.ceph.osd.10.smithi150.stderr:2024-02-05T04:53:29.229+0000 7f3075a4d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 34018) UID: 0 2024-02-05T04:53:29.290 INFO:teuthology.orchestra.run.smithi091.stdout:{ 2024-02-05T04:53:29.290 INFO:teuthology.orchestra.run.smithi091.stdout: "ops": [], 2024-02-05T04:53:29.290 INFO:teuthology.orchestra.run.smithi091.stdout: "complaint_time": 30, 2024-02-05T04:53:29.291 INFO:teuthology.orchestra.run.smithi091.stdout: "num_blocked_ops": 0 2024-02-05T04:53:29.291 INFO:teuthology.orchestra.run.smithi091.stdout:} 2024-02-05T04:53:29.299 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-02-05T04:53:29.331 INFO:tasks.ceph.osd.12.smithi062.stderr:2024-02-05T04:53:29.330+0000 7f5e5847e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 34695) UID: 0 2024-02-05T04:53:29.405 INFO:teuthology.orchestra.run.smithi091.stdout:{ 2024-02-05T04:53:29.405 INFO:teuthology.orchestra.run.smithi091.stdout: "size": 20, 2024-02-05T04:53:29.406 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 600, 2024-02-05T04:53:29.406 INFO:teuthology.orchestra.run.smithi091.stdout: "ops": [ 2024-02-05T04:53:29.406 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.406 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.ds4 37/31 ECSubWrite(tid=23, reqid=client.4634.0:225, at_version=37'14, trim_to=0'0, roll_forward_to=34'12))", 2024-02-05T04:53:29.406 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:29.146542+0000", 2024-02-05T04:53:29.407 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 0.25815648200000002, 2024-02-05T04:53:29.407 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0049434240000000001, 2024-02-05T04:53:29.407 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:29.407 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:29.407 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:29.407 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.408 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:29.408 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.146542+0000", 2024-02-05T04:53:29.408 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:29.408 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.408 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.409 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:29.409 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.146542+0000", 2024-02-05T04:53:29.409 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:29.409 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.409 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.409 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:29.410 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.146542+0000", 2024-02-05T04:53:29.410 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.1199999999999999e-07 2024-02-05T04:53:29.410 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.410 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.410 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:29.410 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.147100+0000", 2024-02-05T04:53:29.411 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00055791499999999997 2024-02-05T04:53:29.411 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.411 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.411 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:29.411 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.147111+0000", 2024-02-05T04:53:29.411 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.1038e-05 2024-02-05T04:53:29.413 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.413 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.413 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:29.413 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.147119+0000", 2024-02-05T04:53:29.413 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.3389999999999993e-06 2024-02-05T04:53:29.413 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.413 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.413 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:29.413 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.147173+0000", 2024-02-05T04:53:29.413 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.3823999999999998e-05 2024-02-05T04:53:29.413 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.413 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.414 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:29.414 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.147180+0000", 2024-02-05T04:53:29.414 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.3899999999999998e-06 2024-02-05T04:53:29.414 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.414 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.414 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:29.414 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.151438+0000", 2024-02-05T04:53:29.414 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0042581229999999999 2024-02-05T04:53:29.414 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.414 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.414 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:29.414 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.151485+0000", 2024-02-05T04:53:29.414 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.7482999999999999e-05 2024-02-05T04:53:29.414 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:29.414 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:29.415 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:29.415 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.415 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.415 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.ds4 37/31 ECSubWrite(tid=24, reqid=client.4634.0:225, at_version=0'0, trim_to=0'0, roll_forward_to=37'14))", 2024-02-05T04:53:29.415 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:29.152884+0000", 2024-02-05T04:53:29.415 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 0.25181382099999999, 2024-02-05T04:53:29.415 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0041159070000000002, 2024-02-05T04:53:29.415 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:29.415 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:29.415 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:29.415 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.415 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:29.415 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.152884+0000", 2024-02-05T04:53:29.415 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:29.415 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.416 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.416 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:29.416 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.152884+0000", 2024-02-05T04:53:29.416 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:29.416 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.416 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.416 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:29.416 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.152885+0000", 2024-02-05T04:53:29.416 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.3200000000000004e-07 2024-02-05T04:53:29.416 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.416 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.416 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:29.416 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.152896+0000", 2024-02-05T04:53:29.417 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.0485999999999999e-05 2024-02-05T04:53:29.417 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.417 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.417 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:29.417 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.152910+0000", 2024-02-05T04:53:29.417 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.3862e-05 2024-02-05T04:53:29.417 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.417 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.418 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:29.418 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.152918+0000", 2024-02-05T04:53:29.418 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.0709999999999995e-06 2024-02-05T04:53:29.418 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.418 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.418 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:29.418 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.152983+0000", 2024-02-05T04:53:29.418 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.5445999999999996e-05 2024-02-05T04:53:29.418 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.418 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.418 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:29.419 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.152993+0000", 2024-02-05T04:53:29.419 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.6590000000000002e-06 2024-02-05T04:53:29.419 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.419 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.419 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:29.419 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.156968+0000", 2024-02-05T04:53:29.419 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0039752030000000001 2024-02-05T04:53:29.419 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.419 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.419 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:29.419 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.157000+0000", 2024-02-05T04:53:29.419 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.2348000000000003e-05 2024-02-05T04:53:29.419 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:29.419 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:29.420 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:29.420 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.420 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.420 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.4s4 37/31 ECSubWrite(tid=22, reqid=client.4634.0:239, at_version=37'12, trim_to=0'0, roll_forward_to=34'10))", 2024-02-05T04:53:29.420 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:29.157627+0000", 2024-02-05T04:53:29.420 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 0.24707097, 2024-02-05T04:53:29.420 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0038875250000000002, 2024-02-05T04:53:29.420 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:29.420 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:29.420 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:29.420 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.420 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:29.420 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.157627+0000", 2024-02-05T04:53:29.420 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:29.420 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.420 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.421 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:29.421 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.157627+0000", 2024-02-05T04:53:29.421 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:29.421 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.421 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.421 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:29.421 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.157627+0000", 2024-02-05T04:53:29.422 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.48e-07 2024-02-05T04:53:29.422 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.422 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.422 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:29.422 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.157791+0000", 2024-02-05T04:53:29.422 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00016373700000000001 2024-02-05T04:53:29.422 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.422 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.422 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:29.422 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.157802+0000", 2024-02-05T04:53:29.422 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.0591000000000001e-05 2024-02-05T04:53:29.422 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.422 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.422 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:29.423 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.157810+0000", 2024-02-05T04:53:29.423 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.8730000000000007e-06 2024-02-05T04:53:29.423 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.423 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.423 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:29.423 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.157880+0000", 2024-02-05T04:53:29.423 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.0251000000000002e-05 2024-02-05T04:53:29.423 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.423 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.424 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:29.424 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.157890+0000", 2024-02-05T04:53:29.424 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.0111e-05 2024-02-05T04:53:29.424 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.424 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.424 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:29.424 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.161485+0000", 2024-02-05T04:53:29.424 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0035949609999999998 2024-02-05T04:53:29.424 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.424 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.424 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:29.424 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.161515+0000", 2024-02-05T04:53:29.424 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.9753000000000001e-05 2024-02-05T04:53:29.424 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:29.424 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:29.424 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:29.425 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.425 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.425 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.4s4 37/31 ECSubWrite(tid=23, reqid=client.4634.0:240, at_version=37'13, trim_to=0'0, roll_forward_to=34'10))", 2024-02-05T04:53:29.425 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:29.159160+0000", 2024-02-05T04:53:29.425 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 0.24553854, 2024-02-05T04:53:29.425 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0027387750000000002, 2024-02-05T04:53:29.425 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:29.425 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:29.425 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:29.425 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.425 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:29.425 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.159160+0000", 2024-02-05T04:53:29.425 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:29.425 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.426 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.426 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:29.426 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.159160+0000", 2024-02-05T04:53:29.426 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:29.426 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.426 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.426 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:29.426 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.159160+0000", 2024-02-05T04:53:29.426 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.3099999999999997e-07 2024-02-05T04:53:29.426 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.426 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.426 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:29.426 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.159216+0000", 2024-02-05T04:53:29.426 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.5705999999999998e-05 2024-02-05T04:53:29.426 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.427 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.427 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:29.427 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.159230+0000", 2024-02-05T04:53:29.427 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.3961999999999999e-05 2024-02-05T04:53:29.427 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.427 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.427 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:29.427 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.159242+0000", 2024-02-05T04:53:29.427 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.1532000000000001e-05 2024-02-05T04:53:29.427 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.427 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.427 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:29.427 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.159308+0000", 2024-02-05T04:53:29.427 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.6866000000000007e-05 2024-02-05T04:53:29.427 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.428 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.428 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:29.428 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.159317+0000", 2024-02-05T04:53:29.429 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.9979999999999992e-06 2024-02-05T04:53:29.429 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.429 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.429 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:29.429 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.161868+0000", 2024-02-05T04:53:29.429 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0025511150000000001 2024-02-05T04:53:29.429 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.429 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.429 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:29.429 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.161898+0000", 2024-02-05T04:53:29.429 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.9964999999999999e-05 2024-02-05T04:53:29.429 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:29.430 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:29.430 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:29.430 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.430 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.430 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.4s4 37/31 ECSubWrite(tid=24, reqid=client.4634.0:241, at_version=37'14, trim_to=0'0, roll_forward_to=34'10))", 2024-02-05T04:53:29.430 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:29.160285+0000", 2024-02-05T04:53:29.430 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 0.24441317700000001, 2024-02-05T04:53:29.430 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.001650482, 2024-02-05T04:53:29.430 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:29.430 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:29.430 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:29.430 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.430 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:29.430 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.160285+0000", 2024-02-05T04:53:29.430 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:29.431 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.431 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.431 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:29.431 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.160285+0000", 2024-02-05T04:53:29.431 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:29.431 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.431 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.431 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:29.431 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.160286+0000", 2024-02-05T04:53:29.431 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.1699999999999997e-07 2024-02-05T04:53:29.431 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.431 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.431 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:29.431 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.160309+0000", 2024-02-05T04:53:29.431 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.3679000000000001e-05 2024-02-05T04:53:29.432 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.432 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.432 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:29.432 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.160331+0000", 2024-02-05T04:53:29.432 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.1742999999999998e-05 2024-02-05T04:53:29.432 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.432 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.432 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:29.432 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.160339+0000", 2024-02-05T04:53:29.432 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.1310000000000006e-06 2024-02-05T04:53:29.432 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.432 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.432 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:29.432 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.160395+0000", 2024-02-05T04:53:29.433 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.5664000000000001e-05 2024-02-05T04:53:29.433 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.433 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.433 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:29.433 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.160408+0000", 2024-02-05T04:53:29.433 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.3396e-05 2024-02-05T04:53:29.433 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.433 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.433 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:29.433 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.161912+0000", 2024-02-05T04:53:29.433 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0015036909999999999 2024-02-05T04:53:29.433 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.433 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.433 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:29.434 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.161936+0000", 2024-02-05T04:53:29.434 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.3461000000000001e-05 2024-02-05T04:53:29.434 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:29.434 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:29.434 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:29.434 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.434 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.434 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.4s4 37/31 ECSubWrite(tid=25, reqid=client.4634.0:241, at_version=0'0, trim_to=0'0, roll_forward_to=37'14))", 2024-02-05T04:53:29.434 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:29.162525+0000", 2024-02-05T04:53:29.434 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 0.242173201, 2024-02-05T04:53:29.434 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0023241220000000001, 2024-02-05T04:53:29.434 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:29.434 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:29.434 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:29.434 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.435 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:29.435 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.162525+0000", 2024-02-05T04:53:29.435 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:29.435 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.435 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.435 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:29.435 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.162525+0000", 2024-02-05T04:53:29.435 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:29.435 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.435 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.435 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:29.435 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.162526+0000", 2024-02-05T04:53:29.435 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.4000000000000001e-07 2024-02-05T04:53:29.435 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.435 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.436 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:29.436 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.162543+0000", 2024-02-05T04:53:29.436 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.7244e-05 2024-02-05T04:53:29.436 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.436 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.436 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:29.436 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.162556+0000", 2024-02-05T04:53:29.436 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.291e-05 2024-02-05T04:53:29.436 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.436 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.436 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:29.436 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.162574+0000", 2024-02-05T04:53:29.436 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.8141000000000001e-05 2024-02-05T04:53:29.436 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.436 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.437 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:29.437 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.162639+0000", 2024-02-05T04:53:29.437 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.4830000000000001e-05 2024-02-05T04:53:29.437 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.437 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.437 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:29.437 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.162648+0000", 2024-02-05T04:53:29.437 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.0259999999999996e-06 2024-02-05T04:53:29.437 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.437 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.437 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:29.437 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.164823+0000", 2024-02-05T04:53:29.437 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0021748480000000001 2024-02-05T04:53:29.437 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.437 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.437 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:29.438 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.164849+0000", 2024-02-05T04:53:29.438 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.6483e-05 2024-02-05T04:53:29.438 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:29.438 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:29.438 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:29.438 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:29.438 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:29.438 INFO:teuthology.orchestra.run.smithi091.stdout:} 2024-02-05T04:53:29.440 INFO:tasks.ceph.osd.7.smithi196.stderr:2024-02-05T04:53:29.438+0000 7fb281419640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34606) UID: 0 2024-02-05T04:53:29.440 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-02-05T04:53:29.540 INFO:tasks.ceph.osd.13.smithi091.stderr:2024-02-05T04:53:29.540+0000 7f526fd4d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 34281) UID: 0 2024-02-05T04:53:29.546 INFO:teuthology.orchestra.run.smithi091.stdout:{ 2024-02-05T04:53:29.546 INFO:teuthology.orchestra.run.smithi091.stdout: "ops": [], 2024-02-05T04:53:29.546 INFO:teuthology.orchestra.run.smithi091.stdout: "num_ops": 0 2024-02-05T04:53:29.546 INFO:teuthology.orchestra.run.smithi091.stdout:} 2024-02-05T04:53:29.555 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-02-05T04:53:29.641 INFO:tasks.ceph.osd.3.smithi196.stderr:2024-02-05T04:53:29.641+0000 7fc8817c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34605) UID: 0 2024-02-05T04:53:29.700 INFO:teuthology.orchestra.run.smithi091.stdout:{ 2024-02-05T04:53:29.700 INFO:teuthology.orchestra.run.smithi091.stdout: "ops": [], 2024-02-05T04:53:29.700 INFO:teuthology.orchestra.run.smithi091.stdout: "complaint_time": 30, 2024-02-05T04:53:29.700 INFO:teuthology.orchestra.run.smithi091.stdout: "num_blocked_ops": 0 2024-02-05T04:53:29.700 INFO:teuthology.orchestra.run.smithi091.stdout:} 2024-02-05T04:53:29.709 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-02-05T04:53:29.752 INFO:tasks.ceph.osd.1.smithi091.stderr:2024-02-05T04:53:29.752+0000 7ff9dc283640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34275) UID: 0 2024-02-05T04:53:29.817 INFO:teuthology.orchestra.run.smithi091.stdout:{ 2024-02-05T04:53:29.817 INFO:teuthology.orchestra.run.smithi091.stdout: "size": 20, 2024-02-05T04:53:29.817 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 600, 2024-02-05T04:53:29.817 INFO:teuthology.orchestra.run.smithi091.stdout: "ops": [ 2024-02-05T04:53:29.817 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.818 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.ds5 37/31 ECSubWrite(tid=23, reqid=client.4634.0:225, at_version=37'14, trim_to=0'0, roll_forward_to=34'12))", 2024-02-05T04:53:29.818 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:29.146661+0000", 2024-02-05T04:53:29.818 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 0.66791077300000001, 2024-02-05T04:53:29.818 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0047034900000000003, 2024-02-05T04:53:29.818 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:29.818 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:29.819 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:29.819 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.819 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:29.819 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.146661+0000", 2024-02-05T04:53:29.819 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:29.819 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.819 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.819 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:29.819 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.146661+0000", 2024-02-05T04:53:29.819 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:29.819 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.819 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.819 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:29.820 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.146661+0000", 2024-02-05T04:53:29.820 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.2099999999999997e-07 2024-02-05T04:53:29.820 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.820 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.820 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:29.820 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.147143+0000", 2024-02-05T04:53:29.820 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00048204699999999999 2024-02-05T04:53:29.820 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.820 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.821 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:29.821 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.147159+0000", 2024-02-05T04:53:29.821 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.5885000000000001e-05 2024-02-05T04:53:29.821 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.821 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.821 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:29.821 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.147167+0000", 2024-02-05T04:53:29.821 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.5920000000000003e-06 2024-02-05T04:53:29.821 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.821 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.821 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:29.821 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.147223+0000", 2024-02-05T04:53:29.821 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.6140000000000001e-05 2024-02-05T04:53:29.821 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.822 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.822 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:29.822 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.147229+0000", 2024-02-05T04:53:29.822 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.2999999999999998e-06 2024-02-05T04:53:29.822 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.822 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.822 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:29.822 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.151327+0000", 2024-02-05T04:53:29.822 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0040972379999999996 2024-02-05T04:53:29.822 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.822 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.822 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:29.822 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.151364+0000", 2024-02-05T04:53:29.822 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.7766999999999999e-05 2024-02-05T04:53:29.823 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:29.823 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:29.823 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:29.823 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.823 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.823 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:227 3.6s0 3:6dc5620a:::smithi19639362-11:head [append 0~0] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:53:29.823 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:29.147210+0000", 2024-02-05T04:53:29.823 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 0.66736195899999995, 2024-02-05T04:53:29.823 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00706225, 2024-02-05T04:53:29.823 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:29.823 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:29.823 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:53:29.823 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:53:29.823 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:29.823 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 227 2024-02-05T04:53:29.824 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.824 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:29.824 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.824 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:29.824 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.147210+0000", 2024-02-05T04:53:29.824 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:29.824 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.824 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.824 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:29.824 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.147210+0000", 2024-02-05T04:53:29.824 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:29.824 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.825 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.825 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:29.825 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.147211+0000", 2024-02-05T04:53:29.825 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.823e-06 2024-02-05T04:53:29.825 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.825 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.825 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:29.825 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.147216+0000", 2024-02-05T04:53:29.825 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.938e-06 2024-02-05T04:53:29.825 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.825 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.825 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:29.825 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.147230+0000", 2024-02-05T04:53:29.825 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.3303e-05 2024-02-05T04:53:29.826 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.826 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.826 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:29.826 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.147237+0000", 2024-02-05T04:53:29.826 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.8199999999999997e-06 2024-02-05T04:53:29.826 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.826 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.826 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:29.826 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.147294+0000", 2024-02-05T04:53:29.826 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.6653999999999997e-05 2024-02-05T04:53:29.826 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.826 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.826 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:53:29.827 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.147391+0000", 2024-02-05T04:53:29.827 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.6607000000000005e-05 2024-02-05T04:53:29.827 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.827 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.827 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:29.827 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.148109+0000", 2024-02-05T04:53:29.827 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00071871000000000001 2024-02-05T04:53:29.827 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.827 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.827 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:29.827 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.154087+0000", 2024-02-05T04:53:29.827 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.005977675 2024-02-05T04:53:29.827 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.827 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.828 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "commit_sent", 2024-02-05T04:53:29.828 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.154198+0000", 2024-02-05T04:53:29.828 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.000110828 2024-02-05T04:53:29.828 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.828 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.828 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:29.828 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.154272+0000", 2024-02-05T04:53:29.828 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.3892000000000003e-05 2024-02-05T04:53:29.828 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:29.828 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:29.828 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:29.828 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.828 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.828 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:228 3.6s0 3:6dc5620a:::smithi19639362-11:head [rollback 3] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:53:29.829 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:29.147258+0000", 2024-02-05T04:53:29.829 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 0.66731391600000001, 2024-02-05T04:53:29.829 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0097154210000000001, 2024-02-05T04:53:29.829 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:29.829 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:29.829 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:53:29.829 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:53:29.829 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:29.829 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 228 2024-02-05T04:53:29.829 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.829 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:29.829 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.829 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:29.829 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.147258+0000", 2024-02-05T04:53:29.829 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:29.829 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.830 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.830 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:29.830 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.147258+0000", 2024-02-05T04:53:29.830 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:29.830 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.830 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.830 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:29.830 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.147258+0000", 2024-02-05T04:53:29.830 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.3000000000000001e-07 2024-02-05T04:53:29.830 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.830 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.830 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:29.830 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.147261+0000", 2024-02-05T04:53:29.830 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.429e-06 2024-02-05T04:53:29.830 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.831 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.831 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:29.831 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.147268+0000", 2024-02-05T04:53:29.831 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.7109999999999995e-06 2024-02-05T04:53:29.831 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.831 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.831 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:29.831 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.147275+0000", 2024-02-05T04:53:29.831 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.2140000000000001e-06 2024-02-05T04:53:29.831 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.831 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.831 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:29.831 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.148902+0000", 2024-02-05T04:53:29.832 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0016272630000000001 2024-02-05T04:53:29.832 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.832 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.832 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:53:29.832 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.148962+0000", 2024-02-05T04:53:29.832 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.0158999999999998e-05 2024-02-05T04:53:29.832 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.832 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.832 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:29.832 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.149301+0000", 2024-02-05T04:53:29.832 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.000339227 2024-02-05T04:53:29.832 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.833 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.833 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:29.833 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.156821+0000", 2024-02-05T04:53:29.833 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0075194629999999997 2024-02-05T04:53:29.833 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.833 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.833 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "commit_sent", 2024-02-05T04:53:29.833 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.156887+0000", 2024-02-05T04:53:29.833 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.6748000000000005e-05 2024-02-05T04:53:29.833 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.833 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.833 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:29.833 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.156973+0000", 2024-02-05T04:53:29.833 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.5677e-05 2024-02-05T04:53:29.833 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:29.834 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:29.834 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:29.834 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.834 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.834 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:229 3.6s0 3:6dc5620a:::smithi19639362-11:head [append 0~0] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:53:29.834 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:29.147292+0000", 2024-02-05T04:53:29.834 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 0.66727939999999997, 2024-02-05T04:53:29.834 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0099616229999999993, 2024-02-05T04:53:29.834 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:29.834 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:29.834 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:53:29.834 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:53:29.834 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:29.834 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 229 2024-02-05T04:53:29.835 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.835 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:29.835 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.835 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:29.835 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.147292+0000", 2024-02-05T04:53:29.835 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:29.835 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.835 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.835 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:29.835 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.147292+0000", 2024-02-05T04:53:29.835 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:29.835 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.835 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.835 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:29.836 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.147293+0000", 2024-02-05T04:53:29.836 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.1200000000000003e-07 2024-02-05T04:53:29.836 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.836 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.836 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:29.837 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.147295+0000", 2024-02-05T04:53:29.837 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.3309999999999998e-06 2024-02-05T04:53:29.837 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.837 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.837 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:29.837 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.147302+0000", 2024-02-05T04:53:29.837 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.3769999999999998e-06 2024-02-05T04:53:29.837 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.838 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.838 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:29.838 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.147308+0000", 2024-02-05T04:53:29.838 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.8499999999999999e-06 2024-02-05T04:53:29.838 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.838 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.838 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:29.838 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.151615+0000", 2024-02-05T04:53:29.838 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.004306374 2024-02-05T04:53:29.838 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.838 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.838 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:53:29.839 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.151698+0000", 2024-02-05T04:53:29.839 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.3741999999999999e-05 2024-02-05T04:53:29.839 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.839 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.839 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:29.839 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.152107+0000", 2024-02-05T04:53:29.839 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.000408265 2024-02-05T04:53:29.839 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.839 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.839 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:29.839 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.156975+0000", 2024-02-05T04:53:29.839 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0048685459999999996 2024-02-05T04:53:29.839 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.840 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.840 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "commit_sent", 2024-02-05T04:53:29.840 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.157198+0000", 2024-02-05T04:53:29.840 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00022250500000000001 2024-02-05T04:53:29.840 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.840 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.840 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:29.840 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.157254+0000", 2024-02-05T04:53:29.840 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.6121e-05 2024-02-05T04:53:29.840 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:29.840 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:29.840 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:29.840 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.841 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.841 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWriteReply(3.6s0 37/31 ECSubWriteReply(tid=74, last_complete=37'6, committed=1, applied=1))", 2024-02-05T04:53:29.841 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:29.149064+0000", 2024-02-05T04:53:29.841 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 0.66550751399999997, 2024-02-05T04:53:29.841 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.003259765, 2024-02-05T04:53:29.841 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:29.841 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:29.841 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:29.841 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.841 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:29.841 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.149064+0000", 2024-02-05T04:53:29.841 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:29.841 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.842 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.842 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:29.842 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.149064+0000", 2024-02-05T04:53:29.842 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:29.842 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.842 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.842 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:29.842 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.149065+0000", 2024-02-05T04:53:29.842 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.8800000000000002e-07 2024-02-05T04:53:29.842 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.842 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.842 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:29.842 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.149070+0000", 2024-02-05T04:53:29.843 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.062e-06 2024-02-05T04:53:29.843 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.843 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.843 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:29.843 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.149080+0000", 2024-02-05T04:53:29.843 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.0555e-05 2024-02-05T04:53:29.843 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.843 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.843 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:29.843 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.149089+0000", 2024-02-05T04:53:29.843 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.5110000000000003e-06 2024-02-05T04:53:29.844 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.844 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.844 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:29.844 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.152307+0000", 2024-02-05T04:53:29.844 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0032178609999999998 2024-02-05T04:53:29.844 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.844 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.844 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:29.844 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.152324+0000", 2024-02-05T04:53:29.844 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.6988000000000001e-05 2024-02-05T04:53:29.844 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:29.844 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:29.844 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:29.844 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.845 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.845 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWriteReply(3.6s0 34/31 ECSubWriteReply(tid=74, last_complete=37'6, committed=1, applied=1))", 2024-02-05T04:53:29.845 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:29.149230+0000", 2024-02-05T04:53:29.845 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 0.66534200200000004, 2024-02-05T04:53:29.845 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0031319239999999999, 2024-02-05T04:53:29.845 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:29.845 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:29.845 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:29.845 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.845 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:29.845 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.149230+0000", 2024-02-05T04:53:29.845 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:29.846 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.846 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.846 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:29.846 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.149230+0000", 2024-02-05T04:53:29.846 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:29.846 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.846 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.846 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:29.846 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.149230+0000", 2024-02-05T04:53:29.846 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.1999999999999999e-07 2024-02-05T04:53:29.846 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.846 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.846 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:29.846 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.149237+0000", 2024-02-05T04:53:29.846 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.9009999999999997e-06 2024-02-05T04:53:29.847 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.847 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.847 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:29.847 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.149247+0000", 2024-02-05T04:53:29.847 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.6649999999999998e-06 2024-02-05T04:53:29.847 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.847 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.847 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:29.847 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.149253+0000", 2024-02-05T04:53:29.847 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.5969999999999997e-06 2024-02-05T04:53:29.847 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.847 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.847 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:29.847 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.152347+0000", 2024-02-05T04:53:29.848 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0030938580000000001 2024-02-05T04:53:29.848 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.848 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.848 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:29.848 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.152361+0000", 2024-02-05T04:53:29.848 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.4283e-05 2024-02-05T04:53:29.848 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:29.848 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:29.848 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:29.848 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.848 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.848 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:234 3.bs0 3:db93bf34:::smithi19639362-24:3 [read 0~393216,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:53:29.848 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:29.149613+0000", 2024-02-05T04:53:29.849 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 0.66495879499999999, 2024-02-05T04:53:29.849 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0039999320000000003, 2024-02-05T04:53:29.849 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:29.849 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "started", 2024-02-05T04:53:29.849 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:53:29.849 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:53:29.849 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:29.849 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 234 2024-02-05T04:53:29.849 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.849 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:29.849 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.849 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:29.849 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.149613+0000", 2024-02-05T04:53:29.849 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:29.849 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.850 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.850 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:29.850 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.149613+0000", 2024-02-05T04:53:29.850 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:29.850 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.850 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.850 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:29.850 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.149613+0000", 2024-02-05T04:53:29.850 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.7999999999999995e-07 2024-02-05T04:53:29.850 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.850 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.850 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:29.850 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.149617+0000", 2024-02-05T04:53:29.850 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.484e-06 2024-02-05T04:53:29.851 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.851 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.851 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:29.851 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.149624+0000", 2024-02-05T04:53:29.851 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.3799999999999996e-06 2024-02-05T04:53:29.851 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.851 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.851 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:29.851 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.149630+0000", 2024-02-05T04:53:29.851 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.541e-06 2024-02-05T04:53:29.851 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.851 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.851 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:29.851 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.150043+0000", 2024-02-05T04:53:29.851 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.000413361 2024-02-05T04:53:29.852 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.852 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.852 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:53:29.852 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.150125+0000", 2024-02-05T04:53:29.852 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.1794999999999999e-05 2024-02-05T04:53:29.852 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.852 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.852 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:29.852 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.153613+0000", 2024-02-05T04:53:29.852 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0034877910000000001 2024-02-05T04:53:29.852 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:29.852 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:29.852 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:29.853 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.853 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.853 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:235 3.bs0 3:db93bf34:::smithi19639362-24:3 [read 0~393216,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:53:29.853 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:29.149641+0000", 2024-02-05T04:53:29.853 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 0.66493089900000002, 2024-02-05T04:53:29.853 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0050206540000000003, 2024-02-05T04:53:29.853 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:29.853 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "started", 2024-02-05T04:53:29.853 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:53:29.853 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:53:29.853 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:29.853 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 235 2024-02-05T04:53:29.853 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.853 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:29.853 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.854 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:29.854 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.149641+0000", 2024-02-05T04:53:29.854 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:29.854 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.854 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.854 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:29.854 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.149641+0000", 2024-02-05T04:53:29.854 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:29.854 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.854 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.854 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:29.854 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.149641+0000", 2024-02-05T04:53:29.854 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.5999999999999999e-07 2024-02-05T04:53:29.854 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.854 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.855 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:29.855 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.149645+0000", 2024-02-05T04:53:29.855 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.557e-06 2024-02-05T04:53:29.855 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.855 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.855 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:29.855 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.149651+0000", 2024-02-05T04:53:29.855 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.9359999999999996e-06 2024-02-05T04:53:29.855 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.855 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.855 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:29.855 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.149655+0000", 2024-02-05T04:53:29.855 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.9060000000000001e-06 2024-02-05T04:53:29.855 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.855 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.856 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:29.856 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.150294+0000", 2024-02-05T04:53:29.856 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00063865799999999996 2024-02-05T04:53:29.856 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.856 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.856 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:53:29.856 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.150343+0000", 2024-02-05T04:53:29.856 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.8822000000000001e-05 2024-02-05T04:53:29.856 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.856 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.856 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:29.856 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.154661+0000", 2024-02-05T04:53:29.856 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0043184149999999999 2024-02-05T04:53:29.856 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:29.857 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:29.857 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:29.857 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.857 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.857 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:236 3.bs0 3:de65491d:::smithi19639362-2:head [read 0~32768,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:53:29.857 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:29.149662+0000", 2024-02-05T04:53:29.857 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 0.66490933699999999, 2024-02-05T04:53:29.857 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.005495832, 2024-02-05T04:53:29.857 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:29.857 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "started", 2024-02-05T04:53:29.857 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:53:29.857 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:53:29.857 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:29.857 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 236 2024-02-05T04:53:29.858 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.858 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:29.858 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.858 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:29.858 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.149662+0000", 2024-02-05T04:53:29.858 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:29.858 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.858 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.858 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:29.858 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.149662+0000", 2024-02-05T04:53:29.858 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:29.858 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.858 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.858 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:29.858 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.149663+0000", 2024-02-05T04:53:29.858 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.7099999999999997e-07 2024-02-05T04:53:29.859 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.859 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.859 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:29.859 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.149665+0000", 2024-02-05T04:53:29.859 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.5969999999999999e-06 2024-02-05T04:53:29.859 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.859 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.859 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:29.859 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.149671+0000", 2024-02-05T04:53:29.859 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.8100000000000003e-06 2024-02-05T04:53:29.859 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.859 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.859 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:29.859 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.149676+0000", 2024-02-05T04:53:29.859 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.87e-06 2024-02-05T04:53:29.860 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.860 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.860 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:29.860 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.152608+0000", 2024-02-05T04:53:29.860 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0029316479999999998 2024-02-05T04:53:29.860 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.860 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.860 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:53:29.860 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.152684+0000", 2024-02-05T04:53:29.860 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.6178000000000001e-05 2024-02-05T04:53:29.860 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.860 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.860 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:29.860 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.155158+0000", 2024-02-05T04:53:29.860 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0024743579999999999 2024-02-05T04:53:29.861 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:29.861 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:29.861 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:29.861 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.861 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.861 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:237 3.bs0 3:de65491d:::smithi19639362-2:head [read 0~32768,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:53:29.861 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:29.149685+0000", 2024-02-05T04:53:29.861 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 0.66488619999999998, 2024-02-05T04:53:29.861 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0056856859999999997, 2024-02-05T04:53:29.861 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:29.861 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "started", 2024-02-05T04:53:29.861 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:53:29.861 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:53:29.861 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:29.862 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 237 2024-02-05T04:53:29.862 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.862 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:29.862 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.862 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:29.862 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.149685+0000", 2024-02-05T04:53:29.862 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:29.862 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.862 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.862 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:29.862 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.149685+0000", 2024-02-05T04:53:29.862 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:29.862 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.862 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.863 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:29.863 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.149686+0000", 2024-02-05T04:53:29.863 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.6100000000000002e-07 2024-02-05T04:53:29.863 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.863 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.863 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:29.863 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.149694+0000", 2024-02-05T04:53:29.863 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.7399999999999993e-06 2024-02-05T04:53:29.863 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.863 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.863 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:29.863 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.149701+0000", 2024-02-05T04:53:29.863 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.9870000000000002e-06 2024-02-05T04:53:29.863 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.864 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.864 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:29.864 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.149707+0000", 2024-02-05T04:53:29.864 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.2700000000000004e-06 2024-02-05T04:53:29.864 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.864 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.864 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:29.864 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.152903+0000", 2024-02-05T04:53:29.864 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.003196094 2024-02-05T04:53:29.864 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.864 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.864 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:53:29.864 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.152980+0000", 2024-02-05T04:53:29.865 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.7348000000000006e-05 2024-02-05T04:53:29.865 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.865 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.865 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:29.865 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.155371+0000", 2024-02-05T04:53:29.865 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0023908860000000001 2024-02-05T04:53:29.865 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:29.865 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:29.865 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:29.865 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.866 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.866 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:238 3.bs0 3:de65491d:::smithi19639362-2:head [read 0~32768,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:53:29.866 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:29.149714+0000", 2024-02-05T04:53:29.866 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 0.66485749900000002, 2024-02-05T04:53:29.866 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0064755780000000001, 2024-02-05T04:53:29.866 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:29.866 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "started", 2024-02-05T04:53:29.866 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:53:29.866 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:53:29.867 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:29.867 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 238 2024-02-05T04:53:29.867 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.867 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:29.867 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.867 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:29.867 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.149714+0000", 2024-02-05T04:53:29.867 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:29.868 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.868 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.868 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:29.868 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.149714+0000", 2024-02-05T04:53:29.868 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:29.868 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.868 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.869 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:29.869 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.149714+0000", 2024-02-05T04:53:29.869 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.96e-07 2024-02-05T04:53:29.869 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.869 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.869 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:29.869 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.149721+0000", 2024-02-05T04:53:29.869 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.2920000000000001e-06 2024-02-05T04:53:29.869 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.870 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.870 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:29.870 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.149731+0000", 2024-02-05T04:53:29.870 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.0199e-05 2024-02-05T04:53:29.870 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.870 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.870 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:29.870 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.149736+0000", 2024-02-05T04:53:29.871 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.7990000000000001e-06 2024-02-05T04:53:29.871 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.871 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.871 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:29.871 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.155400+0000", 2024-02-05T04:53:29.871 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0056640550000000003 2024-02-05T04:53:29.871 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.871 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.871 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:53:29.872 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.155489+0000", 2024-02-05T04:53:29.872 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.9258000000000005e-05 2024-02-05T04:53:29.872 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.872 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.872 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:29.872 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.156190+0000", 2024-02-05T04:53:29.872 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.000700579 2024-02-05T04:53:29.872 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:29.873 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:29.873 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:29.873 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.873 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.873 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.ds5 37/31 ECSubWrite(tid=24, reqid=client.4634.0:225, at_version=0'0, trim_to=0'0, roll_forward_to=37'14))", 2024-02-05T04:53:29.873 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:29.152759+0000", 2024-02-05T04:53:29.873 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 0.66181242799999995, 2024-02-05T04:53:29.873 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0042624610000000004, 2024-02-05T04:53:29.874 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:29.874 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:29.874 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:29.874 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.874 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:29.874 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.152759+0000", 2024-02-05T04:53:29.874 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:29.874 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.874 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.875 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:29.875 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.152759+0000", 2024-02-05T04:53:29.875 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:29.875 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.875 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.875 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:29.875 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.152760+0000", 2024-02-05T04:53:29.875 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.8999999999999997e-07 2024-02-05T04:53:29.875 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.876 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.876 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:29.876 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.152772+0000", 2024-02-05T04:53:29.876 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.2232000000000001e-05 2024-02-05T04:53:29.876 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.876 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.876 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:29.876 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.152782+0000", 2024-02-05T04:53:29.877 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.6569999999999992e-06 2024-02-05T04:53:29.877 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.877 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.877 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:29.877 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.152789+0000", 2024-02-05T04:53:29.877 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.5700000000000004e-06 2024-02-05T04:53:29.877 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.877 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.877 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:29.878 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.152842+0000", 2024-02-05T04:53:29.878 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.2707000000000002e-05 2024-02-05T04:53:29.878 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.878 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.878 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:29.878 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.152851+0000", 2024-02-05T04:53:29.878 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.0089999999999996e-06 2024-02-05T04:53:29.878 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.879 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.879 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:29.879 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.156991+0000", 2024-02-05T04:53:29.879 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0041398620000000002 2024-02-05T04:53:29.879 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.879 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.879 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:29.879 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.157022+0000", 2024-02-05T04:53:29.879 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.0933999999999999e-05 2024-02-05T04:53:29.880 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:29.880 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:29.880 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:29.880 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.880 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.880 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.4s6 37/31 ECSubWrite(tid=22, reqid=client.4634.0:239, at_version=37'12, trim_to=0'0, roll_forward_to=34'10))", 2024-02-05T04:53:29.880 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:29.157968+0000", 2024-02-05T04:53:29.880 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 0.6566033, 2024-02-05T04:53:29.880 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.003728168, 2024-02-05T04:53:29.881 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:29.881 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:29.881 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:29.881 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.881 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:29.881 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.157968+0000", 2024-02-05T04:53:29.881 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:29.881 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.881 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.881 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:29.881 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.157968+0000", 2024-02-05T04:53:29.881 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:29.881 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.881 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.881 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:29.882 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.157969+0000", 2024-02-05T04:53:29.882 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.5000000000000002e-07 2024-02-05T04:53:29.882 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.882 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.882 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:29.882 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.158146+0000", 2024-02-05T04:53:29.882 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.000177033 2024-02-05T04:53:29.882 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.882 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.882 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:29.882 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.158486+0000", 2024-02-05T04:53:29.882 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00033961899999999999 2024-02-05T04:53:29.882 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.882 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.883 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:29.883 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.158499+0000", 2024-02-05T04:53:29.883 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.3908e-05 2024-02-05T04:53:29.883 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.883 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.883 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:29.883 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.158560+0000", 2024-02-05T04:53:29.883 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.0693999999999999e-05 2024-02-05T04:53:29.883 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.883 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.883 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:29.883 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.158569+0000", 2024-02-05T04:53:29.883 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.1900000000000001e-06 2024-02-05T04:53:29.883 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.883 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.884 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:29.884 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.161661+0000", 2024-02-05T04:53:29.884 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.003091973 2024-02-05T04:53:29.884 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.884 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.884 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:29.884 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.161696+0000", 2024-02-05T04:53:29.884 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.5101000000000002e-05 2024-02-05T04:53:29.884 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:29.884 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:29.884 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:29.884 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.884 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.884 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:243 3.7s0 3:e8f6ce71:::smithi19639362-28:head [create,setxattr YOk\\Qsqnjk`Uh[icZasWhcZnad[h[k[Qmk]TihbR (1823) in=40b,setxattr QnfshqY (34) in=7b,setxattr V_tZOrZWhpPekPNs_toQaraRlW\\sWffpapfPROXb (1614) in=40b,setxattr aiTbV]hXcpXjkS]VQonZZkfogbUSnO[bajbZWdRf (839) in=40b,setxattr pZdlecZrV]qp]`sTkf_bSjZVZjhbScXfb[Sr_bsa (18524) in=40b,setxattr kXtOdjofgYOa_Tck\\nYsbVOfgtldphnQ`]P[j]lX (1295) in=40b,setxattr dRpscRhc^OtlbThWSmXVSNge]s (34) in=26b,setxattr T\\s[QdNhlRZh\\\\[X]NpSccmfRcqRaXScZOZ\\ZR^e (1999) in=40b,setxattr YTcfnqlbrRa^rhctrncU_Wa]laamPmYsaqg_f^N` (1471) in=40b,setxattr l_Qeh\\RcafrRYN_YZf_iiaWagZfe[sop]^\\qTggq (561) in=40b,setxattr NShVOQe\\dRYY[tPettNbkgkTVonNshnbYeePWcla (314) in=40b,setxattr ^iVidVQX]Yn`XTY\\pOXm`_PNX`qeckhbQoR`aQWZ (8841) in=40b,setxattr _jkmO]cXrN_S^act\\QOOjpUZOj_Qf^otnZbpWZPU (981) in=40b,setxattr bNbhPX[YP[lppmc`PkWcpnS`XsQNpW\\UYnWr\\_RX (1927) in=40b,setxattr talTebhpYS[[TXcgodoQR\\tf`fVekppjlkWnTqf[ (1753) in=40b,setxattr rp\\XTgd\\X^qaodkhY[elq^RagRhN_NR^j^NmjOUh (806) in=40b,setxattr ]\\oXSo`PWafUO^dbhR^i`hifQgWjRhR[rRa^bnej (70) in=40b,setxattr UkYrdYaZakpnReakOdc\\^bO[YPqfdkdtbNkghaSQ (279) in=40b,setxattr YtUs_hcRdQqgZV]NaadOflcb\\_c^\\Zg_Y\\]snXXZ (701) in=40b,setxattr NRZdh]WclahTe^_btdqe\\^mVYqQOaXaa\\aPVXWnk (19016) in=40b] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:53:29.885 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:29.186667+0000", 2024-02-05T04:53:29.885 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 0.62790460000000003, 2024-02-05T04:53:29.885 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00443042, 2024-02-05T04:53:29.885 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:29.885 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:29.885 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:53:29.885 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:53:29.885 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:29.885 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 243 2024-02-05T04:53:29.885 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.885 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:29.885 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.885 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:29.885 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.186667+0000", 2024-02-05T04:53:29.886 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:29.886 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.886 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.886 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:29.886 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.186667+0000", 2024-02-05T04:53:29.886 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:29.886 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.886 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.886 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:29.886 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.186669+0000", 2024-02-05T04:53:29.886 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.1490000000000001e-06 2024-02-05T04:53:29.886 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.886 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.886 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:29.886 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.186785+0000", 2024-02-05T04:53:29.887 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00011583599999999999 2024-02-05T04:53:29.887 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.887 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.887 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:29.887 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.186805+0000", 2024-02-05T04:53:29.887 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.0182000000000001e-05 2024-02-05T04:53:29.887 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.887 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.887 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:29.887 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.186815+0000", 2024-02-05T04:53:29.887 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.9920000000000002e-06 2024-02-05T04:53:29.887 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.887 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.887 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:29.887 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.186884+0000", 2024-02-05T04:53:29.888 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.8447000000000004e-05 2024-02-05T04:53:29.888 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.888 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.888 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:53:29.888 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.186976+0000", 2024-02-05T04:53:29.888 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.2427000000000001e-05 2024-02-05T04:53:29.888 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.888 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.888 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:29.888 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.187672+0000", 2024-02-05T04:53:29.888 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00069570400000000001 2024-02-05T04:53:29.888 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.888 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.888 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:29.889 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.189361+0000", 2024-02-05T04:53:29.889 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.001689103 2024-02-05T04:53:29.889 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.889 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.889 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "commit_sent", 2024-02-05T04:53:29.889 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.191006+0000", 2024-02-05T04:53:29.889 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.001644951 2024-02-05T04:53:29.889 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.889 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.889 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:29.889 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.191097+0000", 2024-02-05T04:53:29.889 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.1629e-05 2024-02-05T04:53:29.889 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:29.889 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:29.890 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:29.890 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.890 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.890 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:247 3.1s0 3:8451ec0c:::smithi19639362-8:head [copy-from ver 6] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:53:29.890 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:29.186841+0000", 2024-02-05T04:53:29.890 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 0.62773069299999995, 2024-02-05T04:53:29.890 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.011720046, 2024-02-05T04:53:29.890 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:29.890 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:29.890 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:53:29.890 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:53:29.890 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:29.890 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 247 2024-02-05T04:53:29.890 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.890 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:29.891 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.891 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:29.891 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.186841+0000", 2024-02-05T04:53:29.891 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:29.891 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.891 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.891 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:29.891 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.186841+0000", 2024-02-05T04:53:29.891 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:29.891 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.891 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.891 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:29.891 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.186841+0000", 2024-02-05T04:53:29.891 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.0900000000000002e-07 2024-02-05T04:53:29.891 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.892 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.892 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:29.892 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.186844+0000", 2024-02-05T04:53:29.892 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.0970000000000002e-06 2024-02-05T04:53:29.892 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.892 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.892 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:29.892 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.186849+0000", 2024-02-05T04:53:29.892 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.9150000000000004e-06 2024-02-05T04:53:29.892 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.892 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.892 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:29.892 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.186853+0000", 2024-02-05T04:53:29.893 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.41e-06 2024-02-05T04:53:29.893 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.893 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.893 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:29.893 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.186906+0000", 2024-02-05T04:53:29.893 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.3505999999999999e-05 2024-02-05T04:53:29.893 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.893 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.893 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:53:29.893 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.187011+0000", 2024-02-05T04:53:29.893 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.000104478 2024-02-05T04:53:29.893 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.893 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.893 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:29.894 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.192835+0000", 2024-02-05T04:53:29.894 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0058247689999999996 2024-02-05T04:53:29.894 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.894 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.894 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:29.894 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.198154+0000", 2024-02-05T04:53:29.894 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0053189470000000001 2024-02-05T04:53:29.894 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.894 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.894 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "commit_sent", 2024-02-05T04:53:29.894 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.198508+0000", 2024-02-05T04:53:29.895 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.000353333 2024-02-05T04:53:29.895 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.895 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.895 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:29.895 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.198561+0000", 2024-02-05T04:53:29.895 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.3182e-05 2024-02-05T04:53:29.895 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:29.895 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:29.895 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:29.895 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.895 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.895 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:248 3.1s0 3:8451ec0c:::smithi19639362-8:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:53:29.895 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:29.186861+0000", 2024-02-05T04:53:29.896 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 0.627710137, 2024-02-05T04:53:29.896 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0075015220000000001, 2024-02-05T04:53:29.896 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:29.896 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "started", 2024-02-05T04:53:29.896 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:53:29.896 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:53:29.896 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:29.896 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 248 2024-02-05T04:53:29.896 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.896 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:29.896 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.896 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:29.896 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.186861+0000", 2024-02-05T04:53:29.897 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:29.897 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.897 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.897 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:29.897 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.186861+0000", 2024-02-05T04:53:29.897 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:29.897 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.897 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.897 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:29.897 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.186862+0000", 2024-02-05T04:53:29.897 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.6199999999999999e-07 2024-02-05T04:53:29.897 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.897 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.897 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:29.897 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.186863+0000", 2024-02-05T04:53:29.898 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.378e-06 2024-02-05T04:53:29.898 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.898 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.898 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:29.898 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.186869+0000", 2024-02-05T04:53:29.898 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.4990000000000002e-06 2024-02-05T04:53:29.898 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.898 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.898 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:29.898 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.186873+0000", 2024-02-05T04:53:29.898 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.6550000000000003e-06 2024-02-05T04:53:29.898 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.898 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.899 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:29.899 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.187178+0000", 2024-02-05T04:53:29.899 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00030503099999999999 2024-02-05T04:53:29.899 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.899 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.899 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "waiting for blocked object", 2024-02-05T04:53:29.899 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.187222+0000", 2024-02-05T04:53:29.899 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.3726000000000001e-05 2024-02-05T04:53:29.899 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.899 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.899 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:29.899 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.194173+0000", 2024-02-05T04:53:29.899 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0069509139999999999 2024-02-05T04:53:29.899 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.900 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.900 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:53:29.900 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.194269+0000", 2024-02-05T04:53:29.900 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.6391999999999995e-05 2024-02-05T04:53:29.900 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.900 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.900 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:29.900 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.194363+0000", 2024-02-05T04:53:29.900 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.3665000000000005e-05 2024-02-05T04:53:29.900 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:29.900 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:29.900 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:29.900 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.900 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.901 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.es4 37/31 ECSubWrite(tid=20, reqid=client.4634.0:245, at_version=37'13, trim_to=0'0, roll_forward_to=34'10))", 2024-02-05T04:53:29.901 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:29.189739+0000", 2024-02-05T04:53:29.901 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 0.62483258100000005, 2024-02-05T04:53:29.901 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0034766160000000001, 2024-02-05T04:53:29.901 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:29.901 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:29.901 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:29.901 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.901 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:29.901 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.189739+0000", 2024-02-05T04:53:29.901 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:29.901 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.901 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.901 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:29.902 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.189739+0000", 2024-02-05T04:53:29.902 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:29.902 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.902 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.902 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:29.902 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.189740+0000", 2024-02-05T04:53:29.902 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.6499999999999999e-07 2024-02-05T04:53:29.902 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.902 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.902 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:29.902 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.189756+0000", 2024-02-05T04:53:29.902 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.6317e-05 2024-02-05T04:53:29.902 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.902 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.902 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:29.903 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.189767+0000", 2024-02-05T04:53:29.903 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.1318000000000001e-05 2024-02-05T04:53:29.903 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.903 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.903 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:29.903 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.189774+0000", 2024-02-05T04:53:29.903 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.5950000000000004e-06 2024-02-05T04:53:29.903 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.903 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.903 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:29.903 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.189819+0000", 2024-02-05T04:53:29.903 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.5553000000000002e-05 2024-02-05T04:53:29.903 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.903 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.904 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:29.904 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.189826+0000", 2024-02-05T04:53:29.904 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.9659999999999999e-06 2024-02-05T04:53:29.904 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.904 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.904 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:29.904 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.193182+0000", 2024-02-05T04:53:29.904 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.003355284 2024-02-05T04:53:29.904 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.904 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.904 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:29.904 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.193216+0000", 2024-02-05T04:53:29.904 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.4017999999999999e-05 2024-02-05T04:53:29.904 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:29.904 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:29.905 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:29.905 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.905 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.905 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.es4 37/31 ECSubWrite(tid=21, reqid=client.4634.0:246, at_version=37'14, trim_to=0'0, roll_forward_to=34'10))", 2024-02-05T04:53:29.905 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:29.190836+0000", 2024-02-05T04:53:29.905 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 0.62373595900000001, 2024-02-05T04:53:29.905 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0048036299999999997, 2024-02-05T04:53:29.905 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:29.905 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:29.905 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:29.905 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.905 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:29.905 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.190836+0000", 2024-02-05T04:53:29.905 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:29.905 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.906 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.906 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:29.906 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.190836+0000", 2024-02-05T04:53:29.906 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:29.906 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.906 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.906 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:29.906 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.190836+0000", 2024-02-05T04:53:29.906 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.06e-07 2024-02-05T04:53:29.906 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.906 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.906 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:29.906 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.190851+0000", 2024-02-05T04:53:29.906 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.449e-05 2024-02-05T04:53:29.906 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.907 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.907 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:29.907 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.190873+0000", 2024-02-05T04:53:29.907 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.2025e-05 2024-02-05T04:53:29.907 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.907 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.907 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:29.907 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.190882+0000", 2024-02-05T04:53:29.907 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.4630000000000007e-06 2024-02-05T04:53:29.907 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.907 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.907 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:29.907 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.190929+0000", 2024-02-05T04:53:29.907 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.7083999999999998e-05 2024-02-05T04:53:29.907 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.908 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.908 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:29.908 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.190937+0000", 2024-02-05T04:53:29.908 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.3189999999999997e-06 2024-02-05T04:53:29.908 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.908 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.908 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:29.908 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.195607+0000", 2024-02-05T04:53:29.908 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.004670128 2024-02-05T04:53:29.908 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.908 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.908 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:29.908 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.195639+0000", 2024-02-05T04:53:29.908 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.2515000000000003e-05 2024-02-05T04:53:29.909 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:29.909 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:29.909 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:29.909 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.909 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.909 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.es4 37/31 ECSubWrite(tid=22, reqid=client.4634.0:249, at_version=37'16, trim_to=0'0, roll_forward_to=37'12))", 2024-02-05T04:53:29.909 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:29.192189+0000", 2024-02-05T04:53:29.909 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 0.62238282199999995, 2024-02-05T04:53:29.909 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.003478311, 2024-02-05T04:53:29.909 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:29.909 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:29.909 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:29.909 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.909 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:29.909 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.192189+0000", 2024-02-05T04:53:29.909 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:29.910 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.910 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.910 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:29.910 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.192189+0000", 2024-02-05T04:53:29.910 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:29.910 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.910 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.910 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:29.910 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.192189+0000", 2024-02-05T04:53:29.910 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.3600000000000003e-07 2024-02-05T04:53:29.910 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.910 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.910 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:29.911 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.192219+0000", 2024-02-05T04:53:29.911 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.9669e-05 2024-02-05T04:53:29.911 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.911 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.911 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:29.911 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.192232+0000", 2024-02-05T04:53:29.911 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.3084999999999999e-05 2024-02-05T04:53:29.911 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.911 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.911 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:29.911 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.192240+0000", 2024-02-05T04:53:29.911 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.1689999999999992e-06 2024-02-05T04:53:29.911 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.911 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.911 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:29.912 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.192291+0000", 2024-02-05T04:53:29.912 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.0560999999999999e-05 2024-02-05T04:53:29.912 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.912 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.912 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:29.912 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.192297+0000", 2024-02-05T04:53:29.912 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.7219999999999996e-06 2024-02-05T04:53:29.912 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.912 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.912 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:29.912 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.195644+0000", 2024-02-05T04:53:29.912 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.003347222 2024-02-05T04:53:29.912 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.912 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.912 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:29.913 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.195667+0000", 2024-02-05T04:53:29.913 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.3147e-05 2024-02-05T04:53:29.913 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:29.913 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:29.913 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:29.913 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.913 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.913 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.es4 37/31 ECSubWrite(tid=23, reqid=client.4634.0:249, at_version=0'0, trim_to=0'0, roll_forward_to=37'16))", 2024-02-05T04:53:29.913 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:29.198636+0000", 2024-02-05T04:53:29.913 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 0.61593579899999995, 2024-02-05T04:53:29.913 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0032105749999999998, 2024-02-05T04:53:29.913 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:29.913 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:29.914 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:29.914 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.914 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:29.914 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.198636+0000", 2024-02-05T04:53:29.914 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:29.914 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.914 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.914 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:29.914 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.198636+0000", 2024-02-05T04:53:29.914 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:29.914 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.914 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.914 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:29.914 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.198636+0000", 2024-02-05T04:53:29.914 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.3500000000000002e-07 2024-02-05T04:53:29.915 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.915 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.915 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:29.915 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.198645+0000", 2024-02-05T04:53:29.915 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.6240000000000008e-06 2024-02-05T04:53:29.915 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.915 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.915 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:29.915 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.198657+0000", 2024-02-05T04:53:29.915 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.1926000000000001e-05 2024-02-05T04:53:29.915 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.915 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.915 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:29.915 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.198664+0000", 2024-02-05T04:53:29.915 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.0140000000000001e-06 2024-02-05T04:53:29.916 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.916 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.916 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:29.916 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.198719+0000", 2024-02-05T04:53:29.916 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.4948000000000003e-05 2024-02-05T04:53:29.916 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.916 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.916 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:29.916 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.198727+0000", 2024-02-05T04:53:29.916 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.8259999999999993e-06 2024-02-05T04:53:29.916 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.916 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.916 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:29.916 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.201810+0000", 2024-02-05T04:53:29.917 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0030834080000000002 2024-02-05T04:53:29.917 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:29.917 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:29.917 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:29.917 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.201846+0000", 2024-02-05T04:53:29.917 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.6393999999999997e-05 2024-02-05T04:53:29.917 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:29.917 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:29.917 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:29.917 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:29.917 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:29.917 INFO:teuthology.orchestra.run.smithi091.stdout:} 2024-02-05T04:53:29.919 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-02-05T04:53:29.920 INFO:tasks.ceph.osd.6.smithi150.stderr:2024-02-05T04:53:29.918+0000 7f6bea9f0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34015) UID: 0 2024-02-05T04:53:30.020 INFO:tasks.ceph.osd.10.smithi150.stderr:2024-02-05T04:53:30.019+0000 7f3075a4d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 34018) UID: 0 2024-02-05T04:53:30.036 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-02-05T04:53:30.036 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [], 2024-02-05T04:53:30.036 INFO:teuthology.orchestra.run.smithi150.stdout: "num_ops": 0 2024-02-05T04:53:30.036 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-02-05T04:53:30.046 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-02-05T04:53:30.121 INFO:tasks.ceph.osd.15.smithi196.stderr:2024-02-05T04:53:30.121+0000 7f1facc45640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 34611) UID: 0 2024-02-05T04:53:30.127 INFO:teuthology.orchestra.run.smithi062.stderr:nodeep-scrub is unset 2024-02-05T04:53:30.127 INFO:teuthology.orchestra.run.smithi062.stderr:reweighted osd.9 to 1 (10000) 2024-02-05T04:53:30.131 INFO:tasks.rados.rados.0.smithi196.stdout:92: done (0 left) 2024-02-05T04:53:30.131 INFO:tasks.rados.rados.0.smithi196.stdout:93: read oid 39 snap -1 2024-02-05T04:53:30.132 INFO:tasks.rados.rados.0.smithi196.stdout:93: expect (ObjNum 38 snap 0 seq_num 38) 2024-02-05T04:53:30.132 INFO:tasks.rados.rados.0.smithi196.stdout:94: delete oid 26 current snap is 2 2024-02-05T04:53:30.138 INFO:tasks.rados.rados.0.smithi196.stdout:93: expect (ObjNum 38 snap 0 seq_num 38) 2024-02-05T04:53:30.152 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 10 1 2024-02-05T04:53:30.159 INFO:tasks.rados.rados.0.smithi196.stdout:93: done (1 left) 2024-02-05T04:53:30.159 INFO:tasks.rados.rados.0.smithi196.stdout:94: done (0 left) 2024-02-05T04:53:30.159 INFO:tasks.rados.rados.0.smithi196.stdout:95: rmattr oid 11 current snap is 2 2024-02-05T04:53:30.160 INFO:tasks.rados.rados.0.smithi196.stdout:95: done (0 left) 2024-02-05T04:53:30.160 INFO:tasks.rados.rados.0.smithi196.stdout:96: rmattr oid 26 current snap is 2 2024-02-05T04:53:30.160 INFO:tasks.rados.rados.0.smithi196.stdout:96: done (0 left) 2024-02-05T04:53:30.160 INFO:tasks.rados.rados.0.smithi196.stdout:97: rmattr oid 48 current snap is 2 2024-02-05T04:53:30.160 INFO:tasks.rados.rados.0.smithi196.stdout:97: done (0 left) 2024-02-05T04:53:30.160 INFO:tasks.rados.rados.0.smithi196.stdout:append oid 40 current snap is 2 2024-02-05T04:53:30.161 INFO:tasks.rados.rados.0.smithi196.stdout:98: seq_num 66 ranges {1589248=737280,2326528=376832} 2024-02-05T04:53:30.171 INFO:tasks.rados.rados.0.smithi196.stdout:98: writing smithi19639362-40 from 1589248 to 2326528 tid 1 2024-02-05T04:53:30.174 INFO:tasks.rados.rados.0.smithi196.stdout:98: writing smithi19639362-40 from 2326528 to 2703360 tid 2 2024-02-05T04:53:30.174 INFO:tasks.rados.rados.0.smithi196.stdout:append oid 4 current snap is 2 2024-02-05T04:53:30.174 INFO:tasks.rados.rados.0.smithi196.stdout:99: seq_num 67 ranges {65536=753664,819200=376832} 2024-02-05T04:53:30.179 INFO:tasks.rados.rados.0.smithi196.stdout:99: writing smithi19639362-4 from 65536 to 819200 tid 1 2024-02-05T04:53:30.182 INFO:tasks.rados.rados.0.smithi196.stdout:99: writing smithi19639362-4 from 819200 to 1196032 tid 2 2024-02-05T04:53:30.182 INFO:tasks.rados.rados.0.smithi196.stdout:100: read oid 1 snap -1 2024-02-05T04:53:30.183 INFO:tasks.rados.rados.0.smithi196.stdout:100: expect (ObjNum 0 snap 0 seq_num 0) 2024-02-05T04:53:30.183 INFO:tasks.rados.rados.0.smithi196.stdout:101: rollback oid 36 current snap is 2 2024-02-05T04:53:30.183 INFO:tasks.rados.rados.0.smithi196.stdout:101: done (3 left) 2024-02-05T04:53:30.183 INFO:tasks.rados.rados.0.smithi196.stdout:append oid 44 current snap is 2 2024-02-05T04:53:30.183 INFO:tasks.rados.rados.0.smithi196.stdout:98: finishing write tid 1 to smithi19639362-40 2024-02-05T04:53:30.183 INFO:tasks.rados.rados.0.smithi196.stdout:102: seq_num 68 ranges {737280=770048,1507328=376832} 2024-02-05T04:53:30.190 INFO:tasks.rados.rados.0.smithi196.stdout:102: writing smithi19639362-44 from 737280 to 1507328 tid 1 2024-02-05T04:53:30.193 INFO:tasks.rados.rados.0.smithi196.stdout:102: writing smithi19639362-44 from 1507328 to 1884160 tid 2 2024-02-05T04:53:30.193 INFO:tasks.rados.rados.0.smithi196.stdout:103: setattr oid 18 current snap is 2 2024-02-05T04:53:30.193 INFO:tasks.rados.rados.0.smithi196.stdout:98: finishing write tid 2 to smithi19639362-40 2024-02-05T04:53:30.193 INFO:tasks.rados.rados.0.smithi196.stdout:98: finishing write tid 3 to smithi19639362-40 2024-02-05T04:53:30.193 INFO:tasks.rados.rados.0.smithi196.stdout:98: finishing write tid 4 to smithi19639362-40 2024-02-05T04:53:30.193 INFO:tasks.rados.rados.0.smithi196.stdout:98: oid 40 updating version 0 to 17 2024-02-05T04:53:30.193 INFO:tasks.rados.rados.0.smithi196.stdout:98: oid 40 updating version 17 to 18 2024-02-05T04:53:30.193 INFO:tasks.rados.rados.0.smithi196.stdout:98: oid 40 updating version 18 to 19 2024-02-05T04:53:30.194 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 40 v 19 (ObjNum 66 snap 2 seq_num 66) dirty exists 2024-02-05T04:53:30.194 INFO:tasks.rados.rados.0.smithi196.stdout:98: left oid 40 (ObjNum 66 snap 2 seq_num 66) 2024-02-05T04:53:30.195 INFO:tasks.rados.rados.0.smithi196.stdout:98: done (4 left) 2024-02-05T04:53:30.195 INFO:tasks.rados.rados.0.smithi196.stdout:104: read oid 23 snap -1 2024-02-05T04:53:30.195 INFO:tasks.rados.rados.0.smithi196.stdout:104: expect (ObjNum 63 snap 2 seq_num 63) 2024-02-05T04:53:30.195 INFO:tasks.rados.rados.0.smithi196.stdout:append oid 35 current snap is 2 2024-02-05T04:53:30.195 INFO:tasks.rados.rados.0.smithi196.stdout:105: seq_num 69 ranges {573440=786432,1359872=376832} 2024-02-05T04:53:30.202 INFO:tasks.rados.rados.0.smithi196.stdout:105: writing smithi19639362-35 from 573440 to 1359872 tid 1 2024-02-05T04:53:30.204 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-02-05T04:53:30.204 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [], 2024-02-05T04:53:30.204 INFO:teuthology.orchestra.run.smithi150.stdout: "complaint_time": 30, 2024-02-05T04:53:30.205 INFO:teuthology.orchestra.run.smithi150.stdout: "num_blocked_ops": 0 2024-02-05T04:53:30.205 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-02-05T04:53:30.205 INFO:tasks.rados.rados.0.smithi196.stdout:105: writing smithi19639362-35 from 1359872 to 1736704 tid 2 2024-02-05T04:53:30.205 INFO:tasks.rados.rados.0.smithi196.stdout:append oid 6 current snap is 2 2024-02-05T04:53:30.206 INFO:tasks.rados.rados.0.smithi196.stdout:102: finishing write tid 1 to smithi19639362-44 2024-02-05T04:53:30.206 INFO:tasks.rados.rados.0.smithi196.stdout:102: finishing write tid 2 to smithi19639362-44 2024-02-05T04:53:30.206 INFO:tasks.rados.rados.0.smithi196.stdout:102: finishing write tid 3 to smithi19639362-44 2024-02-05T04:53:30.206 INFO:tasks.rados.rados.0.smithi196.stdout:102: finishing write tid 4 to smithi19639362-44 2024-02-05T04:53:30.206 INFO:tasks.rados.rados.0.smithi196.stdout:102: oid 44 updating version 0 to 22 2024-02-05T04:53:30.206 INFO:tasks.rados.rados.0.smithi196.stdout:102: oid 44 version 22 is already newer than 20 2024-02-05T04:53:30.206 INFO:tasks.rados.rados.0.smithi196.stdout:102: oid 44 version 22 is already newer than 21 2024-02-05T04:53:30.206 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 44 v 22 (ObjNum 68 snap 2 seq_num 68) dirty exists 2024-02-05T04:53:30.206 INFO:tasks.rados.rados.0.smithi196.stdout:102: left oid 44 (ObjNum 68 snap 2 seq_num 68) 2024-02-05T04:53:30.206 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 18 v 17 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-02-05T04:53:30.206 INFO:tasks.rados.rados.0.smithi196.stdout:106: seq_num 70 ranges {98304=802816,901120=376832} 2024-02-05T04:53:30.209 INFO:tasks.rados.rados.0.smithi196.stdout:106: writing smithi19639362-6 from 98304 to 901120 tid 1 2024-02-05T04:53:30.211 INFO:tasks.rados.rados.0.smithi196.stdout:106: writing smithi19639362-6 from 901120 to 1277952 tid 2 2024-02-05T04:53:30.212 INFO:tasks.rados.rados.0.smithi196.stdout:102: done (6 left) 2024-02-05T04:53:30.212 INFO:tasks.rados.rados.0.smithi196.stdout:103: done (5 left) 2024-02-05T04:53:30.212 INFO:tasks.rados.rados.0.smithi196.stdout:append oid 17 current snap is 2 2024-02-05T04:53:30.212 INFO:tasks.rados.rados.0.smithi196.stdout:107: seq_num 71 ranges {278528=425984,704512=573440,1277952=196608} 2024-02-05T04:53:30.214 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-02-05T04:53:30.215 INFO:tasks.rados.rados.0.smithi196.stdout:107: writing smithi19639362-17 from 278528 to 704512 tid 1 2024-02-05T04:53:30.218 INFO:tasks.rados.rados.0.smithi196.stdout:107: writing smithi19639362-17 from 704512 to 1277952 tid 2 2024-02-05T04:53:30.219 INFO:tasks.rados.rados.0.smithi196.stdout:107: writing smithi19639362-17 from 1277952 to 1474560 tid 3 2024-02-05T04:53:30.220 INFO:tasks.rados.rados.0.smithi196.stdout:108: snap_create 2024-02-05T04:53:30.220 INFO:tasks.rados.rados.0.smithi196.stdout:104: expect (ObjNum 63 snap 2 seq_num 63) 2024-02-05T04:53:30.222 INFO:tasks.ceph.osd.15.smithi196.stderr:2024-02-05T04:53:30.221+0000 7f1facc45640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 34611) UID: 0 2024-02-05T04:53:30.262 INFO:tasks.rados.rados.0.smithi196.stdout:105: finishing write tid 1 to smithi19639362-35 2024-02-05T04:53:30.262 INFO:tasks.rados.rados.0.smithi196.stdout:105: finishing write tid 2 to smithi19639362-35 2024-02-05T04:53:30.263 INFO:tasks.rados.rados.0.smithi196.stdout:105: finishing write tid 3 to smithi19639362-35 2024-02-05T04:53:30.263 INFO:tasks.rados.rados.0.smithi196.stdout:106: finishing write tid 1 to smithi19639362-6 2024-02-05T04:53:30.263 INFO:tasks.rados.rados.0.smithi196.stdout:105: finishing write tid 4 to smithi19639362-35 2024-02-05T04:53:30.263 INFO:tasks.rados.rados.0.smithi196.stdout:105: oid 35 updating version 0 to 20 2024-02-05T04:53:30.263 INFO:tasks.rados.rados.0.smithi196.stdout:105: oid 35 version 20 is already newer than 18 2024-02-05T04:53:30.263 INFO:tasks.rados.rados.0.smithi196.stdout:105: oid 35 version 20 is already newer than 19 2024-02-05T04:53:30.263 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 35 v 20 (ObjNum 69 snap 2 seq_num 69) dirty exists 2024-02-05T04:53:30.263 INFO:tasks.rados.rados.0.smithi196.stdout:105: left oid 35 (ObjNum 69 snap 2 seq_num 69) 2024-02-05T04:53:30.263 INFO:tasks.rados.rados.0.smithi196.stdout:106: finishing write tid 2 to smithi19639362-6 2024-02-05T04:53:30.263 INFO:tasks.rados.rados.0.smithi196.stdout:106: finishing write tid 3 to smithi19639362-6 2024-02-05T04:53:30.263 INFO:tasks.rados.rados.0.smithi196.stdout:106: finishing write tid 4 to smithi19639362-6 2024-02-05T04:53:30.263 INFO:tasks.rados.rados.0.smithi196.stdout:106: oid 6 updating version 0 to 23 2024-02-05T04:53:30.263 INFO:tasks.rados.rados.0.smithi196.stdout:106: oid 6 version 23 is already newer than 22 2024-02-05T04:53:30.263 INFO:tasks.rados.rados.0.smithi196.stdout:106: oid 6 version 23 is already newer than 21 2024-02-05T04:53:30.264 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 6 v 23 (ObjNum 70 snap 2 seq_num 70) dirty exists 2024-02-05T04:53:30.264 INFO:tasks.rados.rados.0.smithi196.stdout:106: left oid 6 (ObjNum 70 snap 2 seq_num 70) 2024-02-05T04:53:30.322 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-02-05T04:53:30.322 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 20, 2024-02-05T04:53:30.322 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 600, 2024-02-05T04:53:30.322 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [ 2024-02-05T04:53:30.322 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.322 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.4209.0:85 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'55, mlcod=24'50)", 2024-02-05T04:53:30.322 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:09.697543+0000", 2024-02-05T04:53:30.322 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 20.622435415999998, 2024-02-05T04:53:30.322 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0081615030000000005, 2024-02-05T04:53:30.322 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:30.322 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:30.323 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:30.323 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.323 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:30.323 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.697543+0000", 2024-02-05T04:53:30.323 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:30.323 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.323 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.323 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:30.323 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.697543+0000", 2024-02-05T04:53:30.323 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:30.323 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.323 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.323 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:30.323 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.697544+0000", 2024-02-05T04:53:30.323 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.7899999999999999e-07 2024-02-05T04:53:30.324 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.324 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.324 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:30.324 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.697546+0000", 2024-02-05T04:53:30.324 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.7970000000000001e-06 2024-02-05T04:53:30.324 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.324 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.324 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:30.324 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.697553+0000", 2024-02-05T04:53:30.324 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.2929999999999997e-06 2024-02-05T04:53:30.324 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.324 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.324 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:30.325 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.697556+0000", 2024-02-05T04:53:30.325 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.4309999999999998e-06 2024-02-05T04:53:30.325 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.325 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.325 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:30.325 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.697596+0000", 2024-02-05T04:53:30.325 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.977e-05 2024-02-05T04:53:30.325 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.325 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.325 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:53:30.325 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.697607+0000", 2024-02-05T04:53:30.325 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1459e-05 2024-02-05T04:53:30.325 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.325 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.326 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-02-05T04:53:30.326 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.705669+0000", 2024-02-05T04:53:30.326 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0080620009999999992 2024-02-05T04:53:30.326 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.326 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.326 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:30.326 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.705705+0000", 2024-02-05T04:53:30.326 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.5373000000000002e-05 2024-02-05T04:53:30.326 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:30.326 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:30.326 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:30.326 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.326 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.326 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.4209.0:86 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'56, mlcod=24'52)", 2024-02-05T04:53:30.327 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:09.698014+0000", 2024-02-05T04:53:30.327 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 20.621964316, 2024-02-05T04:53:30.327 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0077071750000000001, 2024-02-05T04:53:30.327 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:30.327 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:30.327 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:30.327 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.327 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:30.327 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698014+0000", 2024-02-05T04:53:30.327 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:30.327 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.327 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.327 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:30.327 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698014+0000", 2024-02-05T04:53:30.327 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:30.328 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.328 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.328 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:30.328 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698015+0000", 2024-02-05T04:53:30.328 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.9900000000000002e-07 2024-02-05T04:53:30.328 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.328 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.328 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:30.328 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698018+0000", 2024-02-05T04:53:30.328 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.0019999999999998e-06 2024-02-05T04:53:30.328 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.328 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.328 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:30.329 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698024+0000", 2024-02-05T04:53:30.329 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.251e-06 2024-02-05T04:53:30.329 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.329 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.329 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:30.329 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698028+0000", 2024-02-05T04:53:30.329 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.76e-06 2024-02-05T04:53:30.329 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.329 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.329 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:30.329 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698066+0000", 2024-02-05T04:53:30.329 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.8556999999999997e-05 2024-02-05T04:53:30.329 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.329 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.329 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:53:30.330 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698078+0000", 2024-02-05T04:53:30.330 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1620999999999999e-05 2024-02-05T04:53:30.330 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.330 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.330 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-02-05T04:53:30.330 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.705706+0000", 2024-02-05T04:53:30.330 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0076284400000000002 2024-02-05T04:53:30.330 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.330 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.330 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:30.330 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.705721+0000", 2024-02-05T04:53:30.330 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.5245000000000001e-05 2024-02-05T04:53:30.330 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:30.331 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:30.331 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:30.331 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.331 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.331 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.4209.0:87 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'57, mlcod=24'52)", 2024-02-05T04:53:30.331 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:09.698307+0000", 2024-02-05T04:53:30.331 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 20.621671387999999, 2024-02-05T04:53:30.331 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0074282530000000001, 2024-02-05T04:53:30.331 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:30.331 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:30.331 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:30.331 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.331 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:30.332 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698307+0000", 2024-02-05T04:53:30.332 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:30.332 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.332 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.332 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:30.332 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698307+0000", 2024-02-05T04:53:30.332 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:30.332 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.332 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.332 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:30.332 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698308+0000", 2024-02-05T04:53:30.333 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.9900000000000001e-07 2024-02-05T04:53:30.333 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.333 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.333 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:30.333 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698358+0000", 2024-02-05T04:53:30.333 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.0649000000000002e-05 2024-02-05T04:53:30.333 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.333 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.333 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:30.333 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698366+0000", 2024-02-05T04:53:30.333 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.9000000000000006e-06 2024-02-05T04:53:30.333 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.333 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.334 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:30.334 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698370+0000", 2024-02-05T04:53:30.334 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.7270000000000001e-06 2024-02-05T04:53:30.334 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.334 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.334 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:30.334 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698410+0000", 2024-02-05T04:53:30.334 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.0117000000000002e-05 2024-02-05T04:53:30.334 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.334 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.334 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:53:30.334 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698423+0000", 2024-02-05T04:53:30.334 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2531000000000001e-05 2024-02-05T04:53:30.334 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.334 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.335 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-02-05T04:53:30.335 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.705722+0000", 2024-02-05T04:53:30.335 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0072995270000000001 2024-02-05T04:53:30.335 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.335 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.335 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:30.335 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.705735+0000", 2024-02-05T04:53:30.335 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.3403e-05 2024-02-05T04:53:30.335 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:30.335 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:30.335 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:30.335 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.335 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.335 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.4209.0:88 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'58, mlcod=24'52)", 2024-02-05T04:53:30.335 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:09.698756+0000", 2024-02-05T04:53:30.336 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 20.621222299999999, 2024-02-05T04:53:30.336 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0072080540000000002, 2024-02-05T04:53:30.336 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:30.336 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:30.336 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:30.336 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.336 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:30.336 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698756+0000", 2024-02-05T04:53:30.336 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:30.336 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.336 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.336 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:30.336 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698756+0000", 2024-02-05T04:53:30.337 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:30.337 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.337 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.337 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:30.337 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698757+0000", 2024-02-05T04:53:30.337 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.6899999999999997e-07 2024-02-05T04:53:30.337 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.337 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.337 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:30.337 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698762+0000", 2024-02-05T04:53:30.337 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.8790000000000002e-06 2024-02-05T04:53:30.337 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.337 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.337 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:30.337 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698773+0000", 2024-02-05T04:53:30.338 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1e-05 2024-02-05T04:53:30.338 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.338 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.338 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:30.338 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698780+0000", 2024-02-05T04:53:30.338 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.8920000000000003e-06 2024-02-05T04:53:30.338 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.338 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.338 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:30.338 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.700009+0000", 2024-02-05T04:53:30.338 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.001229665 2024-02-05T04:53:30.338 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.338 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.338 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:53:30.338 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.700022+0000", 2024-02-05T04:53:30.339 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2699e-05 2024-02-05T04:53:30.339 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.339 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.339 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-02-05T04:53:30.339 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.705943+0000", 2024-02-05T04:53:30.339 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0059209759999999997 2024-02-05T04:53:30.339 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.339 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.339 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:30.339 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.705964+0000", 2024-02-05T04:53:30.339 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.1274e-05 2024-02-05T04:53:30.339 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:30.339 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:30.339 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:30.339 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.340 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.340 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.4209.0:110 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'72, mlcod=24'67)", 2024-02-05T04:53:30.340 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:09.853702+0000", 2024-02-05T04:53:30.340 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 20.466276118, 2024-02-05T04:53:30.340 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0063171329999999999, 2024-02-05T04:53:30.340 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:30.340 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:30.340 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:30.340 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.340 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:30.340 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.853702+0000", 2024-02-05T04:53:30.340 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:30.340 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.340 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.341 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:30.341 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.853702+0000", 2024-02-05T04:53:30.341 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:30.341 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.341 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.341 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:30.341 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.853703+0000", 2024-02-05T04:53:30.341 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.1100000000000002e-07 2024-02-05T04:53:30.341 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.341 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.341 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:30.341 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.853706+0000", 2024-02-05T04:53:30.341 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.3629999999999998e-06 2024-02-05T04:53:30.341 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.341 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.341 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:30.342 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.853713+0000", 2024-02-05T04:53:30.342 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.8410000000000002e-06 2024-02-05T04:53:30.342 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.342 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.342 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:30.342 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.853717+0000", 2024-02-05T04:53:30.342 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.777e-06 2024-02-05T04:53:30.342 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.342 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.342 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:30.342 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.856746+0000", 2024-02-05T04:53:30.342 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0030292800000000001 2024-02-05T04:53:30.342 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.342 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.342 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:53:30.343 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.856762+0000", 2024-02-05T04:53:30.343 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.5651999999999999e-05 2024-02-05T04:53:30.343 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.343 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.343 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-02-05T04:53:30.343 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.859984+0000", 2024-02-05T04:53:30.343 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0032218289999999998 2024-02-05T04:53:30.343 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.343 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.343 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:30.343 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.860020+0000", 2024-02-05T04:53:30.343 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.608e-05 2024-02-05T04:53:30.343 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:30.343 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:30.344 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:30.344 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.344 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.344 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.4209.0:111 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'73, mlcod=24'68)", 2024-02-05T04:53:30.344 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:09.854221+0000", 2024-02-05T04:53:30.344 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 20.465757787000001, 2024-02-05T04:53:30.344 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0058190960000000002, 2024-02-05T04:53:30.344 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:30.344 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:30.344 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:30.344 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.344 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:30.344 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.854221+0000", 2024-02-05T04:53:30.344 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:30.344 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.345 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.345 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:30.345 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.854221+0000", 2024-02-05T04:53:30.345 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:30.345 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.345 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.345 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:30.345 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.854221+0000", 2024-02-05T04:53:30.345 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.72e-07 2024-02-05T04:53:30.345 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.345 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.345 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:30.345 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.854224+0000", 2024-02-05T04:53:30.345 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.1779999999999999e-06 2024-02-05T04:53:30.346 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.346 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.346 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:30.346 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.854231+0000", 2024-02-05T04:53:30.346 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.7440000000000001e-06 2024-02-05T04:53:30.346 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.346 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.346 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:30.346 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.854235+0000", 2024-02-05T04:53:30.346 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.8349999999999997e-06 2024-02-05T04:53:30.346 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.346 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.346 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:30.346 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.856952+0000", 2024-02-05T04:53:30.346 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0027175839999999999 2024-02-05T04:53:30.346 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.347 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.347 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:53:30.347 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.856966+0000", 2024-02-05T04:53:30.347 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.3739e-05 2024-02-05T04:53:30.347 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.347 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.347 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-02-05T04:53:30.347 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.860023+0000", 2024-02-05T04:53:30.347 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0030564979999999999 2024-02-05T04:53:30.347 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.347 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.347 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:30.347 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.860040+0000", 2024-02-05T04:53:30.347 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.7146000000000002e-05 2024-02-05T04:53:30.347 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:30.348 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:30.348 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:30.348 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.348 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.348 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.4209.0:112 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'74, mlcod=24'69)", 2024-02-05T04:53:30.348 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:09.854776+0000", 2024-02-05T04:53:30.348 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 20.465202777000002, 2024-02-05T04:53:30.348 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0052805700000000001, 2024-02-05T04:53:30.348 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:30.348 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:30.348 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:30.349 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.349 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:30.349 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.854776+0000", 2024-02-05T04:53:30.349 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:30.349 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.349 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.350 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:30.350 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.854776+0000", 2024-02-05T04:53:30.350 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:30.350 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.350 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.350 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:30.350 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.854777+0000", 2024-02-05T04:53:30.350 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.1999999999999999e-07 2024-02-05T04:53:30.350 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.350 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.350 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:30.350 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.854805+0000", 2024-02-05T04:53:30.350 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.8827999999999999e-05 2024-02-05T04:53:30.350 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.350 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.351 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:30.351 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.854817+0000", 2024-02-05T04:53:30.351 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.165e-05 2024-02-05T04:53:30.351 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.351 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.351 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:30.351 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.854822+0000", 2024-02-05T04:53:30.351 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.293e-06 2024-02-05T04:53:30.351 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.351 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.351 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:30.351 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.857167+0000", 2024-02-05T04:53:30.351 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0023451879999999998 2024-02-05T04:53:30.351 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.351 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.352 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:53:30.352 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.857181+0000", 2024-02-05T04:53:30.352 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.3455000000000001e-05 2024-02-05T04:53:30.352 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.352 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.352 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-02-05T04:53:30.352 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.860042+0000", 2024-02-05T04:53:30.352 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0028610839999999998 2024-02-05T04:53:30.352 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.352 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.352 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:30.352 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.860056+0000", 2024-02-05T04:53:30.352 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.4352000000000001e-05 2024-02-05T04:53:30.352 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:30.352 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:30.353 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:30.353 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.353 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.353 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.4209.0:113 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'75, mlcod=24'69)", 2024-02-05T04:53:30.353 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:09.855187+0000", 2024-02-05T04:53:30.353 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 20.464791375000001, 2024-02-05T04:53:30.353 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.004895684, 2024-02-05T04:53:30.353 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:30.353 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:30.353 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:30.353 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.353 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:30.353 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.855187+0000", 2024-02-05T04:53:30.353 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:30.353 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.353 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.354 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:30.354 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.855187+0000", 2024-02-05T04:53:30.354 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:30.354 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.354 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.354 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:30.354 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.855188+0000", 2024-02-05T04:53:30.354 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.5299999999999999e-07 2024-02-05T04:53:30.354 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.354 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.354 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:30.354 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.855191+0000", 2024-02-05T04:53:30.354 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.4790000000000001e-06 2024-02-05T04:53:30.354 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.354 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.355 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:30.355 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.855199+0000", 2024-02-05T04:53:30.355 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.4769999999999997e-06 2024-02-05T04:53:30.355 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.355 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.355 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:30.355 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.855203+0000", 2024-02-05T04:53:30.355 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.6149999999999999e-06 2024-02-05T04:53:30.355 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.355 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.355 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:30.355 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.857370+0000", 2024-02-05T04:53:30.355 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.002166692 2024-02-05T04:53:30.355 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.355 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.355 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:53:30.356 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.857381+0000", 2024-02-05T04:53:30.356 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1394e-05 2024-02-05T04:53:30.356 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.356 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.356 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-02-05T04:53:30.356 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.860058+0000", 2024-02-05T04:53:30.356 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0026767190000000001 2024-02-05T04:53:30.356 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.356 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.356 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:30.356 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.860083+0000", 2024-02-05T04:53:30.356 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.4854999999999998e-05 2024-02-05T04:53:30.356 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:30.356 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:30.356 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:30.357 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.357 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.357 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.4209.0:115 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'77, mlcod=24'69)", 2024-02-05T04:53:30.357 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:09.856748+0000", 2024-02-05T04:53:30.357 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 20.463230900999999, 2024-02-05T04:53:30.357 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0058814849999999997, 2024-02-05T04:53:30.357 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:30.357 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:30.357 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:30.357 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.357 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:30.357 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.856748+0000", 2024-02-05T04:53:30.357 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:30.357 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.357 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.357 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:30.358 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.856748+0000", 2024-02-05T04:53:30.358 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:30.358 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.358 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.358 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:30.358 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.856748+0000", 2024-02-05T04:53:30.358 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.4099999999999998e-07 2024-02-05T04:53:30.358 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.358 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.358 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:30.358 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.856781+0000", 2024-02-05T04:53:30.358 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.3056999999999999e-05 2024-02-05T04:53:30.358 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.358 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.358 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:30.359 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.856791+0000", 2024-02-05T04:53:30.359 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.5659999999999996e-06 2024-02-05T04:53:30.359 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.359 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.359 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:30.359 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.856799+0000", 2024-02-05T04:53:30.359 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.8720000000000002e-06 2024-02-05T04:53:30.359 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.359 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.359 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:30.359 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.857805+0000", 2024-02-05T04:53:30.359 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0010061180000000001 2024-02-05T04:53:30.360 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.360 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.360 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:53:30.360 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.857820+0000", 2024-02-05T04:53:30.360 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.4883e-05 2024-02-05T04:53:30.360 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.360 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.360 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-02-05T04:53:30.360 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.862596+0000", 2024-02-05T04:53:30.360 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0047763249999999997 2024-02-05T04:53:30.360 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.360 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.360 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:30.360 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.862629+0000", 2024-02-05T04:53:30.360 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.2923000000000003e-05 2024-02-05T04:53:30.361 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:30.361 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:30.361 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:30.361 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.361 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.361 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.4209.0:116 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'78, mlcod=24'69)", 2024-02-05T04:53:30.361 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:09.857396+0000", 2024-02-05T04:53:30.361 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 20.462582566999998, 2024-02-05T04:53:30.361 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0052485070000000003, 2024-02-05T04:53:30.361 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:30.361 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:30.361 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:30.361 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.361 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:30.361 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.857396+0000", 2024-02-05T04:53:30.362 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:30.362 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.362 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.362 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:30.362 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.857396+0000", 2024-02-05T04:53:30.362 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:30.362 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.362 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.362 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:30.362 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.857397+0000", 2024-02-05T04:53:30.362 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.3799999999999996e-07 2024-02-05T04:53:30.362 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.362 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.362 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:30.362 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.857402+0000", 2024-02-05T04:53:30.362 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.3149999999999999e-06 2024-02-05T04:53:30.363 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.363 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.363 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:30.363 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.857411+0000", 2024-02-05T04:53:30.363 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.4180000000000003e-06 2024-02-05T04:53:30.363 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.363 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.363 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:30.363 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.857417+0000", 2024-02-05T04:53:30.363 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.2789999999999998e-06 2024-02-05T04:53:30.363 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.363 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.363 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:30.363 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.859810+0000", 2024-02-05T04:53:30.363 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0023927530000000001 2024-02-05T04:53:30.364 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.364 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.364 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:53:30.364 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.859824+0000", 2024-02-05T04:53:30.364 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.419e-05 2024-02-05T04:53:30.364 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.364 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.364 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-02-05T04:53:30.364 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.862631+0000", 2024-02-05T04:53:30.364 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0028069380000000001 2024-02-05T04:53:30.364 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.364 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.364 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:30.364 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.862645+0000", 2024-02-05T04:53:30.364 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.3876000000000001e-05 2024-02-05T04:53:30.365 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:30.365 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:30.365 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:30.365 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.365 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.365 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.4209.0:117 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'79, mlcod=24'69)", 2024-02-05T04:53:30.365 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:09.857803+0000", 2024-02-05T04:53:30.365 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 20.462175175999999, 2024-02-05T04:53:30.365 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0076301729999999996, 2024-02-05T04:53:30.365 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:30.365 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:30.365 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:30.365 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.365 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:30.365 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.857803+0000", 2024-02-05T04:53:30.366 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:30.366 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.366 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.366 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:30.366 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.857803+0000", 2024-02-05T04:53:30.366 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:30.366 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.366 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.366 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:30.366 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.857804+0000", 2024-02-05T04:53:30.366 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.1999999999999999e-07 2024-02-05T04:53:30.366 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.366 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.366 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:30.366 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.857810+0000", 2024-02-05T04:53:30.367 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.8690000000000001e-06 2024-02-05T04:53:30.367 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.367 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.367 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:30.367 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.857821+0000", 2024-02-05T04:53:30.367 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1059e-05 2024-02-05T04:53:30.367 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.367 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.367 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:30.367 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.857827+0000", 2024-02-05T04:53:30.367 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.1430000000000003e-06 2024-02-05T04:53:30.367 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.367 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.367 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:30.367 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.860138+0000", 2024-02-05T04:53:30.367 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0023111160000000002 2024-02-05T04:53:30.368 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.368 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.368 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:53:30.368 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.860152+0000", 2024-02-05T04:53:30.368 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.3896999999999999e-05 2024-02-05T04:53:30.368 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.368 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.368 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-02-05T04:53:30.368 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.865400+0000", 2024-02-05T04:53:30.368 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0052475040000000001 2024-02-05T04:53:30.368 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.368 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.368 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:30.368 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.865434+0000", 2024-02-05T04:53:30.369 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.3865000000000002e-05 2024-02-05T04:53:30.369 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:30.369 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:30.369 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:30.369 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.369 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.369 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.4209.0:139 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'97, mlcod=25'93)", 2024-02-05T04:53:30.369 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:10.001153+0000", 2024-02-05T04:53:30.369 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 20.318826056999999, 2024-02-05T04:53:30.369 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0050519220000000004, 2024-02-05T04:53:30.369 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:30.369 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:30.369 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:30.369 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.369 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:30.369 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:10.001153+0000", 2024-02-05T04:53:30.370 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:30.370 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.370 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.370 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:30.370 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:10.001153+0000", 2024-02-05T04:53:30.370 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:30.370 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.370 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.370 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:30.370 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:10.001153+0000", 2024-02-05T04:53:30.370 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.4200000000000001e-07 2024-02-05T04:53:30.370 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.370 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.370 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:30.370 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:10.001156+0000", 2024-02-05T04:53:30.370 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.3699999999999999e-06 2024-02-05T04:53:30.371 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.371 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.371 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:30.371 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:10.001163+0000", 2024-02-05T04:53:30.371 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.917e-06 2024-02-05T04:53:30.371 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.371 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.371 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:30.371 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:10.001168+0000", 2024-02-05T04:53:30.371 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.0359999999999997e-06 2024-02-05T04:53:30.371 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.371 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.371 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:30.371 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:10.001202+0000", 2024-02-05T04:53:30.371 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.3686000000000002e-05 2024-02-05T04:53:30.372 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.372 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.372 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:53:30.372 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:10.001215+0000", 2024-02-05T04:53:30.372 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2782999999999999e-05 2024-02-05T04:53:30.372 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.372 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.372 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-02-05T04:53:30.372 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:10.006093+0000", 2024-02-05T04:53:30.372 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0048779560000000001 2024-02-05T04:53:30.372 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.372 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.372 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:30.372 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:10.006204+0000", 2024-02-05T04:53:30.372 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00011173199999999999 2024-02-05T04:53:30.373 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:30.373 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:30.373 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:30.373 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.373 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.373 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.3s4 37/31 ECSubWrite(tid=29, reqid=client.4634.0:218, at_version=37'12, trim_to=0'0, roll_forward_to=37'10))", 2024-02-05T04:53:30.373 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:29.134242+0000", 2024-02-05T04:53:30.373 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 1.1857363480000001, 2024-02-05T04:53:30.373 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0046501859999999997, 2024-02-05T04:53:30.373 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:30.373 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:53:30.373 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:30.373 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.373 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:30.373 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.134242+0000", 2024-02-05T04:53:30.374 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:30.374 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.374 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.374 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:30.374 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.134242+0000", 2024-02-05T04:53:30.374 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:30.374 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.374 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.374 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:30.374 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.134243+0000", 2024-02-05T04:53:30.374 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.5799999999999999e-07 2024-02-05T04:53:30.374 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.374 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.374 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:30.374 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.134745+0000", 2024-02-05T04:53:30.374 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00050213099999999995 2024-02-05T04:53:30.375 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.375 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.375 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:30.375 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.134758+0000", 2024-02-05T04:53:30.375 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2831e-05 2024-02-05T04:53:30.375 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.375 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.375 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:30.375 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.134764+0000", 2024-02-05T04:53:30.375 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.635e-06 2024-02-05T04:53:30.375 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.375 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.375 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:30.375 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.134814+0000", 2024-02-05T04:53:30.375 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.9032000000000001e-05 2024-02-05T04:53:30.376 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.376 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.376 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:30.376 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.134820+0000", 2024-02-05T04:53:30.376 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.4969999999999998e-06 2024-02-05T04:53:30.376 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.376 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.376 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:30.376 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.138862+0000", 2024-02-05T04:53:30.376 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0040417760000000004 2024-02-05T04:53:30.376 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.376 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.376 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:30.376 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.138892+0000", 2024-02-05T04:53:30.377 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.0626000000000001e-05 2024-02-05T04:53:30.377 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:30.377 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:30.377 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:30.377 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.377 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.377 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.3s4 37/31 ECSubWrite(tid=30, reqid=client.4634.0:219, at_version=37'13, trim_to=0'0, roll_forward_to=37'10))", 2024-02-05T04:53:30.377 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:29.137013+0000", 2024-02-05T04:53:30.377 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 1.182965896, 2024-02-05T04:53:30.377 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0048480470000000003, 2024-02-05T04:53:30.377 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:30.377 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:53:30.377 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:30.377 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.377 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:30.377 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.137013+0000", 2024-02-05T04:53:30.378 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:30.378 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.378 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.378 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:30.378 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.137013+0000", 2024-02-05T04:53:30.378 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:30.378 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.378 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.378 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:30.378 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.137013+0000", 2024-02-05T04:53:30.378 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.2500000000000005e-07 2024-02-05T04:53:30.378 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.378 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.378 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:30.378 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.137139+0000", 2024-02-05T04:53:30.378 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000125246 2024-02-05T04:53:30.379 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.379 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.379 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:30.379 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.137151+0000", 2024-02-05T04:53:30.379 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2597e-05 2024-02-05T04:53:30.379 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.379 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.379 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:30.379 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.137158+0000", 2024-02-05T04:53:30.379 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.9419999999999998e-06 2024-02-05T04:53:30.379 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.379 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.379 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:30.379 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.137208+0000", 2024-02-05T04:53:30.379 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.0294999999999999e-05 2024-02-05T04:53:30.380 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.380 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.380 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:30.380 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.137216+0000", 2024-02-05T04:53:30.380 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.7849999999999992e-06 2024-02-05T04:53:30.380 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.380 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.380 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:30.380 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.141821+0000", 2024-02-05T04:53:30.380 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.004604258 2024-02-05T04:53:30.380 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.380 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.380 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:30.380 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.141861+0000", 2024-02-05T04:53:30.380 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.0198999999999999e-05 2024-02-05T04:53:30.381 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:30.381 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:30.381 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:30.381 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.381 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.381 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.4634.0:244 3.es0 3:796382fe:::smithi19639362-29:head [append 0~0] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:53:30.381 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:29.186772+0000", 2024-02-05T04:53:30.381 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 1.133206269, 2024-02-05T04:53:30.381 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0046280339999999996, 2024-02-05T04:53:30.381 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:30.381 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:30.381 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-02-05T04:53:30.381 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.4634", 2024-02-05T04:53:30.381 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:30.381 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 244 2024-02-05T04:53:30.381 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.382 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:30.382 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.382 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:30.382 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186772+0000", 2024-02-05T04:53:30.382 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:30.382 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.382 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.382 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:30.382 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186772+0000", 2024-02-05T04:53:30.382 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:30.382 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.382 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.382 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:30.382 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186775+0000", 2024-02-05T04:53:30.382 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.5500000000000001e-06 2024-02-05T04:53:30.383 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.383 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.383 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:30.383 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186785+0000", 2024-02-05T04:53:30.383 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.8430000000000005e-06 2024-02-05T04:53:30.383 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.383 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.383 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:30.383 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186815+0000", 2024-02-05T04:53:30.383 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.0406e-05 2024-02-05T04:53:30.383 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.383 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.383 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:30.383 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186830+0000", 2024-02-05T04:53:30.383 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.4883e-05 2024-02-05T04:53:30.383 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.384 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.384 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:30.384 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186928+0000", 2024-02-05T04:53:30.384 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.7831000000000006e-05 2024-02-05T04:53:30.384 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.384 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.384 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:53:30.384 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.187055+0000", 2024-02-05T04:53:30.384 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00012738699999999999 2024-02-05T04:53:30.384 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.384 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.384 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:30.384 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.188131+0000", 2024-02-05T04:53:30.384 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0010757049999999999 2024-02-05T04:53:30.384 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.385 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.385 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:30.385 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.191091+0000", 2024-02-05T04:53:30.385 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.002959756 2024-02-05T04:53:30.385 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.385 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.385 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-02-05T04:53:30.385 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.191323+0000", 2024-02-05T04:53:30.385 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00023186100000000001 2024-02-05T04:53:30.385 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.385 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.385 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:30.385 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.191400+0000", 2024-02-05T04:53:30.385 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.7812000000000006e-05 2024-02-05T04:53:30.385 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:30.385 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:30.386 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:30.386 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.386 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.386 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.4634.0:245 3.es0 3:796382fe:::smithi19639362-29:head [rollback 3] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:53:30.386 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:29.186870+0000", 2024-02-05T04:53:30.386 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 1.1331084, 2024-02-05T04:53:30.386 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0071777610000000004, 2024-02-05T04:53:30.386 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:30.386 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:30.386 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-02-05T04:53:30.386 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.4634", 2024-02-05T04:53:30.386 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:30.386 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 245 2024-02-05T04:53:30.386 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.386 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:30.386 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.387 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:30.387 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186870+0000", 2024-02-05T04:53:30.387 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:30.387 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.387 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.387 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:30.387 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186870+0000", 2024-02-05T04:53:30.387 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:30.387 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.387 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.387 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:30.387 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186870+0000", 2024-02-05T04:53:30.387 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.0400000000000002e-07 2024-02-05T04:53:30.387 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.387 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.388 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:30.388 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186874+0000", 2024-02-05T04:53:30.388 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.348e-06 2024-02-05T04:53:30.388 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.388 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.388 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:30.388 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186878+0000", 2024-02-05T04:53:30.388 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.481e-06 2024-02-05T04:53:30.388 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.388 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.388 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:30.388 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186881+0000", 2024-02-05T04:53:30.388 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.1879999999999998e-06 2024-02-05T04:53:30.388 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.389 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.389 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:30.389 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.188893+0000", 2024-02-05T04:53:30.389 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.002011231 2024-02-05T04:53:30.389 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.389 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.389 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:53:30.389 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.189032+0000", 2024-02-05T04:53:30.389 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000139467 2024-02-05T04:53:30.389 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.389 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.389 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:30.389 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.189572+0000", 2024-02-05T04:53:30.389 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00053986300000000002 2024-02-05T04:53:30.389 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.389 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.390 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:30.390 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.192597+0000", 2024-02-05T04:53:30.390 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0030245659999999998 2024-02-05T04:53:30.390 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.390 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.390 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-02-05T04:53:30.390 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.193877+0000", 2024-02-05T04:53:30.390 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.001280109 2024-02-05T04:53:30.390 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.390 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.390 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:30.390 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.194048+0000", 2024-02-05T04:53:30.390 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000171204 2024-02-05T04:53:30.390 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:30.390 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:30.391 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:30.391 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.391 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.391 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.4634.0:246 3.es0 3:796382fe:::smithi19639362-29:head [append 0~0] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:53:30.391 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:29.186925+0000", 2024-02-05T04:53:30.391 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 1.1330538560000001, 2024-02-05T04:53:30.391 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0088703129999999995, 2024-02-05T04:53:30.391 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:30.391 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:30.391 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-02-05T04:53:30.391 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.4634", 2024-02-05T04:53:30.391 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:30.391 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 246 2024-02-05T04:53:30.391 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.391 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:30.392 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.392 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:30.392 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186925+0000", 2024-02-05T04:53:30.392 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:30.392 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.392 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.392 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:30.392 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186925+0000", 2024-02-05T04:53:30.392 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:30.392 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.392 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.392 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:30.392 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186925+0000", 2024-02-05T04:53:30.392 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.53e-07 2024-02-05T04:53:30.392 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.393 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.393 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:30.393 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186928+0000", 2024-02-05T04:53:30.393 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.6900000000000001e-06 2024-02-05T04:53:30.393 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.393 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.393 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:30.393 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186933+0000", 2024-02-05T04:53:30.393 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.502e-06 2024-02-05T04:53:30.393 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.393 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.393 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:30.393 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186937+0000", 2024-02-05T04:53:30.393 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.6200000000000001e-06 2024-02-05T04:53:30.393 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.394 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.394 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:30.394 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.190223+0000", 2024-02-05T04:53:30.394 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0032863050000000002 2024-02-05T04:53:30.394 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.394 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.394 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:53:30.394 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.190312+0000", 2024-02-05T04:53:30.394 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.8821999999999998e-05 2024-02-05T04:53:30.394 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.394 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.394 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:30.394 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.190713+0000", 2024-02-05T04:53:30.394 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00040077400000000001 2024-02-05T04:53:30.394 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.394 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.395 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:30.395 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.193148+0000", 2024-02-05T04:53:30.395 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.002435544 2024-02-05T04:53:30.395 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.395 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.395 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-02-05T04:53:30.395 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.195751+0000", 2024-02-05T04:53:30.395 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0026025000000000002 2024-02-05T04:53:30.395 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.395 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.395 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:30.395 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.195795+0000", 2024-02-05T04:53:30.395 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.4202999999999997e-05 2024-02-05T04:53:30.395 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:30.395 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:30.396 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:30.396 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.396 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.396 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.4634.0:249 3.es0 3:70a5b9dd:::smithi19639362-46:head [delete] snapc 3=[3] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:53:30.396 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:29.186943+0000", 2024-02-05T04:53:30.396 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 1.133035896, 2024-02-05T04:53:30.396 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.011398484, 2024-02-05T04:53:30.396 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:30.396 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:30.396 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-02-05T04:53:30.396 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.4634", 2024-02-05T04:53:30.396 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:30.396 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 249 2024-02-05T04:53:30.397 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.397 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:30.397 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.397 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:30.397 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186943+0000", 2024-02-05T04:53:30.397 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:30.397 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.397 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.397 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:30.397 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186943+0000", 2024-02-05T04:53:30.397 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:30.397 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.397 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.397 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:30.397 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186943+0000", 2024-02-05T04:53:30.397 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.1899999999999999e-07 2024-02-05T04:53:30.398 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.398 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.398 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:30.398 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186944+0000", 2024-02-05T04:53:30.398 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.207e-06 2024-02-05T04:53:30.398 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.398 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.398 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:30.398 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186948+0000", 2024-02-05T04:53:30.398 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.7129999999999999e-06 2024-02-05T04:53:30.398 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.398 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.398 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:30.398 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186951+0000", 2024-02-05T04:53:30.398 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.9979999999999999e-06 2024-02-05T04:53:30.399 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.399 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.399 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:30.399 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.191572+0000", 2024-02-05T04:53:30.399 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0046207669999999996 2024-02-05T04:53:30.399 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.399 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.399 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:53:30.399 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.191630+0000", 2024-02-05T04:53:30.399 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.8774999999999999e-05 2024-02-05T04:53:30.399 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.399 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.399 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:30.399 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.192049+0000", 2024-02-05T04:53:30.399 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00041888 2024-02-05T04:53:30.400 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.400 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.400 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:30.400 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.194188+0000", 2024-02-05T04:53:30.400 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0021391359999999998 2024-02-05T04:53:30.400 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.400 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.400 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-02-05T04:53:30.400 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.198284+0000", 2024-02-05T04:53:30.400 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0040960509999999999 2024-02-05T04:53:30.400 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.400 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.400 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:30.400 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.198341+0000", 2024-02-05T04:53:30.400 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.6737999999999998e-05 2024-02-05T04:53:30.401 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:30.401 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:30.401 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:30.401 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.401 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.401 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.2s1 37/31 ECSubWrite(tid=19, reqid=client.4634.0:251, at_version=37'11, trim_to=0'0, roll_forward_to=34'8))", 2024-02-05T04:53:30.401 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:29.200609+0000", 2024-02-05T04:53:30.401 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 1.119369791, 2024-02-05T04:53:30.401 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0073060429999999999, 2024-02-05T04:53:30.401 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:30.401 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:53:30.401 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:30.401 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.401 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:30.401 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.200609+0000", 2024-02-05T04:53:30.402 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:30.402 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.402 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.402 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:30.402 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.200609+0000", 2024-02-05T04:53:30.402 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:30.402 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.402 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.402 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:30.402 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.200609+0000", 2024-02-05T04:53:30.402 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.9599999999999999e-07 2024-02-05T04:53:30.402 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.402 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.402 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:30.402 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.200622+0000", 2024-02-05T04:53:30.402 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2833000000000001e-05 2024-02-05T04:53:30.403 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.403 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.403 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:30.403 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.200633+0000", 2024-02-05T04:53:30.403 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0509999999999999e-05 2024-02-05T04:53:30.403 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.403 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.403 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:30.403 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.200640+0000", 2024-02-05T04:53:30.403 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.2150000000000004e-06 2024-02-05T04:53:30.403 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.403 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.403 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:30.403 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.200685+0000", 2024-02-05T04:53:30.403 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.5571000000000001e-05 2024-02-05T04:53:30.404 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.404 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.404 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:30.404 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.200691+0000", 2024-02-05T04:53:30.404 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.5990000000000001e-06 2024-02-05T04:53:30.404 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.404 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.404 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:30.404 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.207875+0000", 2024-02-05T04:53:30.404 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0071843130000000003 2024-02-05T04:53:30.404 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.404 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.404 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:30.404 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.207915+0000", 2024-02-05T04:53:30.404 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.9505999999999997e-05 2024-02-05T04:53:30.405 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:30.405 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:30.405 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:30.405 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.405 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.405 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.2s1 37/31 ECSubWrite(tid=20, reqid=client.4634.0:252, at_version=37'12, trim_to=0'0, roll_forward_to=34'8))", 2024-02-05T04:53:30.405 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:29.201260+0000", 2024-02-05T04:53:30.405 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 1.11871878, 2024-02-05T04:53:30.405 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0066905130000000004, 2024-02-05T04:53:30.405 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:30.405 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:53:30.405 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:30.405 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.405 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:30.405 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.201260+0000", 2024-02-05T04:53:30.405 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:30.406 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.406 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.406 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:30.406 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.201260+0000", 2024-02-05T04:53:30.406 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:30.406 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.406 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.406 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:30.406 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.201260+0000", 2024-02-05T04:53:30.406 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.2500000000000005e-07 2024-02-05T04:53:30.406 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.406 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.406 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:30.406 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.201272+0000", 2024-02-05T04:53:30.406 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1105e-05 2024-02-05T04:53:30.407 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.407 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.407 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:30.407 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.201282+0000", 2024-02-05T04:53:30.407 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0572e-05 2024-02-05T04:53:30.407 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.407 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.407 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:30.407 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.201288+0000", 2024-02-05T04:53:30.407 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.9699999999999996e-06 2024-02-05T04:53:30.407 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.407 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.407 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:30.407 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.202134+0000", 2024-02-05T04:53:30.407 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00084635299999999999 2024-02-05T04:53:30.408 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.408 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.408 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:30.408 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.202140+0000", 2024-02-05T04:53:30.408 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.054e-06 2024-02-05T04:53:30.408 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.408 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.408 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:30.408 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.207923+0000", 2024-02-05T04:53:30.408 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0057820989999999997 2024-02-05T04:53:30.408 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:30.408 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:30.408 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:30.408 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.207950+0000", 2024-02-05T04:53:30.408 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.7735000000000001e-05 2024-02-05T04:53:30.408 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:30.409 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:30.409 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:30.409 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:30.409 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:30.409 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-02-05T04:53:30.410 INFO:tasks.ceph.osd.8.smithi062.stderr:2024-02-05T04:53:30.409+0000 7f9385bf2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 34693) UID: 0 2024-02-05T04:53:30.411 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-02-05T04:53:30.412 INFO:tasks.rados.rados.0.smithi196.stdout:99: finishing write tid 1 to smithi19639362-4 2024-02-05T04:53:30.412 INFO:tasks.rados.rados.0.smithi196.stdout:99: finishing write tid 2 to smithi19639362-4 2024-02-05T04:53:30.412 INFO:tasks.rados.rados.0.smithi196.stdout:99: finishing write tid 3 to smithi19639362-4 2024-02-05T04:53:30.412 INFO:tasks.rados.rados.0.smithi196.stdout:100: expect (ObjNum 0 snap 0 seq_num 0) 2024-02-05T04:53:30.413 INFO:tasks.rados.rados.0.smithi196.stdout:99: finishing write tid 4 to smithi19639362-4 2024-02-05T04:53:30.413 INFO:tasks.rados.rados.0.smithi196.stdout:99: oid 4 updating version 0 to 4 2024-02-05T04:53:30.413 INFO:tasks.rados.rados.0.smithi196.stdout:99: oid 4 updating version 4 to 5 2024-02-05T04:53:30.413 INFO:tasks.rados.rados.0.smithi196.stdout:99: oid 4 version 5 is already newer than 3 2024-02-05T04:53:30.413 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 4 v 5 (ObjNum 67 snap 2 seq_num 67) dirty exists 2024-02-05T04:53:30.413 INFO:tasks.rados.rados.0.smithi196.stdout:99: left oid 4 (ObjNum 67 snap 2 seq_num 67) 2024-02-05T04:53:30.413 INFO:tasks.rados.rados.0.smithi196.stdout:107: finishing write tid 1 to smithi19639362-17 2024-02-05T04:53:30.413 INFO:tasks.rados.rados.0.smithi196.stdout:107: finishing write tid 2 to smithi19639362-17 2024-02-05T04:53:30.413 INFO:tasks.rados.rados.0.smithi196.stdout:107: finishing write tid 3 to smithi19639362-17 2024-02-05T04:53:30.413 INFO:tasks.rados.rados.0.smithi196.stdout:107: finishing write tid 4 to smithi19639362-17 2024-02-05T04:53:30.413 INFO:tasks.rados.rados.0.smithi196.stdout:107: finishing write tid 5 to smithi19639362-17 2024-02-05T04:53:30.413 INFO:tasks.rados.rados.0.smithi196.stdout:107: oid 17 updating version 0 to 20 2024-02-05T04:53:30.413 INFO:tasks.rados.rados.0.smithi196.stdout:107: oid 17 version 20 is already newer than 18 2024-02-05T04:53:30.413 INFO:tasks.rados.rados.0.smithi196.stdout:107: oid 17 version 20 is already newer than 17 2024-02-05T04:53:30.413 INFO:tasks.rados.rados.0.smithi196.stdout:107: oid 17 version 20 is already newer than 19 2024-02-05T04:53:30.413 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 17 v 20 (ObjNum 71 snap 2 seq_num 71) dirty exists 2024-02-05T04:53:30.414 INFO:tasks.rados.rados.0.smithi196.stdout:107: left oid 17 (ObjNum 71 snap 2 seq_num 71) 2024-02-05T04:53:30.437 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T04:53:30.511 INFO:tasks.ceph.osd.8.smithi062.stderr:2024-02-05T04:53:30.511+0000 7f9385bf2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 34693) UID: 0 2024-02-05T04:53:30.518 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-02-05T04:53:30.518 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [], 2024-02-05T04:53:30.518 INFO:teuthology.orchestra.run.smithi150.stdout: "num_ops": 0 2024-02-05T04:53:30.518 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-02-05T04:53:30.526 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-02-05T04:53:30.560 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0: {} 2024-02-05T04:53:30.561 INFO:teuthology.orchestra.run.smithi062.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T04:53:30.579 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1: {} 2024-02-05T04:53:30.579 INFO:teuthology.orchestra.run.smithi062.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T04:53:30.596 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2: {} 2024-02-05T04:53:30.596 INFO:teuthology.orchestra.run.smithi062.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T04:53:30.613 INFO:tasks.ceph.osd.14.smithi150.stderr:2024-02-05T04:53:30.612+0000 7f5f81083640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 34020) UID: 0 2024-02-05T04:53:30.614 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3: {} 2024-02-05T04:53:30.614 INFO:teuthology.orchestra.run.smithi062.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T04:53:30.631 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4: {} 2024-02-05T04:53:30.631 INFO:teuthology.orchestra.run.smithi062.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T04:53:30.633 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-02-05T04:53:30.633 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [], 2024-02-05T04:53:30.633 INFO:teuthology.orchestra.run.smithi150.stdout: "complaint_time": 30, 2024-02-05T04:53:30.633 INFO:teuthology.orchestra.run.smithi150.stdout: "num_blocked_ops": 0 2024-02-05T04:53:30.633 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-02-05T04:53:30.642 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-02-05T04:53:30.649 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5: {} 2024-02-05T04:53:30.649 INFO:teuthology.orchestra.run.smithi062.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T04:53:30.667 INFO:teuthology.orchestra.run.smithi062.stdout:osd.6: {} 2024-02-05T04:53:30.667 INFO:teuthology.orchestra.run.smithi062.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T04:53:30.684 INFO:teuthology.orchestra.run.smithi062.stdout:osd.7: {} 2024-02-05T04:53:30.685 INFO:teuthology.orchestra.run.smithi062.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T04:53:30.701 INFO:teuthology.orchestra.run.smithi062.stdout:osd.8: {} 2024-02-05T04:53:30.702 INFO:teuthology.orchestra.run.smithi062.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-05T04:53:30.714 INFO:tasks.ceph.osd.4.smithi062.stderr:2024-02-05T04:53:30.713+0000 7f708c308640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34689) UID: 0 2024-02-05T04:53:30.719 INFO:teuthology.orchestra.run.smithi062.stdout:osd.9: {} 2024-02-05T04:53:30.719 INFO:teuthology.orchestra.run.smithi062.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-05T04:53:30.737 INFO:teuthology.orchestra.run.smithi062.stdout:osd.10: {} 2024-02-05T04:53:30.737 INFO:teuthology.orchestra.run.smithi062.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-05T04:53:30.755 INFO:teuthology.orchestra.run.smithi062.stdout:osd.11: {} 2024-02-05T04:53:30.755 INFO:teuthology.orchestra.run.smithi062.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-05T04:53:30.772 INFO:teuthology.orchestra.run.smithi062.stdout:osd.12: {} 2024-02-05T04:53:30.772 INFO:teuthology.orchestra.run.smithi062.stderr:osd.12: osd_enable_op_tracker = '' 2024-02-05T04:53:30.788 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:30.789 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:30.789 INFO:teuthology.orchestra.run.smithi062.stdout:osd.13: {} 2024-02-05T04:53:30.789 INFO:teuthology.orchestra.run.smithi062.stderr:osd.13: osd_enable_op_tracker = '' 2024-02-05T04:53:30.797 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-02-05T04:53:30.807 INFO:teuthology.orchestra.run.smithi062.stdout:osd.14: {} 2024-02-05T04:53:30.807 INFO:teuthology.orchestra.run.smithi062.stderr:osd.14: osd_enable_op_tracker = '' 2024-02-05T04:53:30.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-02-05T04:53:30.806+0000 7f0822d9d640 0 -- 172.21.15.62:0/3676096263 >> [v2:172.21.15.196:6826/3187639683,v1:172.21.15.196:6827/3187639683] conn(0x7f0824221d10 msgr2=0x7f0824224100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-02-05T04:53:30.814 INFO:tasks.ceph.osd.11.smithi196.stderr:2024-02-05T04:53:30.813+0000 7f85a8d86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 34609) UID: 0 2024-02-05T04:53:30.903 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:30.904 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:30.912 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-02-05T04:53:30.915 INFO:tasks.ceph.osd.14.smithi150.stderr:2024-02-05T04:53:30.914+0000 7f5f81083640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 34020) UID: 0 2024-02-05T04:53:31.016 INFO:tasks.ceph.osd.1.smithi091.stderr:2024-02-05T04:53:31.016+0000 7ff9dc283640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34275) UID: 0 2024-02-05T04:53:31.018 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:31.018 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:31.026 INFO:teuthology.orchestra.run.smithi062.stdout:osd.15: {} 2024-02-05T04:53:31.026 INFO:teuthology.orchestra.run.smithi062.stderr:osd.15: osd_enable_op_tracker = '' 2024-02-05T04:53:31.027 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-02-05T04:53:31.117 INFO:tasks.ceph.osd.2.smithi150.stderr:2024-02-05T04:53:31.115+0000 7f7337bf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 34014) UID: 0 2024-02-05T04:53:31.128 INFO:teuthology.orchestra.run.smithi062.stderr:reweighted osd.10 to 1 (10000) 2024-02-05T04:53:31.132 INFO:tasks.rados.rados.0.smithi196.stdout:99: done (6 left) 2024-02-05T04:53:31.132 INFO:tasks.rados.rados.0.smithi196.stdout:100: done (5 left) 2024-02-05T04:53:31.132 INFO:tasks.rados.rados.0.smithi196.stdout:104: done (4 left) 2024-02-05T04:53:31.132 INFO:tasks.rados.rados.0.smithi196.stdout:105: done (3 left) 2024-02-05T04:53:31.132 INFO:tasks.rados.rados.0.smithi196.stdout:106: done (2 left) 2024-02-05T04:53:31.132 INFO:tasks.rados.rados.0.smithi196.stdout:107: done (1 left) 2024-02-05T04:53:31.132 INFO:tasks.rados.rados.0.smithi196.stdout:108: done (0 left) 2024-02-05T04:53:31.132 INFO:tasks.rados.rados.0.smithi196.stdout:append oid 1 current snap is 3 2024-02-05T04:53:31.132 INFO:tasks.rados.rados.0.smithi196.stdout:109: seq_num 72 ranges {32768=442368,475136=720896,1196032=49152} 2024-02-05T04:53:31.134 INFO:tasks.rados.rados.0.smithi196.stdout:109: writing smithi19639362-1 from 32768 to 475136 tid 1 2024-02-05T04:53:31.136 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:31.136 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:31.139 INFO:tasks.rados.rados.0.smithi196.stdout:109: writing smithi19639362-1 from 475136 to 1196032 tid 2 2024-02-05T04:53:31.139 INFO:tasks.rados.rados.0.smithi196.stdout:109: writing smithi19639362-1 from 1196032 to 1245184 tid 3 2024-02-05T04:53:31.139 INFO:tasks.rados.rados.0.smithi196.stdout:append oid 23 current snap is 3 2024-02-05T04:53:31.139 INFO:tasks.rados.rados.0.smithi196.stdout:110: seq_num 73 ranges {1441792=458752,1900544=475136,2375680=294912} 2024-02-05T04:53:31.144 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 11 1 2024-02-05T04:53:31.147 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-02-05T04:53:31.149 INFO:tasks.rados.rados.0.smithi196.stdout:110: writing smithi19639362-23 from 1441792 to 1900544 tid 1 2024-02-05T04:53:31.152 INFO:tasks.rados.rados.0.smithi196.stdout:110: writing smithi19639362-23 from 1900544 to 2375680 tid 2 2024-02-05T04:53:31.155 INFO:tasks.rados.rados.0.smithi196.stdout:110: writing smithi19639362-23 from 2375680 to 2670592 tid 3 2024-02-05T04:53:31.155 INFO:tasks.rados.rados.0.smithi196.stdout:append oid 37 current snap is 3 2024-02-05T04:53:31.155 INFO:tasks.rados.rados.0.smithi196.stdout:111: seq_num 74 ranges {606208=475136,1081344=622592,1703936=147456} 2024-02-05T04:53:31.162 INFO:tasks.rados.rados.0.smithi196.stdout:111: writing smithi19639362-37 from 606208 to 1081344 tid 1 2024-02-05T04:53:31.165 INFO:tasks.rados.rados.0.smithi196.stdout:111: writing smithi19639362-37 from 1081344 to 1703936 tid 2 2024-02-05T04:53:31.165 INFO:tasks.rados.rados.0.smithi196.stdout:111: writing smithi19639362-37 from 1703936 to 1851392 tid 3 2024-02-05T04:53:31.166 INFO:tasks.rados.rados.0.smithi196.stdout:112: rollback oid 34 current snap is 3 2024-02-05T04:53:31.166 INFO:tasks.rados.rados.0.smithi196.stdout:rollback oid 34 to 2 2024-02-05T04:53:31.166 INFO:tasks.rados.rados.0.smithi196.stdout:append oid 2 current snap is 3 2024-02-05T04:53:31.166 INFO:tasks.rados.rados.0.smithi196.stdout:113: seq_num 75 ranges {32768=491520,524288=770048} 2024-02-05T04:53:31.169 INFO:tasks.rados.rados.0.smithi196.stdout:113: writing smithi19639362-2 from 32768 to 524288 tid 1 2024-02-05T04:53:31.174 INFO:tasks.rados.rados.0.smithi196.stdout:113: writing smithi19639362-2 from 524288 to 1294336 tid 2 2024-02-05T04:53:31.174 INFO:tasks.rados.rados.0.smithi196.stdout:append oid 26 current snap is 3 2024-02-05T04:53:31.174 INFO:tasks.rados.rados.0.smithi196.stdout:109: finishing write tid 1 to smithi19639362-1 2024-02-05T04:53:31.174 INFO:tasks.rados.rados.0.smithi196.stdout:114: seq_num 76 ranges {0=507904,507904=507904,1015808=262144} 2024-02-05T04:53:31.177 INFO:tasks.rados.rados.0.smithi196.stdout:114: writing smithi19639362-26 from 0 to 507904 tid 1 2024-02-05T04:53:31.180 INFO:tasks.rados.rados.0.smithi196.stdout:114: writing smithi19639362-26 from 507904 to 1015808 tid 2 2024-02-05T04:53:31.183 INFO:tasks.rados.rados.0.smithi196.stdout:114: writing smithi19639362-26 from 1015808 to 1277952 tid 3 2024-02-05T04:53:31.183 INFO:tasks.rados.rados.0.smithi196.stdout:109: finishing write tid 2 to smithi19639362-1 2024-02-05T04:53:31.183 INFO:tasks.rados.rados.0.smithi196.stdout:109: finishing write tid 3 to smithi19639362-1 2024-02-05T04:53:31.183 INFO:tasks.rados.rados.0.smithi196.stdout:109: finishing write tid 4 to smithi19639362-1 2024-02-05T04:53:31.183 INFO:tasks.rados.rados.0.smithi196.stdout:109: finishing write tid 5 to smithi19639362-1 2024-02-05T04:53:31.183 INFO:tasks.rados.rados.0.smithi196.stdout:109: oid 1 updating version 0 to 22 2024-02-05T04:53:31.183 INFO:tasks.rados.rados.0.smithi196.stdout:109: oid 1 updating version 22 to 23 2024-02-05T04:53:31.183 INFO:tasks.rados.rados.0.smithi196.stdout:109: oid 1 updating version 23 to 25 2024-02-05T04:53:31.183 INFO:tasks.rados.rados.0.smithi196.stdout:109: oid 1 version 25 is already newer than 24 2024-02-05T04:53:31.183 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 1 v 25 (ObjNum 72 snap 3 seq_num 72) dirty exists 2024-02-05T04:53:31.183 INFO:tasks.rados.rados.0.smithi196.stdout:109: left oid 1 (ObjNum 72 snap 3 seq_num 72) 2024-02-05T04:53:31.184 INFO:tasks.rados.rados.0.smithi196.stdout:110: finishing write tid 1 to smithi19639362-23 2024-02-05T04:53:31.184 INFO:tasks.rados.rados.0.smithi196.stdout:110: finishing write tid 2 to smithi19639362-23 2024-02-05T04:53:31.184 INFO:tasks.rados.rados.0.smithi196.stdout:110: finishing write tid 3 to smithi19639362-23 2024-02-05T04:53:31.184 INFO:tasks.rados.rados.0.smithi196.stdout:110: finishing write tid 4 to smithi19639362-23 2024-02-05T04:53:31.184 INFO:tasks.rados.rados.0.smithi196.stdout:115: read oid 25 snap -1 2024-02-05T04:53:31.184 INFO:tasks.rados.rados.0.smithi196.stdout:115: expect (ObjNum 38 snap 0 seq_num 38) 2024-02-05T04:53:31.184 INFO:tasks.rados.rados.0.smithi196.stdout:109: done (6 left) 2024-02-05T04:53:31.184 INFO:tasks.rados.rados.0.smithi196.stdout:116: copy_from oid 11 from oid 38 current snap is 3 2024-02-05T04:53:31.184 INFO:tasks.rados.rados.0.smithi196.stdout:110: finishing write tid 5 to smithi19639362-23 2024-02-05T04:53:31.184 INFO:tasks.rados.rados.0.smithi196.stdout:110: oid 23 updating version 0 to 26 2024-02-05T04:53:31.184 INFO:tasks.rados.rados.0.smithi196.stdout:110: oid 23 updating version 26 to 28 2024-02-05T04:53:31.184 INFO:tasks.rados.rados.0.smithi196.stdout:110: oid 23 version 28 is already newer than 27 2024-02-05T04:53:31.184 INFO:tasks.rados.rados.0.smithi196.stdout:110: oid 23 updating version 28 to 29 2024-02-05T04:53:31.185 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 23 v 29 (ObjNum 73 snap 3 seq_num 73) dirty exists 2024-02-05T04:53:31.185 INFO:tasks.rados.rados.0.smithi196.stdout:110: left oid 23 (ObjNum 73 snap 3 seq_num 73) 2024-02-05T04:53:31.185 INFO:tasks.rados.rados.0.smithi196.stdout:112: finishing rollback tid 0 to smithi19639362-34 2024-02-05T04:53:31.185 INFO:tasks.rados.rados.0.smithi196.stdout:111: finishing write tid 1 to smithi19639362-37 2024-02-05T04:53:31.185 INFO:tasks.rados.rados.0.smithi196.stdout:111: finishing write tid 2 to smithi19639362-37 2024-02-05T04:53:31.185 INFO:tasks.rados.rados.0.smithi196.stdout:110: done (6 left) 2024-02-05T04:53:31.185 INFO:tasks.rados.rados.0.smithi196.stdout:117: rollback oid 16 current snap is 3 2024-02-05T04:53:31.185 INFO:tasks.rados.rados.0.smithi196.stdout:rollback oid 16 to 2 2024-02-05T04:53:31.185 INFO:tasks.rados.rados.0.smithi196.stdout:111: finishing write tid 3 to smithi19639362-37 2024-02-05T04:53:31.186 INFO:tasks.rados.rados.0.smithi196.stdout:112: finishing rollback tid 1 to smithi19639362-34 2024-02-05T04:53:31.186 INFO:tasks.rados.rados.0.smithi196.stdout:112: finishing rollback tid 2 to smithi19639362-34 2024-02-05T04:53:31.186 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 34 v 33 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-02-05T04:53:31.186 INFO:tasks.rados.rados.0.smithi196.stdout:111: finishing write tid 4 to smithi19639362-37 2024-02-05T04:53:31.186 INFO:tasks.rados.rados.0.smithi196.stdout:111: finishing write tid 5 to smithi19639362-37 2024-02-05T04:53:31.186 INFO:tasks.rados.rados.0.smithi196.stdout:111: oid 37 updating version 0 to 17 2024-02-05T04:53:31.186 INFO:tasks.rados.rados.0.smithi196.stdout:111: oid 37 version 17 is already newer than 14 2024-02-05T04:53:31.186 INFO:tasks.rados.rados.0.smithi196.stdout:111: oid 37 version 17 is already newer than 15 2024-02-05T04:53:31.186 INFO:tasks.rados.rados.0.smithi196.stdout:111: oid 37 version 17 is already newer than 16 2024-02-05T04:53:31.186 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 37 v 17 (ObjNum 74 snap 3 seq_num 74) dirty exists 2024-02-05T04:53:31.186 INFO:tasks.rados.rados.0.smithi196.stdout:111: left oid 37 (ObjNum 74 snap 3 seq_num 74) 2024-02-05T04:53:31.186 INFO:tasks.rados.rados.0.smithi196.stdout:111: done (6 left) 2024-02-05T04:53:31.186 INFO:tasks.rados.rados.0.smithi196.stdout:112: done (5 left) 2024-02-05T04:53:31.186 INFO:tasks.rados.rados.0.smithi196.stdout:118: rmattr oid 15 current snap is 3 2024-02-05T04:53:31.186 INFO:tasks.rados.rados.0.smithi196.stdout:118: done (5 left) 2024-02-05T04:53:31.187 INFO:tasks.rados.rados.0.smithi196.stdout:119: rmattr oid 1 current snap is 3 2024-02-05T04:53:31.187 INFO:tasks.rados.rados.0.smithi196.stdout:119: done (5 left) 2024-02-05T04:53:31.187 INFO:tasks.rados.rados.0.smithi196.stdout:120: snap_create 2024-02-05T04:53:31.187 INFO:tasks.rados.rados.0.smithi196.stdout:113: finishing write tid 1 to smithi19639362-2 2024-02-05T04:53:31.187 INFO:tasks.rados.rados.0.smithi196.stdout:113: finishing write tid 2 to smithi19639362-2 2024-02-05T04:53:31.187 INFO:tasks.rados.rados.0.smithi196.stdout:113: finishing write tid 3 to smithi19639362-2 2024-02-05T04:53:31.188 INFO:tasks.rados.rados.0.smithi196.stdout:113: finishing write tid 4 to smithi19639362-2 2024-02-05T04:53:31.188 INFO:tasks.rados.rados.0.smithi196.stdout:113: oid 2 updating version 0 to 19 2024-02-05T04:53:31.188 INFO:tasks.rados.rados.0.smithi196.stdout:113: oid 2 version 19 is already newer than 17 2024-02-05T04:53:31.188 INFO:tasks.rados.rados.0.smithi196.stdout:113: oid 2 version 19 is already newer than 18 2024-02-05T04:53:31.188 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 2 v 19 (ObjNum 75 snap 3 seq_num 75) dirty exists 2024-02-05T04:53:31.188 INFO:tasks.rados.rados.0.smithi196.stdout:113: left oid 2 (ObjNum 75 snap 3 seq_num 75) 2024-02-05T04:53:31.188 INFO:tasks.rados.rados.0.smithi196.stdout:114: finishing write tid 1 to smithi19639362-26 2024-02-05T04:53:31.188 INFO:tasks.rados.rados.0.smithi196.stdout:117: finishing rollback tid 0 to smithi19639362-16 2024-02-05T04:53:31.188 INFO:tasks.rados.rados.0.smithi196.stdout:114: finishing write tid 2 to smithi19639362-26 2024-02-05T04:53:31.188 INFO:tasks.rados.rados.0.smithi196.stdout:114: finishing write tid 3 to smithi19639362-26 2024-02-05T04:53:31.189 INFO:tasks.rados.rados.0.smithi196.stdout:117: finishing rollback tid 1 to smithi19639362-16 2024-02-05T04:53:31.189 INFO:tasks.rados.rados.0.smithi196.stdout:117: finishing rollback tid 2 to smithi19639362-16 2024-02-05T04:53:31.189 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 16 v 20 (ObjNum 60 snap 2 seq_num 60) dirty exists 2024-02-05T04:53:31.189 INFO:tasks.rados.rados.0.smithi196.stdout:114: finishing write tid 4 to smithi19639362-26 2024-02-05T04:53:31.189 INFO:tasks.rados.rados.0.smithi196.stdout:115: expect (ObjNum 38 snap 0 seq_num 38) 2024-02-05T04:53:31.206 INFO:tasks.rados.rados.0.smithi196.stdout:114: finishing write tid 5 to smithi19639362-26 2024-02-05T04:53:31.206 INFO:tasks.rados.rados.0.smithi196.stdout:114: oid 26 updating version 0 to 13 2024-02-05T04:53:31.206 INFO:tasks.rados.rados.0.smithi196.stdout:114: oid 26 updating version 13 to 15 2024-02-05T04:53:31.206 INFO:tasks.rados.rados.0.smithi196.stdout:114: oid 26 version 15 is already newer than 14 2024-02-05T04:53:31.206 INFO:tasks.rados.rados.0.smithi196.stdout:114: oid 26 version 15 is already newer than 12 2024-02-05T04:53:31.207 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 26 v 15 (ObjNum 76 snap 3 seq_num 76) dirty exists 2024-02-05T04:53:31.207 INFO:tasks.rados.rados.0.smithi196.stdout:114: left oid 26 (ObjNum 76 snap 3 seq_num 76) 2024-02-05T04:53:31.207 INFO:tasks.rados.rados.0.smithi196.stdout:116: finishing copy_from racing read to smithi19639362-11 2024-02-05T04:53:31.207 INFO:tasks.rados.rados.0.smithi196.stdout:116: finishing copy_from to smithi19639362-11 2024-02-05T04:53:31.207 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 11 v 4 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-02-05T04:53:31.217 INFO:tasks.ceph.osd.15.smithi196.stderr:2024-02-05T04:53:31.216+0000 7f1facc45640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 34611) UID: 0 2024-02-05T04:53:31.295 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:31.295 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:31.303 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-02-05T04:53:31.318 INFO:tasks.ceph.osd.5.smithi091.stderr:2024-02-05T04:53:31.317+0000 7f63983b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34277) UID: 0 2024-02-05T04:53:31.409 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:31.409 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:31.418 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-02-05T04:53:31.419 INFO:tasks.ceph.osd.0.smithi062.stderr:2024-02-05T04:53:31.418+0000 7fece8e03640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34688) UID: 0 2024-02-05T04:53:31.520 INFO:tasks.ceph.osd.15.smithi196.stderr:2024-02-05T04:53:31.520+0000 7f1facc45640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 34611) UID: 0 2024-02-05T04:53:31.524 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:31.524 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:31.533 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-02-05T04:53:31.621 INFO:tasks.ceph.osd.4.smithi062.stderr:2024-02-05T04:53:31.620+0000 7f708c308640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34689) UID: 0 2024-02-05T04:53:31.645 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:31.645 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:31.654 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-02-05T04:53:31.722 INFO:tasks.ceph.osd.10.smithi150.stderr:2024-02-05T04:53:31.721+0000 7f3075a4d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 34018) UID: 0 2024-02-05T04:53:31.760 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:31.760 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:31.769 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-02-05T04:53:31.823 INFO:tasks.ceph.osd.3.smithi196.stderr:2024-02-05T04:53:31.822+0000 7fc8817c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34605) UID: 0 2024-02-05T04:53:31.875 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:31.875 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:31.885 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-02-05T04:53:31.926 INFO:tasks.ceph.osd.15.smithi196.stderr:2024-02-05T04:53:31.925+0000 7f1facc45640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 34611) UID: 0 2024-02-05T04:53:32.025 INFO:tasks.ceph.osd.0.smithi062.stderr:2024-02-05T04:53:32.025+0000 7fece8e03640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34688) UID: 0 2024-02-05T04:53:32.031 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:32.031 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:32.039 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-02-05T04:53:32.126 INFO:tasks.ceph.osd.10.smithi150.stderr:2024-02-05T04:53:32.125+0000 7f3075a4d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 34018) UID: 0 2024-02-05T04:53:32.137 INFO:teuthology.orchestra.run.smithi062.stderr:reweighted osd.11 to 1 (10000) 2024-02-05T04:53:32.143 INFO:tasks.rados.rados.0.smithi196.stdout:113: done (5 left) 2024-02-05T04:53:32.143 INFO:tasks.rados.rados.0.smithi196.stdout:114: done (4 left) 2024-02-05T04:53:32.143 INFO:tasks.rados.rados.0.smithi196.stdout:115: done (3 left) 2024-02-05T04:53:32.143 INFO:tasks.rados.rados.0.smithi196.stdout:116: done (2 left) 2024-02-05T04:53:32.143 INFO:tasks.rados.rados.0.smithi196.stdout:117: done (1 left) 2024-02-05T04:53:32.143 INFO:tasks.rados.rados.0.smithi196.stdout:120: done (0 left) 2024-02-05T04:53:32.143 INFO:tasks.rados.rados.0.smithi196.stdout:append oid 26 current snap is 4 2024-02-05T04:53:32.144 INFO:tasks.rados.rados.0.smithi196.stdout:121: seq_num 77 ranges {1277952=524288,1802240=524288,2326528=245760} 2024-02-05T04:53:32.147 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:32.147 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:32.151 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 12 1 2024-02-05T04:53:32.152 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-02-05T04:53:32.154 INFO:tasks.rados.rados.0.smithi196.stdout:121: writing smithi19639362-26 from 1277952 to 1802240 tid 1 2024-02-05T04:53:32.157 INFO:tasks.rados.rados.0.smithi196.stdout:121: writing smithi19639362-26 from 1802240 to 2326528 tid 2 2024-02-05T04:53:32.160 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-02-05T04:53:32.161 INFO:tasks.rados.rados.0.smithi196.stdout:121: writing smithi19639362-26 from 2326528 to 2572288 tid 3 2024-02-05T04:53:32.161 INFO:tasks.rados.rados.0.smithi196.stdout:122: copy_from oid 32 from oid 15 current snap is 4 2024-02-05T04:53:32.161 INFO:tasks.rados.rados.0.smithi196.stdout:123: read oid 5 snap -1 2024-02-05T04:53:32.161 INFO:tasks.rados.rados.0.smithi196.stdout:123: expect (ObjNum 4 snap 0 seq_num 4) 2024-02-05T04:53:32.161 INFO:tasks.rados.rados.0.smithi196.stdout:124: rollback oid 8 current snap is 4 2024-02-05T04:53:32.161 INFO:tasks.rados.rados.0.smithi196.stdout:rollback oid 8 to 2 2024-02-05T04:53:32.161 INFO:tasks.rados.rados.0.smithi196.stdout:append oid 29 current snap is 4 2024-02-05T04:53:32.161 INFO:tasks.rados.rados.0.smithi196.stdout:125: seq_num 78 ranges {1425408=557056,1982464=671744,2654208=98304} 2024-02-05T04:53:32.171 INFO:tasks.rados.rados.0.smithi196.stdout:125: writing smithi19639362-29 from 1425408 to 1982464 tid 1 2024-02-05T04:53:32.175 INFO:tasks.rados.rados.0.smithi196.stdout:125: writing smithi19639362-29 from 1982464 to 2654208 tid 2 2024-02-05T04:53:32.175 INFO:tasks.rados.rados.0.smithi196.stdout:125: writing smithi19639362-29 from 2654208 to 2752512 tid 3 2024-02-05T04:53:32.176 INFO:tasks.rados.rados.0.smithi196.stdout:append oid 39 current snap is 4 2024-02-05T04:53:32.176 INFO:tasks.rados.rados.0.smithi196.stdout:122: finishing copy_from to smithi19639362-32 2024-02-05T04:53:32.176 INFO:tasks.rados.rados.0.smithi196.stdout:122: got expected ENOENT (src dne) 2024-02-05T04:53:32.176 INFO:tasks.rados.rados.0.smithi196.stdout:126: seq_num 79 ranges {638976=573440,1212416=425984,1638400=344064} 2024-02-05T04:53:32.182 INFO:tasks.rados.rados.0.smithi196.stdout:126: writing smithi19639362-39 from 638976 to 1212416 tid 1 2024-02-05T04:53:32.185 INFO:tasks.rados.rados.0.smithi196.stdout:126: writing smithi19639362-39 from 1212416 to 1638400 tid 2 2024-02-05T04:53:32.187 INFO:tasks.rados.rados.0.smithi196.stdout:126: writing smithi19639362-39 from 1638400 to 1982464 tid 3 2024-02-05T04:53:32.187 INFO:tasks.rados.rados.0.smithi196.stdout:127: rollback oid 21 current snap is 4 2024-02-05T04:53:32.187 INFO:tasks.rados.rados.0.smithi196.stdout:rollback oid 21 to 2 2024-02-05T04:53:32.188 INFO:tasks.rados.rados.0.smithi196.stdout:122: finishing copy_from racing read to smithi19639362-32 2024-02-05T04:53:32.188 INFO:tasks.rados.rados.0.smithi196.stdout:124: finishing rollback tid 0 to smithi19639362-8 2024-02-05T04:53:32.188 INFO:tasks.rados.rados.0.smithi196.stdout:123: expect (ObjNum 4 snap 0 seq_num 4) 2024-02-05T04:53:32.191 INFO:tasks.rados.rados.0.smithi196.stdout:121: finishing write tid 1 to smithi19639362-26 2024-02-05T04:53:32.191 INFO:tasks.rados.rados.0.smithi196.stdout:124: finishing rollback tid 1 to smithi19639362-8 2024-02-05T04:53:32.191 INFO:tasks.rados.rados.0.smithi196.stdout:124: finishing rollback tid 2 to smithi19639362-8 2024-02-05T04:53:32.191 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 8 v 21 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-02-05T04:53:32.192 INFO:tasks.rados.rados.0.smithi196.stdout:121: finishing write tid 2 to smithi19639362-26 2024-02-05T04:53:32.192 INFO:tasks.rados.rados.0.smithi196.stdout:121: finishing write tid 3 to smithi19639362-26 2024-02-05T04:53:32.192 INFO:tasks.rados.rados.0.smithi196.stdout:121: finishing write tid 4 to smithi19639362-26 2024-02-05T04:53:32.192 INFO:tasks.rados.rados.0.smithi196.stdout:121: finishing write tid 5 to smithi19639362-26 2024-02-05T04:53:32.192 INFO:tasks.rados.rados.0.smithi196.stdout:121: oid 26 updating version 0 to 21 2024-02-05T04:53:32.192 INFO:tasks.rados.rados.0.smithi196.stdout:121: oid 26 updating version 21 to 22 2024-02-05T04:53:32.192 INFO:tasks.rados.rados.0.smithi196.stdout:121: oid 26 version 22 is already newer than 19 2024-02-05T04:53:32.192 INFO:tasks.rados.rados.0.smithi196.stdout:121: oid 26 version 22 is already newer than 20 2024-02-05T04:53:32.192 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 26 v 22 (ObjNum 77 snap 4 seq_num 77) dirty exists 2024-02-05T04:53:32.192 INFO:tasks.rados.rados.0.smithi196.stdout:121: left oid 26 (ObjNum 77 snap 4 seq_num 77) 2024-02-05T04:53:32.192 INFO:tasks.rados.rados.0.smithi196.stdout:121: done (6 left) 2024-02-05T04:53:32.192 INFO:tasks.rados.rados.0.smithi196.stdout:122: done (5 left) 2024-02-05T04:53:32.192 INFO:tasks.rados.rados.0.smithi196.stdout:123: done (4 left) 2024-02-05T04:53:32.193 INFO:tasks.rados.rados.0.smithi196.stdout:124: done (3 left) 2024-02-05T04:53:32.193 INFO:tasks.rados.rados.0.smithi196.stdout:128: read oid 47 snap -1 2024-02-05T04:53:32.193 INFO:tasks.rados.rados.0.smithi196.stdout:128: expect (ObjNum 46 snap 0 seq_num 46) 2024-02-05T04:53:32.193 INFO:tasks.rados.rados.0.smithi196.stdout:append oid 30 current snap is 4 2024-02-05T04:53:32.193 INFO:tasks.rados.rados.0.smithi196.stdout:125: finishing write tid 1 to smithi19639362-29 2024-02-05T04:53:32.193 INFO:tasks.rados.rados.0.smithi196.stdout:125: finishing write tid 2 to smithi19639362-29 2024-02-05T04:53:32.193 INFO:tasks.rados.rados.0.smithi196.stdout:125: finishing write tid 3 to smithi19639362-29 2024-02-05T04:53:32.193 INFO:tasks.rados.rados.0.smithi196.stdout:125: finishing write tid 4 to smithi19639362-29 2024-02-05T04:53:32.193 INFO:tasks.rados.rados.0.smithi196.stdout:129: seq_num 80 ranges {491520=589824,1081344=573440,1654784=196608} 2024-02-05T04:53:32.197 INFO:tasks.rados.rados.0.smithi196.stdout:129: writing smithi19639362-30 from 491520 to 1081344 tid 1 2024-02-05T04:53:32.200 INFO:tasks.rados.rados.0.smithi196.stdout:129: writing smithi19639362-30 from 1081344 to 1654784 tid 2 2024-02-05T04:53:32.202 INFO:tasks.rados.rados.0.smithi196.stdout:129: writing smithi19639362-30 from 1654784 to 1851392 tid 3 2024-02-05T04:53:32.202 INFO:tasks.rados.rados.0.smithi196.stdout:append oid 12 current snap is 4 2024-02-05T04:53:32.202 INFO:tasks.rados.rados.0.smithi196.stdout:130: seq_num 81 ranges {1064960=606208,1671168=720896,2392064=49152} 2024-02-05T04:53:32.211 INFO:tasks.rados.rados.0.smithi196.stdout:130: writing smithi19639362-12 from 1064960 to 1671168 tid 1 2024-02-05T04:53:32.215 INFO:tasks.rados.rados.0.smithi196.stdout:130: writing smithi19639362-12 from 1671168 to 2392064 tid 2 2024-02-05T04:53:32.216 INFO:tasks.rados.rados.0.smithi196.stdout:130: writing smithi19639362-12 from 2392064 to 2441216 tid 3 2024-02-05T04:53:32.216 INFO:tasks.rados.rados.0.smithi196.stdout:131: copy_from oid 1 from oid 20 current snap is 4 2024-02-05T04:53:32.216 INFO:tasks.rados.rados.0.smithi196.stdout:125: finishing write tid 5 to smithi19639362-29 2024-02-05T04:53:32.217 INFO:tasks.rados.rados.0.smithi196.stdout:125: oid 29 updating version 0 to 22 2024-02-05T04:53:32.217 INFO:tasks.rados.rados.0.smithi196.stdout:125: oid 29 updating version 22 to 25 2024-02-05T04:53:32.217 INFO:tasks.rados.rados.0.smithi196.stdout:125: oid 29 version 25 is already newer than 24 2024-02-05T04:53:32.217 INFO:tasks.rados.rados.0.smithi196.stdout:125: oid 29 version 25 is already newer than 23 2024-02-05T04:53:32.217 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 29 v 25 (ObjNum 78 snap 4 seq_num 78) dirty exists 2024-02-05T04:53:32.217 INFO:tasks.rados.rados.0.smithi196.stdout:125: left oid 29 (ObjNum 78 snap 4 seq_num 78) 2024-02-05T04:53:32.217 INFO:tasks.rados.rados.0.smithi196.stdout:126: finishing write tid 1 to smithi19639362-39 2024-02-05T04:53:32.217 INFO:tasks.rados.rados.0.smithi196.stdout:126: finishing write tid 2 to smithi19639362-39 2024-02-05T04:53:32.217 INFO:tasks.rados.rados.0.smithi196.stdout:125: done (6 left) 2024-02-05T04:53:32.217 INFO:tasks.rados.rados.0.smithi196.stdout:132: snap_create 2024-02-05T04:53:32.217 INFO:tasks.rados.rados.0.smithi196.stdout:127: finishing rollback tid 0 to smithi19639362-21 2024-02-05T04:53:32.217 INFO:tasks.rados.rados.0.smithi196.stdout:126: finishing write tid 3 to smithi19639362-39 2024-02-05T04:53:32.218 INFO:tasks.rados.rados.0.smithi196.stdout:128: expect (ObjNum 46 snap 0 seq_num 46) 2024-02-05T04:53:32.237 INFO:tasks.rados.rados.0.smithi196.stdout:126: finishing write tid 4 to smithi19639362-39 2024-02-05T04:53:32.237 INFO:tasks.rados.rados.0.smithi196.stdout:127: finishing rollback tid 1 to smithi19639362-21 2024-02-05T04:53:32.237 INFO:tasks.rados.rados.0.smithi196.stdout:127: finishing rollback tid 2 to smithi19639362-21 2024-02-05T04:53:32.237 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 21 v 25 (ObjNum 50 snap 0 seq_num 50) dirty exists 2024-02-05T04:53:32.237 INFO:tasks.rados.rados.0.smithi196.stdout:126: finishing write tid 5 to smithi19639362-39 2024-02-05T04:53:32.237 INFO:tasks.rados.rados.0.smithi196.stdout:126: oid 39 updating version 0 to 23 2024-02-05T04:53:32.237 INFO:tasks.rados.rados.0.smithi196.stdout:126: oid 39 updating version 23 to 25 2024-02-05T04:53:32.237 INFO:tasks.rados.rados.0.smithi196.stdout:126: oid 39 version 25 is already newer than 22 2024-02-05T04:53:32.237 INFO:tasks.rados.rados.0.smithi196.stdout:126: oid 39 version 25 is already newer than 24 2024-02-05T04:53:32.237 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 39 v 25 (ObjNum 79 snap 4 seq_num 79) dirty exists 2024-02-05T04:53:32.237 INFO:tasks.rados.rados.0.smithi196.stdout:126: left oid 39 (ObjNum 79 snap 4 seq_num 79) 2024-02-05T04:53:32.238 INFO:tasks.rados.rados.0.smithi196.stdout:129: finishing write tid 1 to smithi19639362-30 2024-02-05T04:53:32.238 INFO:tasks.rados.rados.0.smithi196.stdout:129: finishing write tid 2 to smithi19639362-30 2024-02-05T04:53:32.238 INFO:tasks.rados.rados.0.smithi196.stdout:129: finishing write tid 3 to smithi19639362-30 2024-02-05T04:53:32.238 INFO:tasks.rados.rados.0.smithi196.stdout:129: finishing write tid 4 to smithi19639362-30 2024-02-05T04:53:32.238 INFO:tasks.rados.rados.0.smithi196.stdout:129: finishing write tid 5 to smithi19639362-30 2024-02-05T04:53:32.238 INFO:tasks.rados.rados.0.smithi196.stdout:129: oid 30 updating version 0 to 12 2024-02-05T04:53:32.238 INFO:tasks.rados.rados.0.smithi196.stdout:129: oid 30 updating version 12 to 13 2024-02-05T04:53:32.238 INFO:tasks.rados.rados.0.smithi196.stdout:129: oid 30 updating version 13 to 15 2024-02-05T04:53:32.238 INFO:tasks.rados.rados.0.smithi196.stdout:129: oid 30 version 15 is already newer than 14 2024-02-05T04:53:32.238 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 30 v 15 (ObjNum 80 snap 4 seq_num 80) dirty exists 2024-02-05T04:53:32.238 INFO:tasks.rados.rados.0.smithi196.stdout:129: left oid 30 (ObjNum 80 snap 4 seq_num 80) 2024-02-05T04:53:32.238 INFO:tasks.rados.rados.0.smithi196.stdout:130: finishing write tid 1 to smithi19639362-12 2024-02-05T04:53:32.238 INFO:tasks.rados.rados.0.smithi196.stdout:131: finishing copy_from to smithi19639362-1 2024-02-05T04:53:32.238 INFO:tasks.rados.rados.0.smithi196.stdout:131: got expected ENOENT (src dne) 2024-02-05T04:53:32.238 INFO:tasks.rados.rados.0.smithi196.stdout:131: finishing copy_from racing read to smithi19639362-1 2024-02-05T04:53:32.239 INFO:tasks.rados.rados.0.smithi196.stdout:130: finishing write tid 2 to smithi19639362-12 2024-02-05T04:53:32.239 INFO:tasks.rados.rados.0.smithi196.stdout:130: finishing write tid 3 to smithi19639362-12 2024-02-05T04:53:32.239 INFO:tasks.rados.rados.0.smithi196.stdout:130: finishing write tid 4 to smithi19639362-12 2024-02-05T04:53:32.239 INFO:tasks.rados.rados.0.smithi196.stdout:130: finishing write tid 5 to smithi19639362-12 2024-02-05T04:53:32.239 INFO:tasks.rados.rados.0.smithi196.stdout:130: oid 12 updating version 0 to 22 2024-02-05T04:53:32.239 INFO:tasks.rados.rados.0.smithi196.stdout:130: oid 12 updating version 22 to 24 2024-02-05T04:53:32.239 INFO:tasks.rados.rados.0.smithi196.stdout:130: oid 12 version 24 is already newer than 23 2024-02-05T04:53:32.239 INFO:tasks.rados.rados.0.smithi196.stdout:130: oid 12 version 24 is already newer than 21 2024-02-05T04:53:32.239 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 12 v 24 (ObjNum 81 snap 4 seq_num 81) dirty exists 2024-02-05T04:53:32.239 INFO:tasks.rados.rados.0.smithi196.stdout:130: left oid 12 (ObjNum 81 snap 4 seq_num 81) 2024-02-05T04:53:32.240 INFO:tasks.ceph.osd.4.smithi062.stderr:2024-02-05T04:53:32.237+0000 7f708c308640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34689) UID: 0 2024-02-05T04:53:32.272 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:32.272 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:32.281 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-02-05T04:53:32.327 INFO:tasks.ceph.osd.0.smithi062.stderr:2024-02-05T04:53:32.327+0000 7fece8e03640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34688) UID: 0 2024-02-05T04:53:32.427 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:32.427 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:32.428 INFO:tasks.ceph.osd.3.smithi196.stderr:2024-02-05T04:53:32.428+0000 7fc8817c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34605) UID: 0 2024-02-05T04:53:32.436 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-02-05T04:53:32.529 INFO:tasks.ceph.osd.13.smithi091.stderr:2024-02-05T04:53:32.528+0000 7f526fd4d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 34281) UID: 0 2024-02-05T04:53:32.582 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:32.582 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:32.590 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-02-05T04:53:32.634 INFO:tasks.ceph.osd.7.smithi196.stderr:2024-02-05T04:53:32.633+0000 7fb281419640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34606) UID: 0 2024-02-05T04:53:32.707 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:32.707 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:32.708 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-02-05T04:53:32.749 INFO:tasks.ceph.osd.15.smithi196.stderr:2024-02-05T04:53:32.749+0000 7f1facc45640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 34611) UID: 0 2024-02-05T04:53:32.832 INFO:tasks.ceph.osd.1.smithi091.stderr:2024-02-05T04:53:32.830+0000 7ff9dc283640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34275) UID: 0 2024-02-05T04:53:32.855 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:32.855 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:32.864 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-02-05T04:53:32.932 INFO:tasks.ceph.osd.9.smithi091.stderr:2024-02-05T04:53:32.931+0000 7efc18827640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 34279) UID: 0 2024-02-05T04:53:32.970 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:32.970 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:32.979 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-02-05T04:53:33.033 INFO:tasks.ceph.osd.15.smithi196.stderr:2024-02-05T04:53:33.032+0000 7f1facc45640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 34611) UID: 0 2024-02-05T04:53:33.037 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T04:53:33.084 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:33.084 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:33.093 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-02-05T04:53:33.137 INFO:tasks.ceph.osd.8.smithi062.stderr:2024-02-05T04:53:33.136+0000 7f9385bf2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 34693) UID: 0 2024-02-05T04:53:33.146 INFO:teuthology.orchestra.run.smithi062.stderr:noscrub is set 2024-02-05T04:53:33.146 INFO:teuthology.orchestra.run.smithi062.stderr:reweighted osd.12 to 1 (10000) 2024-02-05T04:53:33.151 INFO:tasks.rados.rados.0.smithi196.stdout:126: done (6 left) 2024-02-05T04:53:33.151 INFO:tasks.rados.rados.0.smithi196.stdout:127: done (5 left) 2024-02-05T04:53:33.151 INFO:tasks.rados.rados.0.smithi196.stdout:128: done (4 left) 2024-02-05T04:53:33.151 INFO:tasks.rados.rados.0.smithi196.stdout:129: done (3 left) 2024-02-05T04:53:33.151 INFO:tasks.rados.rados.0.smithi196.stdout:130: done (2 left) 2024-02-05T04:53:33.151 INFO:tasks.rados.rados.0.smithi196.stdout:131: done (1 left) 2024-02-05T04:53:33.151 INFO:tasks.rados.rados.0.smithi196.stdout:132: done (0 left) 2024-02-05T04:53:33.152 INFO:tasks.rados.rados.0.smithi196.stdout:133: delete oid 12 current snap is 5 2024-02-05T04:53:33.158 INFO:tasks.rados.rados.0.smithi196.stdout:133: done (0 left) 2024-02-05T04:53:33.158 INFO:tasks.rados.rados.0.smithi196.stdout:134: read oid 10 snap 3 2024-02-05T04:53:33.158 INFO:tasks.rados.rados.0.smithi196.stdout:134: expect (ObjNum 53 snap 0 seq_num 53) 2024-02-05T04:53:33.158 INFO:tasks.rados.rados.0.smithi196.stdout:135: snap_create 2024-02-05T04:53:33.167 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 13 1 2024-02-05T04:53:33.169 INFO:tasks.rados.rados.0.smithi196.stdout:134: expect (ObjNum 53 snap 0 seq_num 53) 2024-02-05T04:53:33.174 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0: {} 2024-02-05T04:53:33.175 INFO:teuthology.orchestra.run.smithi062.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T04:53:33.195 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1: {} 2024-02-05T04:53:33.195 INFO:teuthology.orchestra.run.smithi062.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T04:53:33.210 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-02-05T04:53:33.210 INFO:teuthology.orchestra.run.smithi062.stdout: "ops": [], 2024-02-05T04:53:33.210 INFO:teuthology.orchestra.run.smithi062.stdout: "num_ops": 0 2024-02-05T04:53:33.210 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-02-05T04:53:33.213 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2: {} 2024-02-05T04:53:33.214 INFO:teuthology.orchestra.run.smithi062.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T04:53:33.220 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-02-05T04:53:33.232 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3: {} 2024-02-05T04:53:33.232 INFO:teuthology.orchestra.run.smithi062.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T04:53:33.249 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4: {} 2024-02-05T04:53:33.249 INFO:teuthology.orchestra.run.smithi062.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T04:53:33.249 INFO:tasks.ceph.osd.8.smithi062.stderr:2024-02-05T04:53:33.248+0000 7f9385bf2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 34693) UID: 0 2024-02-05T04:53:33.267 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5: {} 2024-02-05T04:53:33.268 INFO:teuthology.orchestra.run.smithi062.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T04:53:33.286 INFO:teuthology.orchestra.run.smithi062.stdout:osd.6: {} 2024-02-05T04:53:33.286 INFO:teuthology.orchestra.run.smithi062.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T04:53:33.304 INFO:teuthology.orchestra.run.smithi062.stdout:osd.7: {} 2024-02-05T04:53:33.304 INFO:teuthology.orchestra.run.smithi062.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T04:53:33.321 INFO:teuthology.orchestra.run.smithi062.stdout:osd.8: {} 2024-02-05T04:53:33.321 INFO:teuthology.orchestra.run.smithi062.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-05T04:53:33.337 INFO:tasks.ceph.osd.14.smithi150.stderr:2024-02-05T04:53:33.336+0000 7f5f81083640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 34020) UID: 0 2024-02-05T04:53:33.339 INFO:teuthology.orchestra.run.smithi062.stdout:osd.9: {} 2024-02-05T04:53:33.339 INFO:teuthology.orchestra.run.smithi062.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-05T04:53:33.341 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-02-05T04:53:33.341 INFO:teuthology.orchestra.run.smithi062.stdout: "ops": [], 2024-02-05T04:53:33.341 INFO:teuthology.orchestra.run.smithi062.stdout: "complaint_time": 30, 2024-02-05T04:53:33.341 INFO:teuthology.orchestra.run.smithi062.stdout: "num_blocked_ops": 0 2024-02-05T04:53:33.341 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-02-05T04:53:33.350 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-02-05T04:53:33.357 INFO:teuthology.orchestra.run.smithi062.stdout:osd.10: {} 2024-02-05T04:53:33.358 INFO:teuthology.orchestra.run.smithi062.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-05T04:53:33.375 INFO:teuthology.orchestra.run.smithi062.stdout:osd.11: {} 2024-02-05T04:53:33.375 INFO:teuthology.orchestra.run.smithi062.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-05T04:53:33.395 INFO:teuthology.orchestra.run.smithi062.stdout:osd.12: {} 2024-02-05T04:53:33.396 INFO:teuthology.orchestra.run.smithi062.stderr:osd.12: osd_enable_op_tracker = '' 2024-02-05T04:53:33.411 INFO:teuthology.orchestra.run.smithi062.stdout:osd.13: {} 2024-02-05T04:53:33.411 INFO:teuthology.orchestra.run.smithi062.stderr:osd.13: osd_enable_op_tracker = '' 2024-02-05T04:53:33.429 INFO:teuthology.orchestra.run.smithi062.stdout:osd.14: {} 2024-02-05T04:53:33.429 INFO:teuthology.orchestra.run.smithi062.stderr:osd.14: osd_enable_op_tracker = '' 2024-02-05T04:53:33.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-02-05T04:53:33.429+0000 7f7760c89640 0 -- 172.21.15.62:0/2724613613 >> [v2:172.21.15.196:6826/3187639683,v1:172.21.15.196:6827/3187639683] conn(0x7f77340339c0 msgr2=0x7f7734035db0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-02-05T04:53:33.438 INFO:tasks.ceph.osd.2.smithi150.stderr:2024-02-05T04:53:33.437+0000 7f7337bf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 34014) UID: 0 2024-02-05T04:53:33.467 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-02-05T04:53:33.467 INFO:teuthology.orchestra.run.smithi062.stdout: "size": 20, 2024-02-05T04:53:33.467 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 600, 2024-02-05T04:53:33.467 INFO:teuthology.orchestra.run.smithi062.stdout: "ops": [ 2024-02-05T04:53:33.467 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.467 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.8s3 37/31 ECSubWrite(tid=25, reqid=osd.4.0:25, at_version=37'10, trim_to=0'0, roll_forward_to=35'8))", 2024-02-05T04:53:33.467 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:29.129097+0000", 2024-02-05T04:53:33.467 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 4.3369072129999999, 2024-02-05T04:53:33.467 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0016204780000000001, 2024-02-05T04:53:33.468 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:33.468 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:33.468 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:33.468 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.468 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:33.468 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.129097+0000", 2024-02-05T04:53:33.468 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:33.468 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.468 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.468 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:33.468 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.129097+0000", 2024-02-05T04:53:33.468 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:33.468 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.468 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.468 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:33.469 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.129097+0000", 2024-02-05T04:53:33.469 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.0100000000000005e-07 2024-02-05T04:53:33.469 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.469 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.469 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:33.469 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.129147+0000", 2024-02-05T04:53:33.469 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.9901000000000002e-05 2024-02-05T04:53:33.469 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.469 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.469 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:33.469 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.129178+0000", 2024-02-05T04:53:33.469 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.0639000000000003e-05 2024-02-05T04:53:33.469 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.469 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.469 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:33.469 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.129193+0000", 2024-02-05T04:53:33.470 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.5194999999999999e-05 2024-02-05T04:53:33.470 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.470 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.470 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:33.470 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.129285+0000", 2024-02-05T04:53:33.470 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.1779000000000003e-05 2024-02-05T04:53:33.470 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.470 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.470 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:33.470 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.129299+0000", 2024-02-05T04:53:33.470 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.4622e-05 2024-02-05T04:53:33.470 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.470 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.470 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:33.470 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.130678+0000", 2024-02-05T04:53:33.471 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0013788310000000001 2024-02-05T04:53:33.471 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.471 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.471 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:33.471 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.130717+0000", 2024-02-05T04:53:33.471 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.8909999999999998e-05 2024-02-05T04:53:33.471 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:33.471 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:33.471 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:33.471 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.471 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.471 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.8s3 37/31 ECSubWrite(tid=28, reqid=osd.4.0:25, at_version=0'0, trim_to=0'0, roll_forward_to=37'10))", 2024-02-05T04:53:33.471 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:29.131644+0000", 2024-02-05T04:53:33.471 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 4.3343594210000003, 2024-02-05T04:53:33.471 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00092228799999999995, 2024-02-05T04:53:33.471 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:33.472 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:33.472 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:33.472 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.472 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:33.472 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.131644+0000", 2024-02-05T04:53:33.472 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:33.472 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.472 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.472 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:33.472 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.131644+0000", 2024-02-05T04:53:33.472 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:33.472 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.473 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.473 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:33.473 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.131645+0000", 2024-02-05T04:53:33.473 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.029e-06 2024-02-05T04:53:33.473 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.473 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.473 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:33.473 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.131670+0000", 2024-02-05T04:53:33.473 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.4794e-05 2024-02-05T04:53:33.473 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.473 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.473 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:33.473 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.131686+0000", 2024-02-05T04:53:33.473 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.5472999999999999e-05 2024-02-05T04:53:33.474 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.474 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.474 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:33.474 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.131696+0000", 2024-02-05T04:53:33.474 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.9119999999999993e-06 2024-02-05T04:53:33.474 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.474 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.474 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:33.474 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.131758+0000", 2024-02-05T04:53:33.474 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.2593000000000002e-05 2024-02-05T04:53:33.474 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.474 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.474 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:33.474 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.131768+0000", 2024-02-05T04:53:33.474 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.5959999999999993e-06 2024-02-05T04:53:33.474 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.475 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.475 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:33.475 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.132531+0000", 2024-02-05T04:53:33.475 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00076315200000000001 2024-02-05T04:53:33.475 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.475 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.475 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:33.475 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.132567+0000", 2024-02-05T04:53:33.475 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.5738999999999998e-05 2024-02-05T04:53:33.475 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:33.475 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:33.475 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:33.475 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.475 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.475 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpRead(3.bs3 37/31 ECSubRead(tid=76, to_read={3:db93bf34:::smithi19639362-24:head=0,98304,0}, subchunks={3:db93bf34:::smithi19639362-24:head=[0,1]}, attrs_to_read=))", 2024-02-05T04:53:33.475 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:29.150071+0000", 2024-02-05T04:53:33.476 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 4.3159329959999999, 2024-02-05T04:53:33.476 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00064157799999999996, 2024-02-05T04:53:33.476 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:33.476 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:33.476 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:33.476 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.476 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:33.476 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.150071+0000", 2024-02-05T04:53:33.476 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:33.476 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.476 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.476 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:33.476 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.150071+0000", 2024-02-05T04:53:33.476 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:33.476 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.477 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.477 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:33.477 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.150071+0000", 2024-02-05T04:53:33.477 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.3599999999999999e-07 2024-02-05T04:53:33.477 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.477 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.477 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:33.477 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.150081+0000", 2024-02-05T04:53:33.477 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0317e-05 2024-02-05T04:53:33.477 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.477 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.477 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:33.477 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.150110+0000", 2024-02-05T04:53:33.477 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.8129000000000001e-05 2024-02-05T04:53:33.477 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.477 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.478 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:33.478 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.150122+0000", 2024-02-05T04:53:33.478 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.2308e-05 2024-02-05T04:53:33.478 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.478 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.478 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:33.478 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.150213+0000", 2024-02-05T04:53:33.478 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.0698999999999993e-05 2024-02-05T04:53:33.478 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.478 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.478 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:33.478 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.150712+0000", 2024-02-05T04:53:33.478 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00049978900000000005 2024-02-05T04:53:33.478 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:33.478 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:33.478 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:33.479 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.479 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.479 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpRead(3.bs3 37/31 ECSubRead(tid=77, to_read={3:db93bf34:::smithi19639362-24:head=0,98304,0}, subchunks={3:db93bf34:::smithi19639362-24:head=[0,1]}, attrs_to_read=))", 2024-02-05T04:53:33.479 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:29.150463+0000", 2024-02-05T04:53:33.479 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 4.3155403740000002, 2024-02-05T04:53:33.479 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000453003, 2024-02-05T04:53:33.479 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:33.479 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:33.479 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:33.479 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.479 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:33.479 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.150463+0000", 2024-02-05T04:53:33.479 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:33.479 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.479 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.479 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:33.480 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.150463+0000", 2024-02-05T04:53:33.480 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:33.480 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.480 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.480 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:33.480 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.150464+0000", 2024-02-05T04:53:33.480 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.7300000000000004e-07 2024-02-05T04:53:33.480 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.480 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.480 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:33.480 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.150474+0000", 2024-02-05T04:53:33.480 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.6679999999999996e-06 2024-02-05T04:53:33.480 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.480 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.481 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:33.481 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.150491+0000", 2024-02-05T04:53:33.481 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.7288000000000001e-05 2024-02-05T04:53:33.481 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.481 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.481 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:33.481 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.150505+0000", 2024-02-05T04:53:33.481 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.3531e-05 2024-02-05T04:53:33.481 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.481 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.481 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:33.481 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.150773+0000", 2024-02-05T04:53:33.481 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000267796 2024-02-05T04:53:33.481 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.481 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.481 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:33.482 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.150916+0000", 2024-02-05T04:53:33.482 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000143747 2024-02-05T04:53:33.482 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:33.482 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:33.482 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:33.482 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.482 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.482 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpRead(3.bs3 37/31 ECSubRead(tid=78, to_read={3:db93bf34:::smithi19639362-24:head=0,98304,0}, subchunks={3:db93bf34:::smithi19639362-24:head=[0,1]}, attrs_to_read=))", 2024-02-05T04:53:33.482 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:29.150556+0000", 2024-02-05T04:53:33.482 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 4.3154478159999998, 2024-02-05T04:53:33.482 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00053283099999999999, 2024-02-05T04:53:33.482 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:33.482 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:33.482 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:33.482 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.483 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:33.483 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.150556+0000", 2024-02-05T04:53:33.483 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:33.483 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.483 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.483 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:33.483 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.150556+0000", 2024-02-05T04:53:33.483 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:33.484 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.484 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.484 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:33.484 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.150557+0000", 2024-02-05T04:53:33.484 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.7999999999999995e-07 2024-02-05T04:53:33.484 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.484 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.484 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:33.484 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.150563+0000", 2024-02-05T04:53:33.484 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.5590000000000002e-06 2024-02-05T04:53:33.484 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.484 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.484 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:33.485 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.150588+0000", 2024-02-05T04:53:33.485 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.4513000000000001e-05 2024-02-05T04:53:33.485 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.485 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.485 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:33.485 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.150598+0000", 2024-02-05T04:53:33.485 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0587e-05 2024-02-05T04:53:33.485 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.485 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.485 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:33.485 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.150956+0000", 2024-02-05T04:53:33.485 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00035813799999999998 2024-02-05T04:53:33.485 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.485 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.485 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:33.486 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.151089+0000", 2024-02-05T04:53:33.486 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000132454 2024-02-05T04:53:33.486 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:33.486 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:33.486 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:33.486 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.486 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.486 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpRead(3.bs3 37/31 ECSubRead(tid=80, to_read={3:de65491d:::smithi19639362-2:head=0,8192,0}, subchunks={3:de65491d:::smithi19639362-2:head=[0,1]}, attrs_to_read=))", 2024-02-05T04:53:33.486 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:29.152960+0000", 2024-02-05T04:53:33.486 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 4.3130434539999998, 2024-02-05T04:53:33.486 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00041001500000000001, 2024-02-05T04:53:33.486 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:33.486 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:33.486 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:33.486 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.486 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:33.487 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.152960+0000", 2024-02-05T04:53:33.487 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:33.487 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.487 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.487 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:33.487 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.152960+0000", 2024-02-05T04:53:33.487 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:33.487 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.487 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.487 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:33.487 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.152961+0000", 2024-02-05T04:53:33.487 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.9899999999999999e-07 2024-02-05T04:53:33.487 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.487 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.487 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:33.488 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.152974+0000", 2024-02-05T04:53:33.488 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.2663999999999999e-05 2024-02-05T04:53:33.488 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.488 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.488 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:33.488 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.152990+0000", 2024-02-05T04:53:33.488 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.5631e-05 2024-02-05T04:53:33.488 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.488 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.488 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:33.488 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.153001+0000", 2024-02-05T04:53:33.488 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1001e-05 2024-02-05T04:53:33.488 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.488 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.488 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:33.489 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.153076+0000", 2024-02-05T04:53:33.489 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.5737999999999999e-05 2024-02-05T04:53:33.489 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.489 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.489 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:33.489 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.153370+0000", 2024-02-05T04:53:33.489 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00029408199999999998 2024-02-05T04:53:33.489 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:33.489 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:33.489 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:33.489 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.489 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.489 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpRead(3.bs3 37/31 ECSubRead(tid=81, to_read={3:de65491d:::smithi19639362-2:head=0,8192,0}, subchunks={3:de65491d:::smithi19639362-2:head=[0,1]}, attrs_to_read=))", 2024-02-05T04:53:33.489 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:29.153414+0000", 2024-02-05T04:53:33.489 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 4.3125897069999999, 2024-02-05T04:53:33.489 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00024249699999999999, 2024-02-05T04:53:33.490 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:33.490 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:33.490 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:33.490 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.490 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:33.490 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.153414+0000", 2024-02-05T04:53:33.490 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:33.490 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.490 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.490 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:33.490 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.153414+0000", 2024-02-05T04:53:33.490 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:33.490 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.490 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.490 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:33.491 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.153415+0000", 2024-02-05T04:53:33.491 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.7000000000000003e-07 2024-02-05T04:53:33.491 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.491 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.491 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:33.491 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.153423+0000", 2024-02-05T04:53:33.491 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.208e-06 2024-02-05T04:53:33.491 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.491 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.491 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:33.491 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.153440+0000", 2024-02-05T04:53:33.491 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.7079e-05 2024-02-05T04:53:33.491 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.491 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.491 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:33.491 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.153451+0000", 2024-02-05T04:53:33.492 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.08e-05 2024-02-05T04:53:33.492 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.492 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.492 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:33.492 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.153547+0000", 2024-02-05T04:53:33.492 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.5748999999999994e-05 2024-02-05T04:53:33.492 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.492 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.492 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:33.492 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.153657+0000", 2024-02-05T04:53:33.492 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00010979100000000001 2024-02-05T04:53:33.492 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:33.492 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:33.492 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:33.493 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.493 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.493 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpRead(3.bs3 37/31 ECSubRead(tid=82, to_read={3:de65491d:::smithi19639362-2:head=0,8192,0}, subchunks={3:de65491d:::smithi19639362-2:head=[0,1]}, attrs_to_read=))", 2024-02-05T04:53:33.493 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:29.155779+0000", 2024-02-05T04:53:33.493 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 4.3102250040000003, 2024-02-05T04:53:33.493 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000169545, 2024-02-05T04:53:33.493 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:33.493 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:33.493 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:33.493 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.493 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:33.493 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.155779+0000", 2024-02-05T04:53:33.493 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:33.493 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.493 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.494 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:33.494 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.155779+0000", 2024-02-05T04:53:33.494 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:33.494 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.494 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.494 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:33.494 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.155779+0000", 2024-02-05T04:53:33.494 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.8100000000000002e-07 2024-02-05T04:53:33.494 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.494 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.494 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:33.494 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.155785+0000", 2024-02-05T04:53:33.494 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.9889999999999997e-06 2024-02-05T04:53:33.494 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.494 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.495 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:33.495 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.155799+0000", 2024-02-05T04:53:33.495 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.3050999999999999e-05 2024-02-05T04:53:33.495 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.495 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.495 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:33.495 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.155806+0000", 2024-02-05T04:53:33.495 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.9410000000000007e-06 2024-02-05T04:53:33.495 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.495 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.495 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:33.495 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.155888+0000", 2024-02-05T04:53:33.495 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.1997999999999997e-05 2024-02-05T04:53:33.495 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.495 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.495 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:33.496 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.155948+0000", 2024-02-05T04:53:33.496 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.9885e-05 2024-02-05T04:53:33.496 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:33.496 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:33.496 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:33.496 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.496 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.496 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.fs4 38/31 ECSubWrite(tid=51, reqid=client.4634.0:290, at_version=38'17, trim_to=0'0, roll_forward_to=34'16))", 2024-02-05T04:53:33.496 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:30.387948+0000", 2024-02-05T04:53:33.496 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 3.0780561469999999, 2024-02-05T04:53:33.496 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0017840239999999999, 2024-02-05T04:53:33.496 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:33.496 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:33.496 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:33.496 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.497 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:33.497 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.387948+0000", 2024-02-05T04:53:33.497 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:33.497 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.497 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.497 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:33.497 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.387948+0000", 2024-02-05T04:53:33.497 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:33.497 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.497 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.497 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:33.497 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.387949+0000", 2024-02-05T04:53:33.497 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.159e-06 2024-02-05T04:53:33.497 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.497 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.498 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:33.498 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.388411+0000", 2024-02-05T04:53:33.498 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00046226999999999998 2024-02-05T04:53:33.498 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.498 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.498 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:33.498 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.388431+0000", 2024-02-05T04:53:33.498 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.0390999999999998e-05 2024-02-05T04:53:33.498 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.498 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.498 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:33.498 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.388442+0000", 2024-02-05T04:53:33.498 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0902e-05 2024-02-05T04:53:33.498 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.498 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.499 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:33.499 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.388517+0000", 2024-02-05T04:53:33.499 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.4648999999999996e-05 2024-02-05T04:53:33.499 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.499 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.499 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:33.499 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.388528+0000", 2024-02-05T04:53:33.499 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.131e-05 2024-02-05T04:53:33.499 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.499 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.499 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:33.499 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.389707+0000", 2024-02-05T04:53:33.499 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0011782629999999999 2024-02-05T04:53:33.499 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.499 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.499 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:33.500 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.389732+0000", 2024-02-05T04:53:33.500 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.508e-05 2024-02-05T04:53:33.500 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:33.500 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:33.500 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:33.500 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.500 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.500 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.fs4 38/31 ECSubWrite(tid=52, reqid=client.4634.0:291, at_version=38'18, trim_to=0'0, roll_forward_to=34'16))", 2024-02-05T04:53:33.500 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:30.389019+0000", 2024-02-05T04:53:33.500 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 3.0769851080000001, 2024-02-05T04:53:33.500 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.001372809, 2024-02-05T04:53:33.500 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:33.500 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:33.500 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:33.500 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.501 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:33.501 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.389019+0000", 2024-02-05T04:53:33.501 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:33.501 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.501 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.501 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:33.501 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.389019+0000", 2024-02-05T04:53:33.501 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:33.501 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.501 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.501 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:33.501 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.389019+0000", 2024-02-05T04:53:33.501 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.2300000000000001e-07 2024-02-05T04:53:33.501 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.501 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.501 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:33.502 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.389319+0000", 2024-02-05T04:53:33.502 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00029977500000000001 2024-02-05T04:53:33.502 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.502 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.502 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:33.502 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.389331+0000", 2024-02-05T04:53:33.502 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.2146e-05 2024-02-05T04:53:33.502 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.502 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.502 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:33.502 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.389339+0000", 2024-02-05T04:53:33.502 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.5129999999999999e-06 2024-02-05T04:53:33.502 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.502 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.502 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:33.503 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.389391+0000", 2024-02-05T04:53:33.503 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.2703e-05 2024-02-05T04:53:33.503 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.503 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.503 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:33.503 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.389401+0000", 2024-02-05T04:53:33.503 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.4050000000000006e-06 2024-02-05T04:53:33.503 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.503 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.503 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:33.503 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.390362+0000", 2024-02-05T04:53:33.503 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00096129699999999995 2024-02-05T04:53:33.503 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.503 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.503 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:33.504 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.390391+0000", 2024-02-05T04:53:33.504 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.9346999999999999e-05 2024-02-05T04:53:33.504 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:33.504 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:33.504 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:33.504 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.504 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.504 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.fs4 38/31 ECSubWrite(tid=53, reqid=client.4634.0:292, at_version=38'19, trim_to=0'0, roll_forward_to=34'16))", 2024-02-05T04:53:33.504 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:30.389890+0000", 2024-02-05T04:53:33.504 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 3.0761141310000002, 2024-02-05T04:53:33.504 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.001022018, 2024-02-05T04:53:33.504 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:33.504 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:33.504 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:33.504 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.505 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:33.505 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.389890+0000", 2024-02-05T04:53:33.505 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:33.505 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.505 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.505 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:33.505 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.389890+0000", 2024-02-05T04:53:33.505 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:33.505 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.505 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.505 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:33.505 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.389890+0000", 2024-02-05T04:53:33.505 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.7100000000000002e-07 2024-02-05T04:53:33.505 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.505 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.505 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:33.506 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.389997+0000", 2024-02-05T04:53:33.506 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000106426 2024-02-05T04:53:33.506 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.506 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.506 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:33.506 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.390011+0000", 2024-02-05T04:53:33.506 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.4124000000000001e-05 2024-02-05T04:53:33.506 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.506 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.506 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:33.506 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.390021+0000", 2024-02-05T04:53:33.506 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0149999999999999e-05 2024-02-05T04:53:33.506 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.506 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.506 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:33.507 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.390069+0000", 2024-02-05T04:53:33.507 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.7901000000000001e-05 2024-02-05T04:53:33.507 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.507 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.507 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:33.507 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.390074+0000", 2024-02-05T04:53:33.507 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.186e-06 2024-02-05T04:53:33.507 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.507 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.507 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:33.507 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.390883+0000", 2024-02-05T04:53:33.507 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00080893899999999995 2024-02-05T04:53:33.507 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.507 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.507 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:33.508 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.390912+0000", 2024-02-05T04:53:33.508 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.8721000000000002e-05 2024-02-05T04:53:33.508 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:33.508 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:33.508 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:33.508 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.508 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.508 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.fs4 38/31 ECSubWrite(tid=54, reqid=client.4634.0:293, at_version=38'20, trim_to=0'0, roll_forward_to=34'16))", 2024-02-05T04:53:33.508 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:30.390622+0000", 2024-02-05T04:53:33.508 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 3.0753816540000001, 2024-02-05T04:53:33.508 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0011679749999999999, 2024-02-05T04:53:33.508 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:33.508 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:33.508 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:33.509 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.509 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:33.509 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.390622+0000", 2024-02-05T04:53:33.509 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:33.509 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.509 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.509 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:33.509 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.390622+0000", 2024-02-05T04:53:33.509 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:33.509 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.509 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.509 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:33.509 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.390623+0000", 2024-02-05T04:53:33.509 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.1600000000000001e-07 2024-02-05T04:53:33.509 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.509 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.510 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:33.510 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.390634+0000", 2024-02-05T04:53:33.510 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1669e-05 2024-02-05T04:53:33.510 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.510 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.510 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:33.510 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.390644+0000", 2024-02-05T04:53:33.510 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.5480000000000007e-06 2024-02-05T04:53:33.510 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.510 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.510 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:33.510 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.390649+0000", 2024-02-05T04:53:33.510 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.5929999999999996e-06 2024-02-05T04:53:33.510 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.510 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.510 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:33.511 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.390694+0000", 2024-02-05T04:53:33.511 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.4348999999999998e-05 2024-02-05T04:53:33.511 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.511 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.511 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:33.511 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.390704+0000", 2024-02-05T04:53:33.511 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0111e-05 2024-02-05T04:53:33.511 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.511 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.511 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:33.511 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.391761+0000", 2024-02-05T04:53:33.511 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0010575409999999999 2024-02-05T04:53:33.511 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.511 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.511 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:33.511 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.391790+0000", 2024-02-05T04:53:33.512 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.8648000000000001e-05 2024-02-05T04:53:33.512 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:33.512 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:33.512 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:33.512 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.512 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.512 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.fs4 38/31 ECSubWrite(tid=55, reqid=client.4634.0:293, at_version=0'0, trim_to=0'0, roll_forward_to=38'20))", 2024-02-05T04:53:33.512 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:30.395478+0000", 2024-02-05T04:53:33.512 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 3.0705258440000001, 2024-02-05T04:53:33.512 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00056105200000000004, 2024-02-05T04:53:33.512 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:33.512 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:33.512 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:33.512 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.512 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:33.513 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.395478+0000", 2024-02-05T04:53:33.513 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:33.513 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.513 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.513 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:33.513 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.395478+0000", 2024-02-05T04:53:33.513 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:33.513 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.513 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.513 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:33.513 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.395479+0000", 2024-02-05T04:53:33.513 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.7700000000000004e-07 2024-02-05T04:53:33.513 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.513 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.514 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:33.514 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.395487+0000", 2024-02-05T04:53:33.514 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.1240000000000005e-06 2024-02-05T04:53:33.514 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.514 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.514 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:33.514 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.395497+0000", 2024-02-05T04:53:33.514 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0736e-05 2024-02-05T04:53:33.514 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.514 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.514 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:33.514 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.395506+0000", 2024-02-05T04:53:33.514 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.3469999999999999e-06 2024-02-05T04:53:33.514 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.514 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.515 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:33.515 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.395548+0000", 2024-02-05T04:53:33.515 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.2030000000000002e-05 2024-02-05T04:53:33.515 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.515 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.515 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:33.515 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.395553+0000", 2024-02-05T04:53:33.515 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.2100000000000001e-06 2024-02-05T04:53:33.515 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.515 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.515 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:33.515 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.396015+0000", 2024-02-05T04:53:33.515 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00046247399999999999 2024-02-05T04:53:33.515 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:33.515 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:33.515 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:33.516 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.396039+0000", 2024-02-05T04:53:33.516 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.3553999999999998e-05 2024-02-05T04:53:33.516 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:33.516 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:33.516 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:33.516 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:33.516 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:33.516 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-02-05T04:53:33.517 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-02-05T04:53:33.559 INFO:tasks.ceph.osd.8.smithi062.stderr:2024-02-05T04:53:33.558+0000 7f9385bf2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 34693) UID: 0 2024-02-05T04:53:33.640 INFO:tasks.ceph.osd.1.smithi091.stderr:2024-02-05T04:53:33.638+0000 7ff9dc283640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34275) UID: 0 2024-02-05T04:53:33.665 INFO:teuthology.orchestra.run.smithi062.stdout:osd.15: {} 2024-02-05T04:53:33.665 INFO:teuthology.orchestra.run.smithi062.stderr:osd.15: osd_enable_op_tracker = '' 2024-02-05T04:53:33.684 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-02-05T04:53:33.684 INFO:teuthology.orchestra.run.smithi062.stdout: "ops": [], 2024-02-05T04:53:33.685 INFO:teuthology.orchestra.run.smithi062.stdout: "num_ops": 0 2024-02-05T04:53:33.685 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-02-05T04:53:33.694 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-02-05T04:53:33.740 INFO:tasks.ceph.osd.5.smithi091.stderr:2024-02-05T04:53:33.739+0000 7f63983b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34277) UID: 0 2024-02-05T04:53:33.842 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-02-05T04:53:33.842 INFO:teuthology.orchestra.run.smithi062.stdout: "ops": [], 2024-02-05T04:53:33.842 INFO:teuthology.orchestra.run.smithi062.stdout: "complaint_time": 30, 2024-02-05T04:53:33.842 INFO:teuthology.orchestra.run.smithi062.stdout: "num_blocked_ops": 0 2024-02-05T04:53:33.842 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-02-05T04:53:33.842 INFO:tasks.ceph.osd.0.smithi062.stderr:2024-02-05T04:53:33.841+0000 7fece8e03640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34688) UID: 0 2024-02-05T04:53:33.850 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-02-05T04:53:33.943 INFO:tasks.ceph.osd.7.smithi196.stderr:2024-02-05T04:53:33.943+0000 7fb281419640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34606) UID: 0 2024-02-05T04:53:33.999 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-02-05T04:53:33.999 INFO:teuthology.orchestra.run.smithi062.stdout: "size": 20, 2024-02-05T04:53:33.999 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 600, 2024-02-05T04:53:33.999 INFO:teuthology.orchestra.run.smithi062.stdout: "ops": [ 2024-02-05T04:53:34.000 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.000 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:218 3.3s0 3:ce48de55:::smithi19639362-16:head [append 0~638976] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:53:34.000 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:29.132294+0000", 2024-02-05T04:53:34.000 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 4.8651252410000003, 2024-02-05T04:53:34.000 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0081869909999999994, 2024-02-05T04:53:34.000 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:34.000 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:34.000 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:53:34.000 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:53:34.000 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:34.000 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 218 2024-02-05T04:53:34.000 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.000 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:34.001 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.001 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:34.001 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.132294+0000", 2024-02-05T04:53:34.001 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.001 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.001 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.001 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:34.001 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.132294+0000", 2024-02-05T04:53:34.001 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.001 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.001 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.001 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:34.001 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.132296+0000", 2024-02-05T04:53:34.001 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.886e-06 2024-02-05T04:53:34.002 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.002 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.002 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:34.002 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.133151+0000", 2024-02-05T04:53:34.002 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00085535699999999995 2024-02-05T04:53:34.002 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.002 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.002 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:34.002 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.133165+0000", 2024-02-05T04:53:34.002 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.396e-05 2024-02-05T04:53:34.002 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.002 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.002 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:34.002 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.133182+0000", 2024-02-05T04:53:34.002 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.6787e-05 2024-02-05T04:53:34.002 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.003 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.003 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:34.003 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.133229+0000", 2024-02-05T04:53:34.003 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.7509999999999997e-05 2024-02-05T04:53:34.003 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.003 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.003 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:53:34.003 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.133324+0000", 2024-02-05T04:53:34.003 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.4950000000000004e-05 2024-02-05T04:53:34.003 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.003 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.003 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:34.003 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.134392+0000", 2024-02-05T04:53:34.003 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0010674219999999999 2024-02-05T04:53:34.003 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.004 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.004 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:34.004 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.138240+0000", 2024-02-05T04:53:34.004 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0038487930000000001 2024-02-05T04:53:34.004 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.004 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.004 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:53:34.004 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.140400+0000", 2024-02-05T04:53:34.004 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0021592690000000001 2024-02-05T04:53:34.004 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.004 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.004 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:34.004 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.140481+0000", 2024-02-05T04:53:34.004 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.1057000000000007e-05 2024-02-05T04:53:34.004 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.005 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:34.005 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.005 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.005 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.005 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:219 3.3s0 3:ce48de55:::smithi19639362-16:head [append 0~376832] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:53:34.005 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:29.134822+0000", 2024-02-05T04:53:34.005 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 4.8625965650000005, 2024-02-05T04:53:34.005 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0088602019999999993, 2024-02-05T04:53:34.006 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:34.006 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:34.006 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:53:34.006 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:53:34.006 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:34.006 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 219 2024-02-05T04:53:34.006 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.006 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:34.006 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.006 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:34.006 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.134822+0000", 2024-02-05T04:53:34.006 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.007 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.007 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.007 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:34.007 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.134822+0000", 2024-02-05T04:53:34.007 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.007 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.007 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.007 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:34.007 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.134825+0000", 2024-02-05T04:53:34.007 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.1229999999999998e-06 2024-02-05T04:53:34.007 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.007 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.007 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:34.007 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.135417+0000", 2024-02-05T04:53:34.007 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00059254900000000005 2024-02-05T04:53:34.008 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.008 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.008 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:34.008 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.135430+0000", 2024-02-05T04:53:34.008 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.2663000000000001e-05 2024-02-05T04:53:34.008 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.008 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.008 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:34.008 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.135436+0000", 2024-02-05T04:53:34.008 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.6959999999999999e-06 2024-02-05T04:53:34.008 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.008 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.008 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:34.009 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.135834+0000", 2024-02-05T04:53:34.009 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000397382 2024-02-05T04:53:34.009 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.009 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.009 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:53:34.009 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.135967+0000", 2024-02-05T04:53:34.009 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00013333 2024-02-05T04:53:34.009 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.009 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.009 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:34.009 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.136753+0000", 2024-02-05T04:53:34.009 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00078585700000000005 2024-02-05T04:53:34.009 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.009 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.010 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:34.010 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.138286+0000", 2024-02-05T04:53:34.010 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0015333390000000001 2024-02-05T04:53:34.010 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.010 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.010 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:53:34.010 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.143611+0000", 2024-02-05T04:53:34.010 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0053249639999999997 2024-02-05T04:53:34.010 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.010 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.010 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:34.010 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.143683+0000", 2024-02-05T04:53:34.010 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.1298999999999996e-05 2024-02-05T04:53:34.011 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.011 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:34.011 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.011 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.011 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.011 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:220 3.3s0 3:ce48de55:::smithi19639362-16:head [setxattr _header (56) in=7b] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:53:34.011 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:29.135450+0000", 2024-02-05T04:53:34.011 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 4.861968643, 2024-02-05T04:53:34.011 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0084937420000000003, 2024-02-05T04:53:34.011 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:34.011 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:34.011 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:53:34.011 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:53:34.011 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:34.011 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 220 2024-02-05T04:53:34.011 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.012 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:34.012 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.012 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:34.012 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.135450+0000", 2024-02-05T04:53:34.012 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.012 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.012 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.012 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:34.012 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.135450+0000", 2024-02-05T04:53:34.012 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.012 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.012 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.012 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:34.012 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.135451+0000", 2024-02-05T04:53:34.013 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.4799999999999997e-07 2024-02-05T04:53:34.013 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.013 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.013 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:34.013 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.135454+0000", 2024-02-05T04:53:34.013 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.3539999999999998e-06 2024-02-05T04:53:34.013 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.013 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.013 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:34.013 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.135458+0000", 2024-02-05T04:53:34.013 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.1509999999999997e-06 2024-02-05T04:53:34.014 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.014 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.014 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:34.014 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.135461+0000", 2024-02-05T04:53:34.014 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.1650000000000002e-06 2024-02-05T04:53:34.014 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.014 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.014 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:34.014 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.137469+0000", 2024-02-05T04:53:34.014 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0020076410000000001 2024-02-05T04:53:34.014 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.014 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.014 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:53:34.014 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.137586+0000", 2024-02-05T04:53:34.014 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000117599 2024-02-05T04:53:34.015 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.015 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.015 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:34.015 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.138018+0000", 2024-02-05T04:53:34.015 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00043188200000000002 2024-02-05T04:53:34.015 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.015 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.015 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:34.015 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.140234+0000", 2024-02-05T04:53:34.015 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0022155159999999998 2024-02-05T04:53:34.015 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.015 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.015 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:53:34.015 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.143857+0000", 2024-02-05T04:53:34.015 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0036236749999999998 2024-02-05T04:53:34.016 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.016 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.016 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:34.016 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.143944+0000", 2024-02-05T04:53:34.016 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.6910999999999995e-05 2024-02-05T04:53:34.016 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.016 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:34.016 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.016 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.016 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.016 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:221 3.3s0 3:ce48de55:::smithi19639362-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:53:34.016 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:29.135468+0000", 2024-02-05T04:53:34.016 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 4.8619515050000004, 2024-02-05T04:53:34.016 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.010019444000000001, 2024-02-05T04:53:34.016 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:34.017 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "started", 2024-02-05T04:53:34.017 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:53:34.017 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:53:34.017 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:34.017 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 221 2024-02-05T04:53:34.017 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.017 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:34.017 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.017 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:34.017 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.135468+0000", 2024-02-05T04:53:34.017 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.017 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.017 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.018 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:34.018 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.135468+0000", 2024-02-05T04:53:34.018 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.018 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.018 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.018 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:34.018 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.135468+0000", 2024-02-05T04:53:34.018 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.4299999999999999e-07 2024-02-05T04:53:34.018 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.018 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.018 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:34.018 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.135469+0000", 2024-02-05T04:53:34.018 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.457e-06 2024-02-05T04:53:34.018 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.018 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.019 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:34.019 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.135475+0000", 2024-02-05T04:53:34.019 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.9939999999999997e-06 2024-02-05T04:53:34.019 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.019 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.019 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:34.019 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.135480+0000", 2024-02-05T04:53:34.019 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.0989999999999998e-06 2024-02-05T04:53:34.019 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.019 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.019 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:34.019 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.138320+0000", 2024-02-05T04:53:34.019 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0028394739999999998 2024-02-05T04:53:34.019 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.019 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.020 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for rw locks", 2024-02-05T04:53:34.020 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.138401+0000", 2024-02-05T04:53:34.020 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.0966999999999999e-05 2024-02-05T04:53:34.020 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.020 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.020 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:34.020 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.144381+0000", 2024-02-05T04:53:34.020 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0059798389999999998 2024-02-05T04:53:34.020 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.020 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.020 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:53:34.020 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.144442+0000", 2024-02-05T04:53:34.020 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.1311999999999998e-05 2024-02-05T04:53:34.021 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.021 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.021 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:34.021 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.145487+0000", 2024-02-05T04:53:34.021 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0010450590000000001 2024-02-05T04:53:34.021 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.021 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:34.021 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.021 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.021 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.021 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(osd.7.23:1 3.3s0 3:c5eac4f6:::smithi19639362-39:head [assert-version v6,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:53:34.021 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:29.137174+0000", 2024-02-05T04:53:34.021 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 4.8602452789999999, 2024-02-05T04:53:34.021 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.006190615, 2024-02-05T04:53:34.022 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:34.022 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "started", 2024-02-05T04:53:34.022 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:34.022 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.022 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:34.022 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.137174+0000", 2024-02-05T04:53:34.022 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.022 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.022 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.022 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:34.022 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.137174+0000", 2024-02-05T04:53:34.022 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.022 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.022 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.022 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:34.022 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.137175+0000", 2024-02-05T04:53:34.022 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.6700000000000001e-06 2024-02-05T04:53:34.023 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.023 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.023 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:34.023 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.137183+0000", 2024-02-05T04:53:34.023 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.4599999999999997e-06 2024-02-05T04:53:34.023 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.023 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.023 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:34.023 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.137196+0000", 2024-02-05T04:53:34.023 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.3139000000000001e-05 2024-02-05T04:53:34.023 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.023 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.023 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:34.023 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.137205+0000", 2024-02-05T04:53:34.023 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.6389999999999998e-06 2024-02-05T04:53:34.023 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.024 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.024 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:34.024 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.138803+0000", 2024-02-05T04:53:34.024 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0015985280000000001 2024-02-05T04:53:34.024 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.024 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.024 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:53:34.024 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.138876+0000", 2024-02-05T04:53:34.024 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.2908000000000001e-05 2024-02-05T04:53:34.024 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.024 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.024 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:34.024 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.143364+0000", 2024-02-05T04:53:34.025 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0044882710000000003 2024-02-05T04:53:34.025 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.025 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:34.025 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.025 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.025 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.025 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.7s4 37/31 ECSubWrite(tid=85, reqid=client.4634.0:243, at_version=0'0, trim_to=0'0, roll_forward_to=37'6))", 2024-02-05T04:53:34.025 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:29.191315+0000", 2024-02-05T04:53:34.025 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 4.8061041800000002, 2024-02-05T04:53:34.025 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0057709909999999996, 2024-02-05T04:53:34.025 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:34.025 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:34.025 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:34.025 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.026 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:34.026 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.191315+0000", 2024-02-05T04:53:34.026 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.026 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.026 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.026 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:34.026 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.191315+0000", 2024-02-05T04:53:34.026 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.026 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.026 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.026 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:34.027 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.191316+0000", 2024-02-05T04:53:34.027 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.8899999999999998e-07 2024-02-05T04:53:34.027 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.027 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.027 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:34.027 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.191338+0000", 2024-02-05T04:53:34.027 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.1903e-05 2024-02-05T04:53:34.027 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.027 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.027 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:34.027 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.191352+0000", 2024-02-05T04:53:34.027 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.4817999999999999e-05 2024-02-05T04:53:34.028 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.028 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.028 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:34.028 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.191361+0000", 2024-02-05T04:53:34.028 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.4090000000000003e-06 2024-02-05T04:53:34.028 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.028 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.028 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:34.028 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.191440+0000", 2024-02-05T04:53:34.028 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.9468000000000006e-05 2024-02-05T04:53:34.028 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.028 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.028 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:34.028 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.191449+0000", 2024-02-05T04:53:34.029 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.9050000000000003e-06 2024-02-05T04:53:34.029 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.029 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.029 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:34.029 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.197039+0000", 2024-02-05T04:53:34.029 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0055895720000000001 2024-02-05T04:53:34.029 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.029 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.029 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:34.029 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.197086+0000", 2024-02-05T04:53:34.029 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.7027000000000002e-05 2024-02-05T04:53:34.029 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.030 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:34.030 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.030 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.030 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.030 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.es1 37/31 ECSubWrite(tid=22, reqid=client.4634.0:249, at_version=37'16, trim_to=0'0, roll_forward_to=37'12))", 2024-02-05T04:53:34.030 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:29.192199+0000", 2024-02-05T04:53:34.030 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 4.8052197899999998, 2024-02-05T04:53:34.030 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0048919699999999998, 2024-02-05T04:53:34.030 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:34.030 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:34.030 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:34.030 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.030 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:34.030 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.192199+0000", 2024-02-05T04:53:34.030 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.031 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.031 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.031 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:34.031 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.192199+0000", 2024-02-05T04:53:34.031 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.031 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.031 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.031 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:34.031 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.192200+0000", 2024-02-05T04:53:34.031 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.5499999999999997e-07 2024-02-05T04:53:34.031 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.031 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.032 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:34.032 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.192222+0000", 2024-02-05T04:53:34.032 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.2042999999999999e-05 2024-02-05T04:53:34.032 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.032 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.032 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:34.032 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.192236+0000", 2024-02-05T04:53:34.032 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.4171e-05 2024-02-05T04:53:34.032 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.032 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.032 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:34.032 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.192246+0000", 2024-02-05T04:53:34.033 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.4790000000000002e-06 2024-02-05T04:53:34.033 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.033 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.033 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:34.033 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.192303+0000", 2024-02-05T04:53:34.033 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.698e-05 2024-02-05T04:53:34.033 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.033 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.033 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:34.033 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.192310+0000", 2024-02-05T04:53:34.033 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.9369999999999998e-06 2024-02-05T04:53:34.033 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.034 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.034 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:34.034 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.197059+0000", 2024-02-05T04:53:34.034 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0047497709999999999 2024-02-05T04:53:34.034 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.034 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.034 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:34.034 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.197091+0000", 2024-02-05T04:53:34.034 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.1834e-05 2024-02-05T04:53:34.034 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.034 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:34.035 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.035 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.035 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.035 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.1s2 37/31 ECSubWrite(tid=86, reqid=client.4634.0:247, at_version=37'15, trim_to=0'0, roll_forward_to=35'13))", 2024-02-05T04:53:34.035 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:29.192941+0000", 2024-02-05T04:53:34.035 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 4.8044779499999999, 2024-02-05T04:53:34.035 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0055093850000000003, 2024-02-05T04:53:34.035 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:34.035 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:34.035 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:34.035 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.035 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:34.035 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.192941+0000", 2024-02-05T04:53:34.036 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.036 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.036 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.036 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:34.036 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.192941+0000", 2024-02-05T04:53:34.036 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.036 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.036 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.036 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:34.036 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.192948+0000", 2024-02-05T04:53:34.036 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.3000000000000004e-06 2024-02-05T04:53:34.036 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.036 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.036 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:34.037 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.193266+0000", 2024-02-05T04:53:34.037 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00031747399999999999 2024-02-05T04:53:34.037 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.037 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.037 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:34.037 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.193282+0000", 2024-02-05T04:53:34.037 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.5716e-05 2024-02-05T04:53:34.037 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.037 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.037 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:34.037 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.193291+0000", 2024-02-05T04:53:34.037 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.1090000000000004e-06 2024-02-05T04:53:34.037 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.037 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.038 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:34.038 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.193347+0000", 2024-02-05T04:53:34.038 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.6628000000000001e-05 2024-02-05T04:53:34.038 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.038 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.038 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:34.038 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.193353+0000", 2024-02-05T04:53:34.038 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.9220000000000002e-06 2024-02-05T04:53:34.038 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.038 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.038 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:34.038 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.198406+0000", 2024-02-05T04:53:34.038 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00505303 2024-02-05T04:53:34.038 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.039 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.039 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:34.039 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.198450+0000", 2024-02-05T04:53:34.039 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.4206000000000003e-05 2024-02-05T04:53:34.039 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.039 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:34.039 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.039 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.039 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.039 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:260 3.3s0 3:c5eac4f6:::smithi19639362-39:head [read 0~638976,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-02-05T04:53:34.039 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:30.131993+0000", 2024-02-05T04:53:34.039 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 3.8654265080000001, 2024-02-05T04:53:34.039 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00496073, 2024-02-05T04:53:34.039 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:34.040 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "started", 2024-02-05T04:53:34.040 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:53:34.040 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:53:34.040 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:34.040 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 260 2024-02-05T04:53:34.040 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.040 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:34.040 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.040 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:34.040 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.131993+0000", 2024-02-05T04:53:34.040 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.040 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.040 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.040 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:34.040 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.131993+0000", 2024-02-05T04:53:34.041 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.041 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.041 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.041 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:34.041 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.131994+0000", 2024-02-05T04:53:34.041 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0160000000000001e-06 2024-02-05T04:53:34.041 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.041 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.041 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:34.041 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.131999+0000", 2024-02-05T04:53:34.041 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.3720000000000004e-06 2024-02-05T04:53:34.041 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.041 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.041 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:34.041 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.132017+0000", 2024-02-05T04:53:34.041 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.7892000000000001e-05 2024-02-05T04:53:34.041 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.042 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.042 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:34.042 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.132025+0000", 2024-02-05T04:53:34.042 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.1859999999999992e-06 2024-02-05T04:53:34.042 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.042 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.042 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:34.042 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.132563+0000", 2024-02-05T04:53:34.042 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00053790299999999995 2024-02-05T04:53:34.042 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.042 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.042 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:53:34.042 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.132634+0000", 2024-02-05T04:53:34.042 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.0961e-05 2024-02-05T04:53:34.042 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.043 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.043 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:34.043 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.136953+0000", 2024-02-05T04:53:34.043 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0043194000000000001 2024-02-05T04:53:34.043 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.043 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:34.043 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.043 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.043 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.043 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:261 3.3s0 3:c5eac4f6:::smithi19639362-39:head [read 0~638976,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-02-05T04:53:34.043 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:30.132041+0000", 2024-02-05T04:53:34.043 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 3.8653776899999999, 2024-02-05T04:53:34.043 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0057721969999999997, 2024-02-05T04:53:34.044 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:34.044 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "started", 2024-02-05T04:53:34.044 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:53:34.044 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:53:34.044 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:34.044 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 261 2024-02-05T04:53:34.044 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.044 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:34.044 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.044 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:34.044 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.132041+0000", 2024-02-05T04:53:34.044 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.044 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.044 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.045 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:34.045 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.132041+0000", 2024-02-05T04:53:34.045 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.045 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.045 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.045 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:34.045 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.132042+0000", 2024-02-05T04:53:34.045 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.5000000000000002e-07 2024-02-05T04:53:34.045 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.045 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.045 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:34.045 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.132046+0000", 2024-02-05T04:53:34.045 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.4680000000000003e-06 2024-02-05T04:53:34.045 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.046 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.046 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:34.046 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.132054+0000", 2024-02-05T04:53:34.046 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.8609999999999998e-06 2024-02-05T04:53:34.046 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.046 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.046 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:34.046 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.132060+0000", 2024-02-05T04:53:34.046 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.711e-06 2024-02-05T04:53:34.046 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.046 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.046 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:34.046 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.133280+0000", 2024-02-05T04:53:34.046 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.001219574 2024-02-05T04:53:34.047 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.047 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.047 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:53:34.047 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.133394+0000", 2024-02-05T04:53:34.047 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000114666 2024-02-05T04:53:34.047 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.047 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.047 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:34.047 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.137814+0000", 2024-02-05T04:53:34.047 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0044192670000000002 2024-02-05T04:53:34.047 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.047 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:34.047 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.047 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.047 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.047 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpReadReply(3.3s0 38/31 ECSubReadReply(tid=37, attrs_read=0))", 2024-02-05T04:53:34.048 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:30.133863+0000", 2024-02-05T04:53:34.048 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 3.8635557780000003, 2024-02-05T04:53:34.048 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0031148080000000002, 2024-02-05T04:53:34.048 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:34.048 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:34.048 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:34.048 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.048 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:34.048 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.133863+0000", 2024-02-05T04:53:34.048 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.048 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.048 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.048 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:34.049 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.133863+0000", 2024-02-05T04:53:34.049 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.049 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.049 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.049 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:34.049 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.133863+0000", 2024-02-05T04:53:34.049 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.049 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.049 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.049 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:34.049 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.133863+0000", 2024-02-05T04:53:34.049 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.049 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.049 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.050 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:34.050 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.133966+0000", 2024-02-05T04:53:34.050 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00010322799999999999 2024-02-05T04:53:34.050 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.050 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.050 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:34.050 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.133975+0000", 2024-02-05T04:53:34.050 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.7600000000000008e-06 2024-02-05T04:53:34.050 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.050 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.050 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:34.050 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.136539+0000", 2024-02-05T04:53:34.050 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0025638309999999999 2024-02-05T04:53:34.050 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.050 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.051 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:34.051 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.136978+0000", 2024-02-05T04:53:34.051 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00043898899999999998 2024-02-05T04:53:34.051 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.051 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:34.051 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.051 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.051 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.051 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.4s5 38/31 ECSubWrite(tid=27, reqid=osd.1.0:27, at_version=38'16, trim_to=0'0, roll_forward_to=37'14))", 2024-02-05T04:53:34.051 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:30.134188+0000", 2024-02-05T04:53:34.051 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 3.8632309970000001, 2024-02-05T04:53:34.051 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0042728870000000004, 2024-02-05T04:53:34.051 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:34.051 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:34.052 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:34.052 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.052 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:34.052 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.134188+0000", 2024-02-05T04:53:34.052 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.052 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.052 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.052 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:34.052 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.134188+0000", 2024-02-05T04:53:34.052 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.052 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.052 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.052 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:34.052 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.134189+0000", 2024-02-05T04:53:34.052 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.9200000000000001e-07 2024-02-05T04:53:34.053 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.053 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.053 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:34.053 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.134209+0000", 2024-02-05T04:53:34.053 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.0781999999999999e-05 2024-02-05T04:53:34.053 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.053 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.053 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:34.053 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.134220+0000", 2024-02-05T04:53:34.053 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0857e-05 2024-02-05T04:53:34.053 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.053 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.053 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:34.053 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.134228+0000", 2024-02-05T04:53:34.054 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.5939999999999996e-06 2024-02-05T04:53:34.054 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.054 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.054 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:34.054 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.134297+0000", 2024-02-05T04:53:34.054 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.8825999999999995e-05 2024-02-05T04:53:34.054 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.054 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.054 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:34.054 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.134305+0000", 2024-02-05T04:53:34.054 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.8419999999999994e-06 2024-02-05T04:53:34.054 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.054 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.054 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:34.054 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.138418+0000", 2024-02-05T04:53:34.055 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0041126749999999997 2024-02-05T04:53:34.055 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.055 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.055 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:34.055 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.138461+0000", 2024-02-05T04:53:34.055 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.2818999999999998e-05 2024-02-05T04:53:34.055 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.055 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:34.055 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.055 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.055 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.055 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.es1 38/31 ECSubWrite(tid=25, reqid=osd.2.0:25, at_version=38'20, trim_to=0'0, roll_forward_to=37'16))", 2024-02-05T04:53:34.055 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:30.134637+0000", 2024-02-05T04:53:34.055 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 3.8627821170000001, 2024-02-05T04:53:34.055 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0038900660000000002, 2024-02-05T04:53:34.056 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:34.056 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:34.056 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:34.056 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.056 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:34.056 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.134637+0000", 2024-02-05T04:53:34.056 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.056 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.056 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.056 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:34.056 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.134637+0000", 2024-02-05T04:53:34.056 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.056 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.056 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.056 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:34.056 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.134637+0000", 2024-02-05T04:53:34.057 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.7400000000000003e-07 2024-02-05T04:53:34.057 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.057 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.057 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:34.057 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.134659+0000", 2024-02-05T04:53:34.057 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.1328999999999999e-05 2024-02-05T04:53:34.057 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.057 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.057 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:34.057 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.134670+0000", 2024-02-05T04:53:34.057 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1416e-05 2024-02-05T04:53:34.057 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.057 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.057 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:34.058 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.134679+0000", 2024-02-05T04:53:34.058 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.755e-06 2024-02-05T04:53:34.058 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.058 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.058 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:34.058 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.134744+0000", 2024-02-05T04:53:34.058 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.4920999999999997e-05 2024-02-05T04:53:34.058 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.058 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.058 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:34.058 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.134753+0000", 2024-02-05T04:53:34.058 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.5680000000000005e-06 2024-02-05T04:53:34.058 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.058 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.058 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:34.059 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.138496+0000", 2024-02-05T04:53:34.059 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0037420769999999999 2024-02-05T04:53:34.059 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.059 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.059 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:34.059 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.138527+0000", 2024-02-05T04:53:34.059 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.1426e-05 2024-02-05T04:53:34.059 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.059 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:34.059 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.059 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.059 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.059 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.4s5 38/31 ECSubWrite(tid=28, reqid=osd.1.0:27, at_version=0'0, trim_to=0'0, roll_forward_to=38'16))", 2024-02-05T04:53:34.059 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:30.139653+0000", 2024-02-05T04:53:34.060 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 3.8577663380000002, 2024-02-05T04:53:34.060 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0060967790000000001, 2024-02-05T04:53:34.060 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:34.060 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:34.060 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:34.060 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.060 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:34.060 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.139653+0000", 2024-02-05T04:53:34.060 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.060 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.060 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.060 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:34.060 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.139653+0000", 2024-02-05T04:53:34.060 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.061 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.061 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.061 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:34.061 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.139653+0000", 2024-02-05T04:53:34.061 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.3999999999999997e-07 2024-02-05T04:53:34.061 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.061 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.061 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:34.061 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.139661+0000", 2024-02-05T04:53:34.061 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.2679999999999994e-06 2024-02-05T04:53:34.061 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.061 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.061 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:34.061 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.139670+0000", 2024-02-05T04:53:34.061 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.7779999999999996e-06 2024-02-05T04:53:34.062 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.062 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.062 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:34.062 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.139679+0000", 2024-02-05T04:53:34.062 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.6610000000000006e-06 2024-02-05T04:53:34.062 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.062 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.062 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:34.062 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.139801+0000", 2024-02-05T04:53:34.062 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000122149 2024-02-05T04:53:34.062 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.062 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.062 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:34.062 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.139812+0000", 2024-02-05T04:53:34.063 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0902e-05 2024-02-05T04:53:34.063 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.063 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.063 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:34.063 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.145702+0000", 2024-02-05T04:53:34.063 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.005890658 2024-02-05T04:53:34.063 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.063 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.063 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:34.063 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.145749+0000", 2024-02-05T04:53:34.063 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.7023e-05 2024-02-05T04:53:34.063 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.063 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:34.064 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.064 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.064 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.064 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.1s2 38/31 ECSubWrite(tid=89, reqid=osd.13.0:89, at_version=38'17, trim_to=0'0, roll_forward_to=37'15))", 2024-02-05T04:53:34.064 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:30.140160+0000", 2024-02-05T04:53:34.064 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 3.857258936, 2024-02-05T04:53:34.064 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0056285220000000004, 2024-02-05T04:53:34.064 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:34.064 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:34.064 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:34.064 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.064 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:34.065 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.140160+0000", 2024-02-05T04:53:34.065 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.065 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.065 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.065 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:34.065 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.140160+0000", 2024-02-05T04:53:34.065 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.065 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.065 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.065 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:34.065 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.140160+0000", 2024-02-05T04:53:34.065 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.0100000000000001e-07 2024-02-05T04:53:34.065 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.065 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.066 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:34.066 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.140183+0000", 2024-02-05T04:53:34.066 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.2708999999999999e-05 2024-02-05T04:53:34.066 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.066 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.066 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:34.066 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.140197+0000", 2024-02-05T04:53:34.066 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.3867e-05 2024-02-05T04:53:34.066 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.066 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.066 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:34.066 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.140206+0000", 2024-02-05T04:53:34.066 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.1900000000000001e-06 2024-02-05T04:53:34.067 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.067 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.067 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:34.067 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.140303+0000", 2024-02-05T04:53:34.067 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.6751999999999998e-05 2024-02-05T04:53:34.067 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.067 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.067 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:34.067 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.140314+0000", 2024-02-05T04:53:34.067 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1260000000000001e-05 2024-02-05T04:53:34.067 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.067 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.067 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:34.067 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.145762+0000", 2024-02-05T04:53:34.068 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0054476330000000003 2024-02-05T04:53:34.068 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.068 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.068 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:34.068 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.145789+0000", 2024-02-05T04:53:34.068 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.6910000000000002e-05 2024-02-05T04:53:34.068 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.068 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:34.068 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.068 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.068 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.068 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.6s2 38/31 ECSubWrite(tid=88, reqid=client.4634.0:262, at_version=38'9, trim_to=0'0, roll_forward_to=37'8))", 2024-02-05T04:53:34.068 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:30.140280+0000", 2024-02-05T04:53:34.068 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 3.8571387850000001, 2024-02-05T04:53:34.069 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0054517059999999997, 2024-02-05T04:53:34.069 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:34.069 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:34.069 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:34.069 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.069 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:34.069 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.140280+0000", 2024-02-05T04:53:34.069 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.069 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.069 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.069 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:34.069 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.140280+0000", 2024-02-05T04:53:34.069 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.069 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.070 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.070 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:34.070 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.140281+0000", 2024-02-05T04:53:34.070 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.7299999999999996e-07 2024-02-05T04:53:34.070 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.070 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.070 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:34.070 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.140304+0000", 2024-02-05T04:53:34.070 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.3595e-05 2024-02-05T04:53:34.070 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.070 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.070 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:34.070 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.140357+0000", 2024-02-05T04:53:34.070 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.2485999999999999e-05 2024-02-05T04:53:34.070 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.071 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.071 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:34.071 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.140366+0000", 2024-02-05T04:53:34.071 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.6150000000000003e-06 2024-02-05T04:53:34.071 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.071 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.071 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:34.071 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.140552+0000", 2024-02-05T04:53:34.071 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00018572099999999999 2024-02-05T04:53:34.071 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.071 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.071 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:34.071 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.140560+0000", 2024-02-05T04:53:34.071 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.2239999999999995e-06 2024-02-05T04:53:34.071 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.071 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.072 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:34.072 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.145684+0000", 2024-02-05T04:53:34.072 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0051238170000000001 2024-02-05T04:53:34.072 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.072 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.072 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:34.072 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.145732+0000", 2024-02-05T04:53:34.072 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.7675e-05 2024-02-05T04:53:34.072 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.072 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:34.072 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.072 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.072 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.072 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.6s2 38/31 ECSubWrite(tid=90, reqid=osd.13.0:90, at_version=38'11, trim_to=0'0, roll_forward_to=37'8))", 2024-02-05T04:53:34.073 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:30.141748+0000", 2024-02-05T04:53:34.073 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 3.8556709639999998, 2024-02-05T04:53:34.073 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0052058299999999998, 2024-02-05T04:53:34.073 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:34.073 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:34.073 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:34.073 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.073 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:34.073 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.141748+0000", 2024-02-05T04:53:34.073 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.073 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.073 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.073 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:34.073 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.141748+0000", 2024-02-05T04:53:34.074 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.074 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.074 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.074 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:34.074 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.141749+0000", 2024-02-05T04:53:34.074 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.2099999999999995e-07 2024-02-05T04:53:34.074 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.074 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.074 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:34.074 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.141773+0000", 2024-02-05T04:53:34.074 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.3677999999999999e-05 2024-02-05T04:53:34.074 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.074 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.074 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:34.075 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.141787+0000", 2024-02-05T04:53:34.075 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.4847e-05 2024-02-05T04:53:34.075 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.075 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.075 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:34.075 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.141797+0000", 2024-02-05T04:53:34.075 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.2369999999999998e-06 2024-02-05T04:53:34.075 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.075 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.075 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:34.075 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.141874+0000", 2024-02-05T04:53:34.075 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.7706000000000004e-05 2024-02-05T04:53:34.075 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.076 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.076 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:34.076 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.141884+0000", 2024-02-05T04:53:34.076 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.0829999999999993e-06 2024-02-05T04:53:34.076 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.076 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.076 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:34.076 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.146917+0000", 2024-02-05T04:53:34.076 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0050333929999999997 2024-02-05T04:53:34.076 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.076 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.076 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:34.076 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.146954+0000", 2024-02-05T04:53:34.077 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.6965000000000003e-05 2024-02-05T04:53:34.077 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.077 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:34.077 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.077 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.077 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.077 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.es1 38/31 ECSubWrite(tid=26, reqid=osd.2.0:25, at_version=0'0, trim_to=0'0, roll_forward_to=38'20))", 2024-02-05T04:53:34.077 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:30.146094+0000", 2024-02-05T04:53:34.077 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 3.8513249950000001, 2024-02-05T04:53:34.077 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0048505620000000001, 2024-02-05T04:53:34.077 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:34.077 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:34.077 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:34.077 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.077 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:34.078 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.146094+0000", 2024-02-05T04:53:34.078 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.078 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.078 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.078 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:34.078 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.146094+0000", 2024-02-05T04:53:34.078 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.078 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.078 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.078 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:34.078 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.146095+0000", 2024-02-05T04:53:34.078 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.2500000000000004e-07 2024-02-05T04:53:34.078 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.079 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.079 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:34.079 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.146110+0000", 2024-02-05T04:53:34.079 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.471e-05 2024-02-05T04:53:34.079 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.079 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.079 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:34.079 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.146127+0000", 2024-02-05T04:53:34.079 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.7031e-05 2024-02-05T04:53:34.079 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.079 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.079 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:34.079 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.146138+0000", 2024-02-05T04:53:34.079 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1280999999999999e-05 2024-02-05T04:53:34.079 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.080 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.080 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:34.080 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.146377+0000", 2024-02-05T04:53:34.080 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000239015 2024-02-05T04:53:34.080 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.080 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.080 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:34.080 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.146390+0000", 2024-02-05T04:53:34.080 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.2894000000000001e-05 2024-02-05T04:53:34.080 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.080 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.080 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:34.080 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.150906+0000", 2024-02-05T04:53:34.080 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0045157360000000002 2024-02-05T04:53:34.080 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.081 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.081 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:34.081 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.150945+0000", 2024-02-05T04:53:34.081 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.8970000000000001e-05 2024-02-05T04:53:34.081 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.081 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:34.081 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.081 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.081 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.081 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.1s2 38/31 ECSubWrite(tid=91, reqid=osd.13.0:89, at_version=0'0, trim_to=0'0, roll_forward_to=38'17))", 2024-02-05T04:53:34.081 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:30.148839+0000", 2024-02-05T04:53:34.081 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 3.8485797320000001, 2024-02-05T04:53:34.081 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0049892369999999997, 2024-02-05T04:53:34.081 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:34.082 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:34.082 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:34.082 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.082 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:34.082 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.148839+0000", 2024-02-05T04:53:34.082 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.082 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.082 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.082 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:34.082 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.148839+0000", 2024-02-05T04:53:34.082 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.082 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.082 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.082 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:34.083 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.148840+0000", 2024-02-05T04:53:34.083 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1939999999999999e-06 2024-02-05T04:53:34.083 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.083 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.083 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:34.083 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.148850+0000", 2024-02-05T04:53:34.083 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.4150000000000005e-06 2024-02-05T04:53:34.083 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.083 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.083 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:34.083 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.148873+0000", 2024-02-05T04:53:34.083 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.2918999999999999e-05 2024-02-05T04:53:34.083 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.083 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.083 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:34.084 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.148883+0000", 2024-02-05T04:53:34.084 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.7459999999999995e-06 2024-02-05T04:53:34.084 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.084 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.084 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:34.084 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.148962+0000", 2024-02-05T04:53:34.084 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.9685999999999996e-05 2024-02-05T04:53:34.084 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.084 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.084 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:34.084 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.148971+0000", 2024-02-05T04:53:34.084 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.9020000000000005e-06 2024-02-05T04:53:34.084 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.084 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.085 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:34.085 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.153669+0000", 2024-02-05T04:53:34.085 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0046982969999999997 2024-02-05T04:53:34.085 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.085 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.085 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:34.085 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.153829+0000", 2024-02-05T04:53:34.085 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000159078 2024-02-05T04:53:34.085 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.085 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:34.085 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.085 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.085 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.085 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.7s4 38/31 ECSubWrite(tid=92, reqid=osd.13.0:92, at_version=38'8, trim_to=0'0, roll_forward_to=37'6))", 2024-02-05T04:53:34.086 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:30.149691+0000", 2024-02-05T04:53:34.086 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 3.847727517, 2024-02-05T04:53:34.086 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0041311400000000002, 2024-02-05T04:53:34.086 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:34.086 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:34.086 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:34.086 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.086 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:34.086 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.149691+0000", 2024-02-05T04:53:34.086 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.086 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.087 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.087 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:34.087 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.149691+0000", 2024-02-05T04:53:34.087 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.087 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.087 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.087 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:34.087 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.149692+0000", 2024-02-05T04:53:34.087 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.5300000000000003e-07 2024-02-05T04:53:34.087 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.087 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.087 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:34.087 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.149727+0000", 2024-02-05T04:53:34.087 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.4609e-05 2024-02-05T04:53:34.088 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.088 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.088 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:34.088 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.149742+0000", 2024-02-05T04:53:34.088 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.5553000000000002e-05 2024-02-05T04:53:34.088 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.088 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.088 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:34.088 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.149751+0000", 2024-02-05T04:53:34.088 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.9649999999999997e-06 2024-02-05T04:53:34.088 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.088 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.088 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:34.089 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.149827+0000", 2024-02-05T04:53:34.089 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.5291999999999997e-05 2024-02-05T04:53:34.089 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.089 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.089 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:34.089 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.149837+0000", 2024-02-05T04:53:34.089 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0129000000000001e-05 2024-02-05T04:53:34.089 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.089 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.089 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:34.089 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.153696+0000", 2024-02-05T04:53:34.089 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0038594219999999999 2024-02-05T04:53:34.089 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.089 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.089 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:34.090 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.153823+0000", 2024-02-05T04:53:34.090 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00012641700000000001 2024-02-05T04:53:34.090 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.090 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:34.090 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.090 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.090 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:34.090 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-02-05T04:53:34.091 INFO:tasks.ceph.osd.3.smithi196.stderr:2024-02-05T04:53:34.091+0000 7fc8817c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34605) UID: 0 2024-02-05T04:53:34.092 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-02-05T04:53:34.154 INFO:teuthology.orchestra.run.smithi062.stderr:reweighted osd.13 to 1 (10000) 2024-02-05T04:53:34.158 INFO:tasks.rados.rados.0.smithi196.stdout:134: done (1 left) 2024-02-05T04:53:34.159 INFO:tasks.rados.rados.0.smithi196.stdout:135: done (0 left) 2024-02-05T04:53:34.159 INFO:tasks.rados.rados.0.smithi196.stdout:136: copy_from oid 21 from oid 28 current snap is 6 2024-02-05T04:53:34.159 INFO:tasks.rados.rados.0.smithi196.stdout:137: delete oid 34 current snap is 6 2024-02-05T04:53:34.161 INFO:tasks.rados.rados.0.smithi196.stdout:137: done (1 left) 2024-02-05T04:53:34.161 INFO:tasks.rados.rados.0.smithi196.stdout:append oid 19 current snap is 6 2024-02-05T04:53:34.162 INFO:tasks.rados.rados.0.smithi196.stdout:138: seq_num 84 ranges {311296=655360,966656=638976,1605632=131072} 2024-02-05T04:53:34.167 INFO:tasks.rados.rados.0.smithi196.stdout:138: writing smithi19639362-19 from 311296 to 966656 tid 1 2024-02-05T04:53:34.169 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 14 1 2024-02-05T04:53:34.170 INFO:tasks.rados.rados.0.smithi196.stdout:138: writing smithi19639362-19 from 966656 to 1605632 tid 2 2024-02-05T04:53:34.172 INFO:tasks.rados.rados.0.smithi196.stdout:138: writing smithi19639362-19 from 1605632 to 1736704 tid 3 2024-02-05T04:53:34.172 INFO:tasks.rados.rados.0.smithi196.stdout:139: read oid 25 snap -1 2024-02-05T04:53:34.172 INFO:tasks.rados.rados.0.smithi196.stdout:139: expect (ObjNum 38 snap 0 seq_num 38) 2024-02-05T04:53:34.173 INFO:tasks.rados.rados.0.smithi196.stdout:140: rmattr oid 43 current snap is 6 2024-02-05T04:53:34.173 INFO:tasks.rados.rados.0.smithi196.stdout:140: done (3 left) 2024-02-05T04:53:34.173 INFO:tasks.rados.rados.0.smithi196.stdout:141: rmattr oid 9 current snap is 6 2024-02-05T04:53:34.173 INFO:tasks.rados.rados.0.smithi196.stdout:141: done (3 left) 2024-02-05T04:53:34.173 INFO:tasks.rados.rados.0.smithi196.stdout:142: copy_from oid 30 from oid 16 current snap is 6 2024-02-05T04:53:34.173 INFO:tasks.rados.rados.0.smithi196.stdout:136: finishing copy_from racing read to smithi19639362-21 2024-02-05T04:53:34.173 INFO:tasks.rados.rados.0.smithi196.stdout:136: finishing copy_from to smithi19639362-21 2024-02-05T04:53:34.173 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 21 v 6 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-02-05T04:53:34.173 INFO:tasks.rados.rados.0.smithi196.stdout:136: done (3 left) 2024-02-05T04:53:34.173 INFO:tasks.rados.rados.0.smithi196.stdout:143: rmattr oid 12 current snap is 6 2024-02-05T04:53:34.173 INFO:tasks.rados.rados.0.smithi196.stdout:143: done (3 left) 2024-02-05T04:53:34.174 INFO:tasks.rados.rados.0.smithi196.stdout:append oid 9 current snap is 6 2024-02-05T04:53:34.174 INFO:tasks.rados.rados.0.smithi196.stdout:144: seq_num 85 ranges {1179648=671744,1851392=770048} 2024-02-05T04:53:34.182 INFO:tasks.rados.rados.0.smithi196.stdout:144: writing smithi19639362-9 from 1179648 to 1851392 tid 1 2024-02-05T04:53:34.186 INFO:tasks.rados.rados.0.smithi196.stdout:144: writing smithi19639362-9 from 1851392 to 2621440 tid 2 2024-02-05T04:53:34.186 INFO:tasks.rados.rados.0.smithi196.stdout:145: snap_remove snap 3 2024-02-05T04:53:34.192 INFO:tasks.ceph.osd.15.smithi196.stderr:2024-02-05T04:53:34.191+0000 7f1facc45640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 34611) UID: 0 2024-02-05T04:53:34.211 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-02-05T04:53:34.212 INFO:teuthology.orchestra.run.smithi062.stdout: "ops": [], 2024-02-05T04:53:34.212 INFO:teuthology.orchestra.run.smithi062.stdout: "num_ops": 0 2024-02-05T04:53:34.212 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-02-05T04:53:34.220 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-02-05T04:53:34.294 INFO:tasks.ceph.osd.15.smithi196.stderr:2024-02-05T04:53:34.293+0000 7f1facc45640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 34611) UID: 0 2024-02-05T04:53:34.369 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-02-05T04:53:34.369 INFO:teuthology.orchestra.run.smithi062.stdout: "ops": [], 2024-02-05T04:53:34.369 INFO:teuthology.orchestra.run.smithi062.stdout: "complaint_time": 30, 2024-02-05T04:53:34.369 INFO:teuthology.orchestra.run.smithi062.stdout: "num_blocked_ops": 0 2024-02-05T04:53:34.369 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-02-05T04:53:34.378 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-02-05T04:53:34.395 INFO:tasks.ceph.osd.2.smithi150.stderr:2024-02-05T04:53:34.394+0000 7f7337bf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 34014) UID: 0 2024-02-05T04:53:34.488 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-02-05T04:53:34.488 INFO:teuthology.orchestra.run.smithi062.stdout: "size": 20, 2024-02-05T04:53:34.489 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 600, 2024-02-05T04:53:34.489 INFO:teuthology.orchestra.run.smithi062.stdout: "ops": [ 2024-02-05T04:53:34.489 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.489 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4209.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-02-05T04:53:34.489 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:08.919510+0000", 2024-02-05T04:53:34.489 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 25.567022543, 2024-02-05T04:53:34.489 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.70688627900000001, 2024-02-05T04:53:34.489 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:34.489 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:34.489 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:53:34.489 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4209", 2024-02-05T04:53:34.489 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3672061555", 2024-02-05T04:53:34.489 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 1 2024-02-05T04:53:34.489 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.490 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:34.490 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.490 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:34.490 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:08.919510+0000", 2024-02-05T04:53:34.490 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.490 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.490 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.490 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:34.490 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:08.919510+0000", 2024-02-05T04:53:34.490 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.490 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.490 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.490 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:34.491 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:08.919512+0000", 2024-02-05T04:53:34.491 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1620000000000001e-06 2024-02-05T04:53:34.491 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.491 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.491 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:34.491 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:08.919525+0000", 2024-02-05T04:53:34.491 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.3369e-05 2024-02-05T04:53:34.491 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.491 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.491 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:34.491 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:08.919536+0000", 2024-02-05T04:53:34.491 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0621e-05 2024-02-05T04:53:34.491 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.491 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.491 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:34.492 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:08.919550+0000", 2024-02-05T04:53:34.492 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.403e-05 2024-02-05T04:53:34.492 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.492 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.492 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:34.492 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:08.919625+0000", 2024-02-05T04:53:34.492 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.5328999999999997e-05 2024-02-05T04:53:34.492 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.492 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.492 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for peered", 2024-02-05T04:53:34.492 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:08.919634+0000", 2024-02-05T04:53:34.492 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.4879999999999996e-06 2024-02-05T04:53:34.492 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.492 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.493 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:34.493 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.624946+0000", 2024-02-05T04:53:34.493 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.70531134600000001 2024-02-05T04:53:34.493 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.493 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.493 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:53:34.493 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.625129+0000", 2024-02-05T04:53:34.493 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00018288199999999999 2024-02-05T04:53:34.493 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.493 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.493 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for subops from 2", 2024-02-05T04:53:34.493 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.625339+0000", 2024-02-05T04:53:34.493 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00021016900000000001 2024-02-05T04:53:34.493 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.494 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.494 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "op_commit", 2024-02-05T04:53:34.494 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.625989+0000", 2024-02-05T04:53:34.494 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00065006700000000005 2024-02-05T04:53:34.494 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.494 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.494 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_commit_rec", 2024-02-05T04:53:34.494 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.626301+0000", 2024-02-05T04:53:34.494 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00031220800000000002 2024-02-05T04:53:34.494 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.494 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.494 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:53:34.494 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.626352+0000", 2024-02-05T04:53:34.494 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.0973000000000001e-05 2024-02-05T04:53:34.494 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.495 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.495 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:34.495 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.626397+0000", 2024-02-05T04:53:34.495 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.4635000000000002e-05 2024-02-05T04:53:34.495 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.495 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:34.495 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.495 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.495 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.495 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4209.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-02-05T04:53:34.495 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:09.696463+0000", 2024-02-05T04:53:34.495 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 24.790070312000001, 2024-02-05T04:53:34.495 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0095939449999999996, 2024-02-05T04:53:34.495 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:34.495 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:34.496 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:53:34.496 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4209", 2024-02-05T04:53:34.496 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3672061555", 2024-02-05T04:53:34.496 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 85 2024-02-05T04:53:34.496 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.496 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:34.496 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.496 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:34.496 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.696463+0000", 2024-02-05T04:53:34.496 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.496 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.496 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.496 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:34.496 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.696463+0000", 2024-02-05T04:53:34.496 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.497 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.497 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.497 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:34.497 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.696463+0000", 2024-02-05T04:53:34.497 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.9400000000000001e-07 2024-02-05T04:53:34.497 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.497 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.497 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:34.497 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.696465+0000", 2024-02-05T04:53:34.497 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.1019999999999999e-06 2024-02-05T04:53:34.497 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.497 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.497 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:34.497 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.696470+0000", 2024-02-05T04:53:34.497 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.6099999999999999e-06 2024-02-05T04:53:34.497 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.498 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.498 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:34.498 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.696473+0000", 2024-02-05T04:53:34.498 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.8949999999999998e-06 2024-02-05T04:53:34.498 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.498 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.498 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:34.498 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.697422+0000", 2024-02-05T04:53:34.498 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00094939000000000004 2024-02-05T04:53:34.498 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.498 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.498 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:53:34.498 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.697489+0000", 2024-02-05T04:53:34.498 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.7293999999999996e-05 2024-02-05T04:53:34.498 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.498 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.499 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for subops from 2", 2024-02-05T04:53:34.499 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.697604+0000", 2024-02-05T04:53:34.499 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000114288 2024-02-05T04:53:34.499 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.499 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.499 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "op_commit", 2024-02-05T04:53:34.499 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.702424+0000", 2024-02-05T04:53:34.499 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0048202649999999998 2024-02-05T04:53:34.499 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.499 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.499 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_commit_rec", 2024-02-05T04:53:34.499 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.706007+0000", 2024-02-05T04:53:34.499 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.003583059 2024-02-05T04:53:34.499 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.500 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.500 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:53:34.500 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.706037+0000", 2024-02-05T04:53:34.500 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.9944999999999999e-05 2024-02-05T04:53:34.500 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.500 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.500 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:34.500 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.706057+0000", 2024-02-05T04:53:34.500 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.9803e-05 2024-02-05T04:53:34.500 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.500 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:34.500 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.500 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.500 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.500 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4209.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-02-05T04:53:34.501 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:09.696477+0000", 2024-02-05T04:53:34.501 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 24.790055958, 2024-02-05T04:53:34.501 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0096605600000000003, 2024-02-05T04:53:34.501 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:34.501 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:34.501 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:53:34.501 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4209", 2024-02-05T04:53:34.501 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3672061555", 2024-02-05T04:53:34.501 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 86 2024-02-05T04:53:34.501 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.501 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:34.501 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.501 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:34.501 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.696477+0000", 2024-02-05T04:53:34.502 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.502 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.502 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.502 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:34.502 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.696477+0000", 2024-02-05T04:53:34.502 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.502 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.502 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.502 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:34.502 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.696477+0000", 2024-02-05T04:53:34.502 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.3200000000000001e-07 2024-02-05T04:53:34.502 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.502 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.502 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:34.502 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.696480+0000", 2024-02-05T04:53:34.502 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.3779999999999999e-06 2024-02-05T04:53:34.503 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.503 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.503 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:34.503 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.696483+0000", 2024-02-05T04:53:34.503 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.23e-06 2024-02-05T04:53:34.503 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.503 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.503 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:34.503 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.696485+0000", 2024-02-05T04:53:34.503 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.4200000000000001e-06 2024-02-05T04:53:34.503 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.503 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.503 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:34.503 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.697977+0000", 2024-02-05T04:53:34.503 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0014921299999999999 2024-02-05T04:53:34.504 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.504 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.504 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:53:34.504 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.698018+0000", 2024-02-05T04:53:34.504 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.0676999999999997e-05 2024-02-05T04:53:34.504 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.504 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.504 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for subops from 2", 2024-02-05T04:53:34.504 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.698082+0000", 2024-02-05T04:53:34.504 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.3466000000000005e-05 2024-02-05T04:53:34.504 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.504 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.504 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "op_commit", 2024-02-05T04:53:34.504 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.702654+0000", 2024-02-05T04:53:34.505 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0045719710000000002 2024-02-05T04:53:34.505 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.505 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.505 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_commit_rec", 2024-02-05T04:53:34.505 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.706093+0000", 2024-02-05T04:53:34.505 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0034398749999999998 2024-02-05T04:53:34.505 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.505 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.505 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:53:34.505 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.706120+0000", 2024-02-05T04:53:34.505 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.6801999999999999e-05 2024-02-05T04:53:34.505 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.505 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.505 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:34.505 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.706138+0000", 2024-02-05T04:53:34.505 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.7379000000000001e-05 2024-02-05T04:53:34.505 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.506 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:34.506 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.506 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.506 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.506 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4209.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-02-05T04:53:34.506 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:09.696526+0000", 2024-02-05T04:53:34.506 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 24.790007248999999, 2024-02-05T04:53:34.506 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0096876010000000005, 2024-02-05T04:53:34.506 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:34.506 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:34.506 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:53:34.506 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4209", 2024-02-05T04:53:34.506 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3672061555", 2024-02-05T04:53:34.506 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 87 2024-02-05T04:53:34.506 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.507 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:34.507 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.507 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:34.507 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.696526+0000", 2024-02-05T04:53:34.507 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.507 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.507 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.507 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:34.507 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.696526+0000", 2024-02-05T04:53:34.507 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.507 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.507 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.507 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:34.507 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.696526+0000", 2024-02-05T04:53:34.507 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.2899999999999999e-07 2024-02-05T04:53:34.507 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.508 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.508 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:34.508 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.696602+0000", 2024-02-05T04:53:34.508 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.5636999999999994e-05 2024-02-05T04:53:34.508 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.508 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.508 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:34.508 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.696607+0000", 2024-02-05T04:53:34.508 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.463e-06 2024-02-05T04:53:34.508 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.508 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.508 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:34.508 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.696611+0000", 2024-02-05T04:53:34.508 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.4199999999999999e-06 2024-02-05T04:53:34.508 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.509 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.509 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:34.509 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.698261+0000", 2024-02-05T04:53:34.509 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0016498039999999999 2024-02-05T04:53:34.509 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.509 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.509 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:53:34.509 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.698300+0000", 2024-02-05T04:53:34.509 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.9468000000000002e-05 2024-02-05T04:53:34.509 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.509 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.509 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for subops from 2", 2024-02-05T04:53:34.509 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.698363+0000", 2024-02-05T04:53:34.509 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.3360000000000003e-05 2024-02-05T04:53:34.509 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.509 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.510 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "op_commit", 2024-02-05T04:53:34.510 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.702658+0000", 2024-02-05T04:53:34.510 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0042946929999999996 2024-02-05T04:53:34.510 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.510 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.510 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_commit_rec", 2024-02-05T04:53:34.510 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.706170+0000", 2024-02-05T04:53:34.510 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0035117070000000002 2024-02-05T04:53:34.510 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.510 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.510 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:53:34.510 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.706196+0000", 2024-02-05T04:53:34.510 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.6129e-05 2024-02-05T04:53:34.510 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.510 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.511 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:34.511 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.706213+0000", 2024-02-05T04:53:34.511 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.7490999999999999e-05 2024-02-05T04:53:34.511 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.511 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:34.511 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.511 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.511 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.511 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4209.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-02-05T04:53:34.511 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:09.696618+0000", 2024-02-05T04:53:34.511 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 24.789914953, 2024-02-05T04:53:34.511 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0096608100000000006, 2024-02-05T04:53:34.511 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:34.511 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:34.511 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:53:34.511 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4209", 2024-02-05T04:53:34.512 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3672061555", 2024-02-05T04:53:34.512 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 88 2024-02-05T04:53:34.512 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.512 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:34.512 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.512 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:34.512 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.696618+0000", 2024-02-05T04:53:34.512 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.512 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.512 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.512 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:34.512 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.696618+0000", 2024-02-05T04:53:34.512 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.512 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.512 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.513 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:34.513 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.696618+0000", 2024-02-05T04:53:34.513 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.7099999999999998e-07 2024-02-05T04:53:34.513 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.513 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.513 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:34.513 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.696621+0000", 2024-02-05T04:53:34.513 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.401e-06 2024-02-05T04:53:34.513 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.513 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.513 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:34.513 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.696624+0000", 2024-02-05T04:53:34.513 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.4680000000000001e-06 2024-02-05T04:53:34.513 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.513 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.514 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:34.514 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.696627+0000", 2024-02-05T04:53:34.514 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.525e-06 2024-02-05T04:53:34.514 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.514 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.514 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:34.514 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.698695+0000", 2024-02-05T04:53:34.514 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0020686290000000002 2024-02-05T04:53:34.514 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.514 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.514 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:53:34.514 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.698736+0000", 2024-02-05T04:53:34.514 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.0555e-05 2024-02-05T04:53:34.514 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.514 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.515 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for subops from 2", 2024-02-05T04:53:34.515 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.698800+0000", 2024-02-05T04:53:34.515 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.4437000000000007e-05 2024-02-05T04:53:34.515 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.515 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.515 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "op_commit", 2024-02-05T04:53:34.515 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.702661+0000", 2024-02-05T04:53:34.515 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0038606640000000002 2024-02-05T04:53:34.515 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.515 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.515 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_commit_rec", 2024-02-05T04:53:34.515 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.706238+0000", 2024-02-05T04:53:34.515 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0035770250000000002 2024-02-05T04:53:34.515 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.515 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.516 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:53:34.516 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.706263+0000", 2024-02-05T04:53:34.516 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.4797999999999999e-05 2024-02-05T04:53:34.516 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.516 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.516 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:34.516 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.706279+0000", 2024-02-05T04:53:34.516 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.6036999999999999e-05 2024-02-05T04:53:34.516 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.516 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:34.516 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.516 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.516 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.516 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4209.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-02-05T04:53:34.516 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:09.850779+0000", 2024-02-05T04:53:34.517 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 24.635754037000002, 2024-02-05T04:53:34.517 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0095974279999999999, 2024-02-05T04:53:34.517 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:34.517 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:34.517 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:53:34.517 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4209", 2024-02-05T04:53:34.517 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3672061555", 2024-02-05T04:53:34.517 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 110 2024-02-05T04:53:34.517 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.517 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:34.517 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.517 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:34.517 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.850779+0000", 2024-02-05T04:53:34.517 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.517 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.517 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.518 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:34.518 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.850779+0000", 2024-02-05T04:53:34.518 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.518 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.518 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.518 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:34.518 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.850780+0000", 2024-02-05T04:53:34.518 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.5199999999999997e-07 2024-02-05T04:53:34.518 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.518 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.518 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:34.518 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.850784+0000", 2024-02-05T04:53:34.518 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.0099999999999997e-06 2024-02-05T04:53:34.518 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.518 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.519 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:34.519 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.850791+0000", 2024-02-05T04:53:34.519 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.2259999999999999e-06 2024-02-05T04:53:34.519 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.519 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.519 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:34.519 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.850796+0000", 2024-02-05T04:53:34.519 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.2089999999999996e-06 2024-02-05T04:53:34.519 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.519 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.519 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:34.519 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.853636+0000", 2024-02-05T04:53:34.519 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0028403199999999999 2024-02-05T04:53:34.519 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.519 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.519 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:53:34.520 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.853688+0000", 2024-02-05T04:53:34.520 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.1391000000000003e-05 2024-02-05T04:53:34.520 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.520 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.520 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for subops from 2", 2024-02-05T04:53:34.520 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.853766+0000", 2024-02-05T04:53:34.520 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.7875999999999998e-05 2024-02-05T04:53:34.520 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.520 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.520 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "op_commit", 2024-02-05T04:53:34.520 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.855430+0000", 2024-02-05T04:53:34.520 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.001664603 2024-02-05T04:53:34.520 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.520 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.520 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_commit_rec", 2024-02-05T04:53:34.521 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.860285+0000", 2024-02-05T04:53:34.521 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0048545230000000003 2024-02-05T04:53:34.521 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.521 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.521 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:53:34.521 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.860340+0000", 2024-02-05T04:53:34.521 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.5531000000000001e-05 2024-02-05T04:53:34.521 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.521 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.521 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:34.521 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.860376+0000", 2024-02-05T04:53:34.521 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.6186999999999998e-05 2024-02-05T04:53:34.521 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.521 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:34.521 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.521 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.522 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.522 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4209.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-02-05T04:53:34.522 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:09.850805+0000", 2024-02-05T04:53:34.522 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 24.635728227000001, 2024-02-05T04:53:34.522 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0096654600000000007, 2024-02-05T04:53:34.522 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:34.522 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:34.522 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:53:34.522 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4209", 2024-02-05T04:53:34.522 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3672061555", 2024-02-05T04:53:34.522 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 111 2024-02-05T04:53:34.522 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.522 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:34.522 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.522 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:34.523 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.850805+0000", 2024-02-05T04:53:34.523 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.523 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.523 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.523 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:34.523 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.850805+0000", 2024-02-05T04:53:34.523 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.523 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.523 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.523 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:34.523 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.850805+0000", 2024-02-05T04:53:34.523 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.3099999999999998e-07 2024-02-05T04:53:34.523 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.523 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.523 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:34.523 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.850809+0000", 2024-02-05T04:53:34.524 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.6370000000000001e-06 2024-02-05T04:53:34.524 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.524 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.524 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:34.524 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.850817+0000", 2024-02-05T04:53:34.524 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.3909999999999998e-06 2024-02-05T04:53:34.524 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.524 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.524 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:34.524 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.850831+0000", 2024-02-05T04:53:34.524 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.3942e-05 2024-02-05T04:53:34.524 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.524 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.524 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:34.524 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.854167+0000", 2024-02-05T04:53:34.524 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0033358530000000002 2024-02-05T04:53:34.525 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.525 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.525 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:53:34.525 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.854212+0000", 2024-02-05T04:53:34.525 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.5393999999999999e-05 2024-02-05T04:53:34.525 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.525 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.525 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for subops from 2", 2024-02-05T04:53:34.525 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.854290+0000", 2024-02-05T04:53:34.525 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.7189999999999998e-05 2024-02-05T04:53:34.525 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.525 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.525 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "op_commit", 2024-02-05T04:53:34.525 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.857996+0000", 2024-02-05T04:53:34.525 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0037059889999999998 2024-02-05T04:53:34.525 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.525 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.526 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_commit_rec", 2024-02-05T04:53:34.526 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.860419+0000", 2024-02-05T04:53:34.526 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0024236309999999999 2024-02-05T04:53:34.526 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.526 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.526 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:53:34.526 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.860451+0000", 2024-02-05T04:53:34.526 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.1433000000000002e-05 2024-02-05T04:53:34.526 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.526 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.526 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:34.526 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.860470+0000", 2024-02-05T04:53:34.526 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.9568999999999998e-05 2024-02-05T04:53:34.526 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.527 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:34.527 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.527 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.527 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.527 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4209.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-02-05T04:53:34.527 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:09.850841+0000", 2024-02-05T04:53:34.527 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 24.635692110000001, 2024-02-05T04:53:34.527 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0096977799999999996, 2024-02-05T04:53:34.527 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:34.527 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:34.527 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:53:34.527 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4209", 2024-02-05T04:53:34.527 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3672061555", 2024-02-05T04:53:34.527 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 112 2024-02-05T04:53:34.527 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.528 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:34.528 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.528 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:34.528 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.850841+0000", 2024-02-05T04:53:34.528 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.528 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.528 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.528 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:34.528 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.850841+0000", 2024-02-05T04:53:34.528 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.528 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.528 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.528 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:34.528 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.850841+0000", 2024-02-05T04:53:34.528 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.3000000000000001e-07 2024-02-05T04:53:34.529 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.529 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.529 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:34.529 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.850961+0000", 2024-02-05T04:53:34.529 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00011995 2024-02-05T04:53:34.529 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.529 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.529 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:34.529 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.850981+0000", 2024-02-05T04:53:34.529 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.9323000000000001e-05 2024-02-05T04:53:34.529 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.529 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.529 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:34.529 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.850987+0000", 2024-02-05T04:53:34.530 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.5729999999999996e-06 2024-02-05T04:53:34.530 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.530 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.530 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:34.530 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.854544+0000", 2024-02-05T04:53:34.530 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0035569809999999999 2024-02-05T04:53:34.530 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.530 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.530 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:53:34.530 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.854626+0000", 2024-02-05T04:53:34.530 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.1725999999999996e-05 2024-02-05T04:53:34.530 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.530 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.530 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for subops from 2", 2024-02-05T04:53:34.530 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.854754+0000", 2024-02-05T04:53:34.530 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00012792899999999999 2024-02-05T04:53:34.531 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.531 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.531 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "op_commit", 2024-02-05T04:53:34.531 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.857998+0000", 2024-02-05T04:53:34.531 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0032445070000000002 2024-02-05T04:53:34.531 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.531 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.531 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_commit_rec", 2024-02-05T04:53:34.531 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.860493+0000", 2024-02-05T04:53:34.531 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0024945700000000002 2024-02-05T04:53:34.531 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.531 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.531 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:53:34.531 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.860520+0000", 2024-02-05T04:53:34.531 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.7141999999999999e-05 2024-02-05T04:53:34.532 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.532 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.532 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:34.532 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.860539+0000", 2024-02-05T04:53:34.532 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.8649000000000001e-05 2024-02-05T04:53:34.532 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.532 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:34.532 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.532 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.532 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.532 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4209.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-02-05T04:53:34.532 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:09.850999+0000", 2024-02-05T04:53:34.532 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 24.635533862999999, 2024-02-05T04:53:34.532 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0095989739999999997, 2024-02-05T04:53:34.532 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:34.533 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:34.533 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:53:34.533 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4209", 2024-02-05T04:53:34.533 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3672061555", 2024-02-05T04:53:34.533 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 113 2024-02-05T04:53:34.533 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.533 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:34.533 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.533 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:34.533 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.850999+0000", 2024-02-05T04:53:34.533 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.533 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.533 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.533 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:34.533 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.850999+0000", 2024-02-05T04:53:34.533 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.534 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.534 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.534 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:34.534 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851000+0000", 2024-02-05T04:53:34.534 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.0900000000000002e-07 2024-02-05T04:53:34.534 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.534 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.534 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:34.534 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851004+0000", 2024-02-05T04:53:34.534 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.4669999999999998e-06 2024-02-05T04:53:34.534 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.534 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.534 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:34.534 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851014+0000", 2024-02-05T04:53:34.534 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.7540000000000001e-06 2024-02-05T04:53:34.534 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.535 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.535 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:34.535 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851019+0000", 2024-02-05T04:53:34.535 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.2610000000000001e-06 2024-02-05T04:53:34.535 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.535 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.535 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:34.535 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.855104+0000", 2024-02-05T04:53:34.535 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0040849609999999998 2024-02-05T04:53:34.535 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.535 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.535 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:53:34.535 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.855150+0000", 2024-02-05T04:53:34.535 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.6143999999999997e-05 2024-02-05T04:53:34.535 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.535 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.535 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for subops from 2", 2024-02-05T04:53:34.536 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.855224+0000", 2024-02-05T04:53:34.536 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.4024999999999997e-05 2024-02-05T04:53:34.536 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.536 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.536 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "op_commit", 2024-02-05T04:53:34.536 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.858002+0000", 2024-02-05T04:53:34.536 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.002777336 2024-02-05T04:53:34.536 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.536 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.536 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_commit_rec", 2024-02-05T04:53:34.536 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.860559+0000", 2024-02-05T04:53:34.536 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0025570910000000001 2024-02-05T04:53:34.536 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.536 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.537 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:53:34.537 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.860583+0000", 2024-02-05T04:53:34.537 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.4562999999999998e-05 2024-02-05T04:53:34.537 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.537 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.537 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:34.537 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.860598+0000", 2024-02-05T04:53:34.537 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.4863e-05 2024-02-05T04:53:34.537 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.537 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:34.537 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.537 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.537 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.537 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4209.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-02-05T04:53:34.537 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:09.851028+0000", 2024-02-05T04:53:34.537 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 24.63550468, 2024-02-05T04:53:34.538 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0097085690000000002, 2024-02-05T04:53:34.538 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:34.538 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:34.538 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:53:34.538 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4209", 2024-02-05T04:53:34.538 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3672061555", 2024-02-05T04:53:34.538 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 114 2024-02-05T04:53:34.538 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.538 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:34.538 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.538 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:34.538 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851028+0000", 2024-02-05T04:53:34.538 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.538 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.538 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.538 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:34.539 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851028+0000", 2024-02-05T04:53:34.539 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.539 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.539 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.539 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:34.539 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851029+0000", 2024-02-05T04:53:34.539 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.9599999999999999e-07 2024-02-05T04:53:34.539 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.539 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.539 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:34.539 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851033+0000", 2024-02-05T04:53:34.539 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.0169999999999998e-06 2024-02-05T04:53:34.539 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.539 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.539 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:34.540 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851044+0000", 2024-02-05T04:53:34.540 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1388e-05 2024-02-05T04:53:34.540 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.540 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.540 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:34.540 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851050+0000", 2024-02-05T04:53:34.540 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.7250000000000002e-06 2024-02-05T04:53:34.540 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.540 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.540 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:34.540 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.855468+0000", 2024-02-05T04:53:34.540 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0044182170000000003 2024-02-05T04:53:34.540 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.540 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.540 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:53:34.541 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.855549+0000", 2024-02-05T04:53:34.541 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.1248999999999994e-05 2024-02-05T04:53:34.541 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.541 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.541 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for subops from 2", 2024-02-05T04:53:34.541 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.855675+0000", 2024-02-05T04:53:34.541 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000126029 2024-02-05T04:53:34.541 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.541 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.541 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "op_commit", 2024-02-05T04:53:34.541 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.858005+0000", 2024-02-05T04:53:34.541 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0023293329999999998 2024-02-05T04:53:34.541 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.541 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.541 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_commit_rec", 2024-02-05T04:53:34.542 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.860645+0000", 2024-02-05T04:53:34.542 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0026406960000000001 2024-02-05T04:53:34.542 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.542 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.542 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:53:34.542 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.860701+0000", 2024-02-05T04:53:34.542 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.5492999999999999e-05 2024-02-05T04:53:34.542 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.542 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.542 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:34.542 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.860737+0000", 2024-02-05T04:53:34.542 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.5926000000000001e-05 2024-02-05T04:53:34.542 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.542 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:34.542 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.542 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.543 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.543 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4209.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-02-05T04:53:34.543 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:09.851060+0000", 2024-02-05T04:53:34.543 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 24.635472880000002, 2024-02-05T04:53:34.543 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.011939682, 2024-02-05T04:53:34.543 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:34.543 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:34.543 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:53:34.543 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4209", 2024-02-05T04:53:34.543 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3672061555", 2024-02-05T04:53:34.543 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 115 2024-02-05T04:53:34.543 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.543 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:34.543 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.543 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:34.543 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851060+0000", 2024-02-05T04:53:34.544 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.544 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.544 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.544 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:34.544 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851060+0000", 2024-02-05T04:53:34.544 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.544 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.544 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.544 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:34.544 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851061+0000", 2024-02-05T04:53:34.544 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.5900000000000002e-07 2024-02-05T04:53:34.544 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.544 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.544 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:34.544 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851142+0000", 2024-02-05T04:53:34.544 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.1631000000000005e-05 2024-02-05T04:53:34.545 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.545 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.545 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:34.545 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851150+0000", 2024-02-05T04:53:34.545 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.2679999999999994e-06 2024-02-05T04:53:34.545 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.545 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.545 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:34.545 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851156+0000", 2024-02-05T04:53:34.545 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.7200000000000003e-06 2024-02-05T04:53:34.545 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.545 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.545 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:34.545 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.856546+0000", 2024-02-05T04:53:34.545 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0053899839999999996 2024-02-05T04:53:34.545 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.545 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.546 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:53:34.546 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.856623+0000", 2024-02-05T04:53:34.546 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.6921000000000004e-05 2024-02-05T04:53:34.546 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.546 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.546 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for subops from 2", 2024-02-05T04:53:34.546 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.856748+0000", 2024-02-05T04:53:34.546 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00012522999999999999 2024-02-05T04:53:34.546 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.546 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.546 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "op_commit", 2024-02-05T04:53:34.546 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.858007+0000", 2024-02-05T04:53:34.546 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0012587779999999999 2024-02-05T04:53:34.546 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.546 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.546 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_commit_rec", 2024-02-05T04:53:34.547 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.862907+0000", 2024-02-05T04:53:34.547 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0048998729999999999 2024-02-05T04:53:34.547 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.547 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.547 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:53:34.547 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.862964+0000", 2024-02-05T04:53:34.547 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.7391e-05 2024-02-05T04:53:34.547 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.547 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.547 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:34.547 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.863000+0000", 2024-02-05T04:53:34.547 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.5426999999999998e-05 2024-02-05T04:53:34.547 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.547 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:34.547 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.547 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.548 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.548 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4209.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-02-05T04:53:34.548 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:09.851167+0000", 2024-02-05T04:53:34.548 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 24.635366036000001, 2024-02-05T04:53:34.548 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.011918911000000001, 2024-02-05T04:53:34.548 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:34.548 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:34.548 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:53:34.548 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4209", 2024-02-05T04:53:34.548 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3672061555", 2024-02-05T04:53:34.548 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 116 2024-02-05T04:53:34.548 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.548 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:34.548 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.549 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:34.549 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851167+0000", 2024-02-05T04:53:34.549 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.549 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.549 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.549 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:34.549 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851167+0000", 2024-02-05T04:53:34.549 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.549 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.549 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.549 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:34.549 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851167+0000", 2024-02-05T04:53:34.549 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.9999999999999998e-07 2024-02-05T04:53:34.549 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.549 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.549 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:34.550 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851172+0000", 2024-02-05T04:53:34.550 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.0400000000000003e-06 2024-02-05T04:53:34.550 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.550 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.550 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:34.550 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851179+0000", 2024-02-05T04:53:34.550 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.063e-06 2024-02-05T04:53:34.550 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.550 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.550 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:34.550 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851184+0000", 2024-02-05T04:53:34.550 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.1220000000000003e-06 2024-02-05T04:53:34.550 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.550 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.550 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:34.550 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.857205+0000", 2024-02-05T04:53:34.551 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0060208630000000004 2024-02-05T04:53:34.551 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.551 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.551 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:53:34.551 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.857278+0000", 2024-02-05T04:53:34.551 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.3089000000000004e-05 2024-02-05T04:53:34.551 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.551 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.551 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for subops from 2", 2024-02-05T04:53:34.551 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.857412+0000", 2024-02-05T04:53:34.551 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00013471800000000001 2024-02-05T04:53:34.551 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.551 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.551 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "op_commit", 2024-02-05T04:53:34.551 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.858009+0000", 2024-02-05T04:53:34.552 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00059697400000000001 2024-02-05T04:53:34.552 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.552 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.552 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_commit_rec", 2024-02-05T04:53:34.552 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.863035+0000", 2024-02-05T04:53:34.552 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0050251489999999996 2024-02-05T04:53:34.552 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.552 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.552 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:53:34.552 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.863066+0000", 2024-02-05T04:53:34.552 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.1680999999999997e-05 2024-02-05T04:53:34.552 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.552 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.552 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:34.552 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.863086+0000", 2024-02-05T04:53:34.553 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.9712e-05 2024-02-05T04:53:34.553 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.553 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:34.553 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.553 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.553 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.553 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4209.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-02-05T04:53:34.553 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:09.851192+0000", 2024-02-05T04:53:34.553 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 24.635341034, 2024-02-05T04:53:34.553 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.014624346999999999, 2024-02-05T04:53:34.553 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:34.553 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:34.553 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:53:34.553 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4209", 2024-02-05T04:53:34.553 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3672061555", 2024-02-05T04:53:34.553 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 117 2024-02-05T04:53:34.553 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.554 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:34.554 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.554 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:34.554 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851192+0000", 2024-02-05T04:53:34.554 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.554 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.554 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.554 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:34.554 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851192+0000", 2024-02-05T04:53:34.554 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.554 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.554 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.554 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:34.554 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851192+0000", 2024-02-05T04:53:34.554 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.5900000000000003e-07 2024-02-05T04:53:34.554 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.555 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.555 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:34.555 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851200+0000", 2024-02-05T04:53:34.555 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.2420000000000003e-06 2024-02-05T04:53:34.555 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.555 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.555 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:34.555 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851206+0000", 2024-02-05T04:53:34.555 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.2419999999999997e-06 2024-02-05T04:53:34.555 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.555 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.555 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:34.555 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851211+0000", 2024-02-05T04:53:34.555 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.7339999999999999e-06 2024-02-05T04:53:34.555 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.555 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.555 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:34.556 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.857705+0000", 2024-02-05T04:53:34.556 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0064942890000000003 2024-02-05T04:53:34.556 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.556 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.556 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:53:34.556 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.857756+0000", 2024-02-05T04:53:34.556 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.1360999999999998e-05 2024-02-05T04:53:34.556 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.556 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.556 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for subops from 2", 2024-02-05T04:53:34.556 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.857841+0000", 2024-02-05T04:53:34.556 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.4610000000000005e-05 2024-02-05T04:53:34.556 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.556 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.556 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "op_commit", 2024-02-05T04:53:34.557 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.858257+0000", 2024-02-05T04:53:34.557 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00041574599999999999 2024-02-05T04:53:34.557 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.557 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.557 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_commit_rec", 2024-02-05T04:53:34.557 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.865722+0000", 2024-02-05T04:53:34.557 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.007465902 2024-02-05T04:53:34.557 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.557 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.557 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:53:34.557 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.865782+0000", 2024-02-05T04:53:34.557 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.9879e-05 2024-02-05T04:53:34.557 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.557 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.557 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:34.558 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.865816+0000", 2024-02-05T04:53:34.558 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.3983000000000003e-05 2024-02-05T04:53:34.558 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.558 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:34.558 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.558 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.558 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.558 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4209.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-02-05T04:53:34.558 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:09.993364+0000", 2024-02-05T04:53:34.558 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 24.493168932, 2024-02-05T04:53:34.558 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0097019250000000001, 2024-02-05T04:53:34.558 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:34.558 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:34.558 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:53:34.558 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4209", 2024-02-05T04:53:34.558 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3672061555", 2024-02-05T04:53:34.559 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 138 2024-02-05T04:53:34.559 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.559 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:34.559 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.559 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:34.559 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.993364+0000", 2024-02-05T04:53:34.559 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.559 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.559 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.559 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:34.559 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.993364+0000", 2024-02-05T04:53:34.559 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.559 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.559 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.559 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:34.559 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.993364+0000", 2024-02-05T04:53:34.560 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.8700000000000002e-07 2024-02-05T04:53:34.560 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.560 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.560 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:34.560 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.993368+0000", 2024-02-05T04:53:34.560 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.6720000000000002e-06 2024-02-05T04:53:34.560 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.560 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.560 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:34.560 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.993372+0000", 2024-02-05T04:53:34.560 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.625e-06 2024-02-05T04:53:34.560 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.560 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.560 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:34.560 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.993379+0000", 2024-02-05T04:53:34.560 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.1210000000000001e-06 2024-02-05T04:53:34.561 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.561 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.561 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:34.561 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.000600+0000", 2024-02-05T04:53:34.561 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0072211829999999999 2024-02-05T04:53:34.561 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.561 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.561 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:53:34.561 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.000672+0000", 2024-02-05T04:53:34.561 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.1967999999999999e-05 2024-02-05T04:53:34.561 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.561 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.561 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for subops from 2", 2024-02-05T04:53:34.561 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.000765+0000", 2024-02-05T04:53:34.561 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.3139000000000004e-05 2024-02-05T04:53:34.562 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.562 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.562 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "op_commit", 2024-02-05T04:53:34.562 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.002487+0000", 2024-02-05T04:53:34.562 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.001721783 2024-02-05T04:53:34.562 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.562 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.562 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_commit_rec", 2024-02-05T04:53:34.562 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.003017+0000", 2024-02-05T04:53:34.562 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00053045200000000005 2024-02-05T04:53:34.562 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.562 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.562 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:53:34.562 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.003047+0000", 2024-02-05T04:53:34.562 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.9229000000000001e-05 2024-02-05T04:53:34.563 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.563 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.563 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:34.563 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.003066+0000", 2024-02-05T04:53:34.563 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.9466e-05 2024-02-05T04:53:34.563 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.563 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:34.563 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.563 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.563 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.563 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4209.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-02-05T04:53:34.563 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:09.993384+0000", 2024-02-05T04:53:34.563 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 24.493149170999999, 2024-02-05T04:53:34.563 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.013170685999999999, 2024-02-05T04:53:34.563 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:34.563 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:34.564 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:53:34.564 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4209", 2024-02-05T04:53:34.564 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3672061555", 2024-02-05T04:53:34.564 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 139 2024-02-05T04:53:34.564 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.564 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:34.564 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.564 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:34.564 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.993384+0000", 2024-02-05T04:53:34.564 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.564 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.564 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.564 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:34.564 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.993384+0000", 2024-02-05T04:53:34.564 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.565 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.565 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.565 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:34.565 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.993384+0000", 2024-02-05T04:53:34.565 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.1400000000000001e-07 2024-02-05T04:53:34.565 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.565 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.565 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:34.565 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.993386+0000", 2024-02-05T04:53:34.565 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.9829999999999999e-06 2024-02-05T04:53:34.565 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.565 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.565 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:34.565 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.993390+0000", 2024-02-05T04:53:34.565 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.4209999999999997e-06 2024-02-05T04:53:34.565 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.566 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.566 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:34.566 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.993393+0000", 2024-02-05T04:53:34.566 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.508e-06 2024-02-05T04:53:34.566 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.566 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.566 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:34.566 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.001088+0000", 2024-02-05T04:53:34.566 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0076952230000000002 2024-02-05T04:53:34.566 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.566 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.566 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:53:34.566 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.001134+0000", 2024-02-05T04:53:34.566 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.5467999999999999e-05 2024-02-05T04:53:34.566 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.566 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.567 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for subops from 2", 2024-02-05T04:53:34.567 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.001203+0000", 2024-02-05T04:53:34.567 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.9509000000000001e-05 2024-02-05T04:53:34.567 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.567 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.567 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "op_commit", 2024-02-05T04:53:34.567 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.002497+0000", 2024-02-05T04:53:34.567 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0012934000000000001 2024-02-05T04:53:34.567 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.567 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.567 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_commit_rec", 2024-02-05T04:53:34.567 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.006502+0000", 2024-02-05T04:53:34.567 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0040053980000000003 2024-02-05T04:53:34.567 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.567 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.567 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:53:34.568 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.006534+0000", 2024-02-05T04:53:34.568 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.2351999999999998e-05 2024-02-05T04:53:34.568 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.568 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.568 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:34.568 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.006555+0000", 2024-02-05T04:53:34.568 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.0210000000000001e-05 2024-02-05T04:53:34.568 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.568 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:34.568 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.568 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.568 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.568 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4209.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-02-05T04:53:34.568 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:09.993397+0000", 2024-02-05T04:53:34.568 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 24.493135924000001, 2024-02-05T04:53:34.569 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.013240236000000001, 2024-02-05T04:53:34.569 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:34.569 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:34.569 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:53:34.569 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4209", 2024-02-05T04:53:34.569 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3672061555", 2024-02-05T04:53:34.569 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 140 2024-02-05T04:53:34.569 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.569 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:34.569 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.569 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:34.569 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.993397+0000", 2024-02-05T04:53:34.569 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.569 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.569 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.569 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:34.570 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.993397+0000", 2024-02-05T04:53:34.570 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.570 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.570 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.570 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:34.570 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.993397+0000", 2024-02-05T04:53:34.570 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.2700000000000001e-07 2024-02-05T04:53:34.570 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.570 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.570 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:34.570 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.993449+0000", 2024-02-05T04:53:34.570 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.1613000000000002e-05 2024-02-05T04:53:34.570 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.570 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.570 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:34.570 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.993453+0000", 2024-02-05T04:53:34.571 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.245e-06 2024-02-05T04:53:34.571 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.571 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.571 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:34.571 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.993456+0000", 2024-02-05T04:53:34.571 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.942e-06 2024-02-05T04:53:34.571 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.571 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.571 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:34.571 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.001438+0000", 2024-02-05T04:53:34.571 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0079815589999999992 2024-02-05T04:53:34.571 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.571 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.571 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:53:34.571 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.001514+0000", 2024-02-05T04:53:34.572 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.6038999999999995e-05 2024-02-05T04:53:34.572 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.572 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.572 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for subops from 2", 2024-02-05T04:53:34.572 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.001642+0000", 2024-02-05T04:53:34.572 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000128713 2024-02-05T04:53:34.572 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.572 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.572 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "op_commit", 2024-02-05T04:53:34.572 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.002506+0000", 2024-02-05T04:53:34.572 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00086372900000000004 2024-02-05T04:53:34.572 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.572 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.572 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_commit_rec", 2024-02-05T04:53:34.572 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.006589+0000", 2024-02-05T04:53:34.573 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0040831690000000002 2024-02-05T04:53:34.573 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.573 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.573 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:53:34.573 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.006620+0000", 2024-02-05T04:53:34.573 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.0817e-05 2024-02-05T04:53:34.573 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.573 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.573 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:34.573 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.006637+0000", 2024-02-05T04:53:34.573 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.7183000000000001e-05 2024-02-05T04:53:34.573 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.573 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:34.573 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.573 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.573 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.574 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4209.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-02-05T04:53:34.574 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:09.993462+0000", 2024-02-05T04:53:34.574 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 24.493071135000001, 2024-02-05T04:53:34.574 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.013244182, 2024-02-05T04:53:34.574 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:34.574 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:34.574 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:53:34.574 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4209", 2024-02-05T04:53:34.574 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3672061555", 2024-02-05T04:53:34.574 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 141 2024-02-05T04:53:34.574 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.574 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:34.574 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.574 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:34.574 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.993462+0000", 2024-02-05T04:53:34.575 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.575 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.575 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.575 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:34.575 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.993462+0000", 2024-02-05T04:53:34.575 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.575 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.575 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.575 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:34.575 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.993462+0000", 2024-02-05T04:53:34.575 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.6100000000000002e-07 2024-02-05T04:53:34.575 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.575 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.575 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:34.575 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.993464+0000", 2024-02-05T04:53:34.575 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.0480000000000001e-06 2024-02-05T04:53:34.576 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.576 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.576 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:34.576 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.993467+0000", 2024-02-05T04:53:34.576 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.1480000000000002e-06 2024-02-05T04:53:34.576 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.576 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.576 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:34.576 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.993470+0000", 2024-02-05T04:53:34.576 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.5320000000000001e-06 2024-02-05T04:53:34.576 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.576 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.576 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:34.576 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.002184+0000", 2024-02-05T04:53:34.576 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0087141649999999994 2024-02-05T04:53:34.577 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.577 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.577 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:53:34.577 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.002227+0000", 2024-02-05T04:53:34.577 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.3272000000000002e-05 2024-02-05T04:53:34.577 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.577 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.577 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for subops from 2", 2024-02-05T04:53:34.577 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.002296+0000", 2024-02-05T04:53:34.577 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.8268000000000004e-05 2024-02-05T04:53:34.577 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.577 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.577 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "op_commit", 2024-02-05T04:53:34.577 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.002752+0000", 2024-02-05T04:53:34.577 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00045654199999999999 2024-02-05T04:53:34.578 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.578 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.578 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_commit_rec", 2024-02-05T04:53:34.578 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.006663+0000", 2024-02-05T04:53:34.578 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.003911126 2024-02-05T04:53:34.578 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.578 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.578 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:53:34.578 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.006689+0000", 2024-02-05T04:53:34.578 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.6061999999999999e-05 2024-02-05T04:53:34.578 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.578 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.578 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:34.578 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.006706+0000", 2024-02-05T04:53:34.578 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.6758000000000001e-05 2024-02-05T04:53:34.578 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.579 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:34.579 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.579 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.579 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.579 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.es2 38/31 ECSubWrite(tid=25, reqid=osd.2.0:25, at_version=38'20, trim_to=0'0, roll_forward_to=37'16))", 2024-02-05T04:53:34.579 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:30.134965+0000", 2024-02-05T04:53:34.579 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 4.3515676619999999, 2024-02-05T04:53:34.579 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.010779733, 2024-02-05T04:53:34.579 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:34.579 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:34.579 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:34.579 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.579 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:34.579 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.134965+0000", 2024-02-05T04:53:34.579 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.580 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.580 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.580 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:34.580 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.134965+0000", 2024-02-05T04:53:34.580 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.580 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.580 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.580 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:34.580 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.134966+0000", 2024-02-05T04:53:34.580 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.4399999999999999e-07 2024-02-05T04:53:34.580 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.580 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.580 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:34.580 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.134990+0000", 2024-02-05T04:53:34.580 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.3895000000000001e-05 2024-02-05T04:53:34.581 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.581 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.581 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:34.581 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.135002+0000", 2024-02-05T04:53:34.581 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1868000000000001e-05 2024-02-05T04:53:34.581 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.581 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.581 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:34.581 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.135011+0000", 2024-02-05T04:53:34.581 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.8920000000000006e-06 2024-02-05T04:53:34.581 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.581 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.581 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:34.581 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.135354+0000", 2024-02-05T04:53:34.581 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00034297400000000001 2024-02-05T04:53:34.581 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.582 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.582 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:34.582 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.135364+0000", 2024-02-05T04:53:34.582 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0205e-05 2024-02-05T04:53:34.582 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.582 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.582 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:34.582 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.145707+0000", 2024-02-05T04:53:34.582 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.010343250999999999 2024-02-05T04:53:34.582 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.582 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.582 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:34.582 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.145745+0000", 2024-02-05T04:53:34.582 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.7904000000000002e-05 2024-02-05T04:53:34.582 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.582 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:34.583 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.583 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.583 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.583 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.6s6 38/31 ECSubWrite(tid=90, reqid=osd.13.0:90, at_version=38'11, trim_to=0'0, roll_forward_to=37'8))", 2024-02-05T04:53:34.583 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:30.142022+0000", 2024-02-05T04:53:34.583 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 4.3445112080000001, 2024-02-05T04:53:34.583 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0092129240000000008, 2024-02-05T04:53:34.583 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:34.583 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:34.583 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:34.583 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.583 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:34.583 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.142022+0000", 2024-02-05T04:53:34.583 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.583 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.584 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.584 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:34.584 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.142022+0000", 2024-02-05T04:53:34.584 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.584 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.584 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.584 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:34.584 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.142023+0000", 2024-02-05T04:53:34.584 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.006e-06 2024-02-05T04:53:34.584 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.584 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.584 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:34.584 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.142046+0000", 2024-02-05T04:53:34.584 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.2773999999999999e-05 2024-02-05T04:53:34.584 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.585 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.585 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:34.585 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.142062+0000", 2024-02-05T04:53:34.585 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.6195999999999999e-05 2024-02-05T04:53:34.585 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.585 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.585 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:34.585 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.142071+0000", 2024-02-05T04:53:34.585 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.3440000000000007e-06 2024-02-05T04:53:34.585 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.585 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.585 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:34.585 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.142149+0000", 2024-02-05T04:53:34.585 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.7578999999999997e-05 2024-02-05T04:53:34.585 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.585 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.586 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:34.586 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.142159+0000", 2024-02-05T04:53:34.586 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0644e-05 2024-02-05T04:53:34.586 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.586 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.586 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:34.586 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.151189+0000", 2024-02-05T04:53:34.586 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0090294450000000005 2024-02-05T04:53:34.586 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.586 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.586 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:34.586 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.151235+0000", 2024-02-05T04:53:34.586 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.5936000000000002e-05 2024-02-05T04:53:34.586 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.586 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:34.586 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.587 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.587 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.587 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.2s5 38/31 ECSubWrite(tid=28, reqid=osd.12.0:28, at_version=38'22, trim_to=0'0, roll_forward_to=37'18))", 2024-02-05T04:53:34.587 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:30.144335+0000", 2024-02-05T04:53:34.587 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 4.3421984680000003, 2024-02-05T04:53:34.587 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0094635199999999996, 2024-02-05T04:53:34.587 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:34.587 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:34.587 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:34.587 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.587 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:34.587 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.144335+0000", 2024-02-05T04:53:34.587 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.587 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.587 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.588 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:34.588 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.144335+0000", 2024-02-05T04:53:34.588 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.588 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.588 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.588 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:34.588 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.144335+0000", 2024-02-05T04:53:34.588 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.5600000000000004e-07 2024-02-05T04:53:34.588 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.588 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.588 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:34.588 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.144373+0000", 2024-02-05T04:53:34.588 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.7731000000000002e-05 2024-02-05T04:53:34.588 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.588 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.588 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:34.589 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.144387+0000", 2024-02-05T04:53:34.589 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.3788000000000001e-05 2024-02-05T04:53:34.589 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.589 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.589 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:34.589 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.144394+0000", 2024-02-05T04:53:34.589 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.2450000000000001e-06 2024-02-05T04:53:34.589 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.589 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.589 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:34.589 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.146480+0000", 2024-02-05T04:53:34.589 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0020859120000000001 2024-02-05T04:53:34.589 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.589 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.589 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:34.589 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.146504+0000", 2024-02-05T04:53:34.590 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.4091999999999999e-05 2024-02-05T04:53:34.590 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.590 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.590 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:34.590 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.153756+0000", 2024-02-05T04:53:34.590 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0072520470000000002 2024-02-05T04:53:34.590 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.590 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.590 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:34.590 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.153798+0000", 2024-02-05T04:53:34.590 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.1848999999999999e-05 2024-02-05T04:53:34.590 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.590 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:34.590 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.590 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.590 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:34.591 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-02-05T04:53:34.592 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-02-05T04:53:34.593 INFO:tasks.ceph.osd.10.smithi150.stderr:2024-02-05T04:53:34.591+0000 7f3075a4d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 34018) UID: 0 2024-02-05T04:53:34.692 INFO:tasks.ceph.osd.8.smithi062.stderr:2024-02-05T04:53:34.692+0000 7f9385bf2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 34693) UID: 0 2024-02-05T04:53:34.700 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-02-05T04:53:34.700 INFO:teuthology.orchestra.run.smithi062.stdout: "ops": [], 2024-02-05T04:53:34.700 INFO:teuthology.orchestra.run.smithi062.stdout: "num_ops": 0 2024-02-05T04:53:34.700 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-02-05T04:53:34.708 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-02-05T04:53:34.794 INFO:tasks.ceph.osd.3.smithi196.stderr:2024-02-05T04:53:34.794+0000 7fc8817c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34605) UID: 0 2024-02-05T04:53:34.855 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-02-05T04:53:34.855 INFO:teuthology.orchestra.run.smithi062.stdout: "ops": [], 2024-02-05T04:53:34.855 INFO:teuthology.orchestra.run.smithi062.stdout: "complaint_time": 30, 2024-02-05T04:53:34.855 INFO:teuthology.orchestra.run.smithi062.stdout: "num_blocked_ops": 0 2024-02-05T04:53:34.856 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-02-05T04:53:34.864 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-02-05T04:53:34.908 INFO:tasks.ceph.osd.8.smithi062.stderr:2024-02-05T04:53:34.907+0000 7f9385bf2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 34693) UID: 0 2024-02-05T04:53:34.973 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-02-05T04:53:34.973 INFO:teuthology.orchestra.run.smithi062.stdout: "size": 20, 2024-02-05T04:53:34.973 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 600, 2024-02-05T04:53:34.973 INFO:teuthology.orchestra.run.smithi062.stdout: "ops": [ 2024-02-05T04:53:34.973 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.973 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.7s2 37/31 ECSubWrite(tid=85, reqid=client.4634.0:243, at_version=0'0, trim_to=0'0, roll_forward_to=37'6))", 2024-02-05T04:53:34.973 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:29.191399+0000", 2024-02-05T04:53:34.974 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 5.7801053329999998, 2024-02-05T04:53:34.974 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0023573959999999999, 2024-02-05T04:53:34.974 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:34.974 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:34.974 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:34.974 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.974 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:34.974 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.191399+0000", 2024-02-05T04:53:34.974 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.974 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.974 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.974 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:34.974 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.191399+0000", 2024-02-05T04:53:34.974 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.974 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.975 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.975 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:34.975 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.191400+0000", 2024-02-05T04:53:34.975 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.7000000000000003e-07 2024-02-05T04:53:34.975 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.975 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.975 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:34.975 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.191412+0000", 2024-02-05T04:53:34.975 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.2344000000000001e-05 2024-02-05T04:53:34.975 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.975 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.975 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:34.975 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.191428+0000", 2024-02-05T04:53:34.975 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.5770999999999999e-05 2024-02-05T04:53:34.975 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.975 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.976 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:34.976 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.191437+0000", 2024-02-05T04:53:34.976 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.1200000000000008e-06 2024-02-05T04:53:34.976 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.976 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.976 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:34.976 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.191500+0000", 2024-02-05T04:53:34.976 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.3085999999999999e-05 2024-02-05T04:53:34.976 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.976 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.976 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:34.976 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.191511+0000", 2024-02-05T04:53:34.976 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1459e-05 2024-02-05T04:53:34.976 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.976 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.976 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:34.977 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.193709+0000", 2024-02-05T04:53:34.977 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0021973180000000002 2024-02-05T04:53:34.977 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.977 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.977 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:34.977 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.193756+0000", 2024-02-05T04:53:34.977 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.7327999999999998e-05 2024-02-05T04:53:34.977 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.977 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:34.977 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.977 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.977 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.977 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:250 3.2s0 3:47e81a58:::smithi19639362-13:head [append 0~0] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:53:34.977 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:29.198960+0000", 2024-02-05T04:53:34.977 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 5.7725434189999998, 2024-02-05T04:53:34.977 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0047770479999999999, 2024-02-05T04:53:34.978 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:34.978 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:34.978 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:53:34.978 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:53:34.978 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:34.978 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 250 2024-02-05T04:53:34.978 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.978 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:34.978 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.978 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:34.978 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.198960+0000", 2024-02-05T04:53:34.978 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.978 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.978 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.978 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:34.978 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.198960+0000", 2024-02-05T04:53:34.979 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.979 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.979 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.979 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:34.979 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.198962+0000", 2024-02-05T04:53:34.979 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.694e-06 2024-02-05T04:53:34.979 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.979 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.979 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:34.979 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.198969+0000", 2024-02-05T04:53:34.979 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.1749999999999999e-06 2024-02-05T04:53:34.979 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.979 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.979 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:34.980 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.198986+0000", 2024-02-05T04:53:34.980 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.6254999999999999e-05 2024-02-05T04:53:34.980 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.980 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.980 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:34.980 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.198996+0000", 2024-02-05T04:53:34.980 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.006e-05 2024-02-05T04:53:34.980 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.980 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.980 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:34.980 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.199067+0000", 2024-02-05T04:53:34.980 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.1493000000000001e-05 2024-02-05T04:53:34.980 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.980 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.980 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:53:34.980 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.199175+0000", 2024-02-05T04:53:34.981 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00010778299999999999 2024-02-05T04:53:34.981 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.981 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.981 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:34.981 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.200020+0000", 2024-02-05T04:53:34.981 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00084552300000000004 2024-02-05T04:53:34.981 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.981 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.981 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:34.981 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.203411+0000", 2024-02-05T04:53:34.981 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0033904059999999999 2024-02-05T04:53:34.981 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.981 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.981 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:53:34.981 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.203677+0000", 2024-02-05T04:53:34.981 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00026615900000000002 2024-02-05T04:53:34.982 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.982 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.982 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:34.982 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.203738+0000", 2024-02-05T04:53:34.982 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.05e-05 2024-02-05T04:53:34.982 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.982 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:34.982 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.982 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.982 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.982 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:251 3.2s0 3:47e81a58:::smithi19639362-13:head [rollback 3] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:53:34.982 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:29.199025+0000", 2024-02-05T04:53:34.982 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 5.7724792770000004, 2024-02-05T04:53:34.982 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.010965871, 2024-02-05T04:53:34.982 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:34.982 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:34.983 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:53:34.983 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:53:34.983 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:34.983 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 251 2024-02-05T04:53:34.983 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.983 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:34.983 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.983 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:34.983 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.199025+0000", 2024-02-05T04:53:34.983 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.983 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.983 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.983 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:34.983 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.199025+0000", 2024-02-05T04:53:34.983 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.984 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.984 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.984 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:34.984 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.199025+0000", 2024-02-05T04:53:34.984 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.6700000000000003e-07 2024-02-05T04:53:34.984 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.984 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.984 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:34.984 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.199030+0000", 2024-02-05T04:53:34.984 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.9620000000000001e-06 2024-02-05T04:53:34.984 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.984 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.984 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:34.984 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.199039+0000", 2024-02-05T04:53:34.984 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.5660000000000007e-06 2024-02-05T04:53:34.985 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.985 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.985 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:34.985 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.199045+0000", 2024-02-05T04:53:34.985 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.9229999999999999e-06 2024-02-05T04:53:34.985 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.985 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.985 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:34.985 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.200375+0000", 2024-02-05T04:53:34.985 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0013305610000000001 2024-02-05T04:53:34.985 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.985 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.985 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:53:34.985 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.200435+0000", 2024-02-05T04:53:34.985 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.9790999999999997e-05 2024-02-05T04:53:34.985 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.986 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.986 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:34.986 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.200748+0000", 2024-02-05T04:53:34.986 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00031337299999999998 2024-02-05T04:53:34.986 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.986 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.986 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:34.986 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.203751+0000", 2024-02-05T04:53:34.986 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.003003019 2024-02-05T04:53:34.986 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.986 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.986 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:53:34.986 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.209915+0000", 2024-02-05T04:53:34.986 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0061635099999999996 2024-02-05T04:53:34.986 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.986 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.987 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:34.987 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.209990+0000", 2024-02-05T04:53:34.987 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.5599000000000006e-05 2024-02-05T04:53:34.987 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.987 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:34.987 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.987 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.987 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.987 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:252 3.2s0 3:47e81a58:::smithi19639362-13:head [append 0~0] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:53:34.987 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:29.199055+0000", 2024-02-05T04:53:34.987 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 5.7724491159999998, 2024-02-05T04:53:34.987 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.011150785, 2024-02-05T04:53:34.987 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:34.987 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:34.987 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:53:34.987 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:53:34.988 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:34.988 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 252 2024-02-05T04:53:34.988 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.988 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:34.988 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.988 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:34.988 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.199055+0000", 2024-02-05T04:53:34.988 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.988 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.988 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.988 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:34.988 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.199055+0000", 2024-02-05T04:53:34.988 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.988 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.988 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.989 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:34.989 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.199055+0000", 2024-02-05T04:53:34.989 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.3500000000000002e-07 2024-02-05T04:53:34.989 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.989 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.989 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:34.989 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.199058+0000", 2024-02-05T04:53:34.989 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.779e-06 2024-02-05T04:53:34.989 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.989 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.989 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:34.989 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.199065+0000", 2024-02-05T04:53:34.989 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.5139999999999998e-06 2024-02-05T04:53:34.989 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.989 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.990 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:34.990 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.199069+0000", 2024-02-05T04:53:34.990 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.5820000000000003e-06 2024-02-05T04:53:34.990 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.990 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.990 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:34.990 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.201028+0000", 2024-02-05T04:53:34.990 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.001959199 2024-02-05T04:53:34.990 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.990 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.990 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:53:34.990 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.201092+0000", 2024-02-05T04:53:34.990 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.3375999999999998e-05 2024-02-05T04:53:34.990 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.990 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.991 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:34.991 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.201409+0000", 2024-02-05T04:53:34.991 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00031745599999999998 2024-02-05T04:53:34.991 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.991 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.991 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:34.991 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.203757+0000", 2024-02-05T04:53:34.991 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0023475219999999999 2024-02-05T04:53:34.991 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.991 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.991 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:53:34.991 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.210134+0000", 2024-02-05T04:53:34.991 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0063771130000000002 2024-02-05T04:53:34.991 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.991 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.991 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:34.992 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.210206+0000", 2024-02-05T04:53:34.992 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.1809000000000003e-05 2024-02-05T04:53:34.992 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.992 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:34.992 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.992 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.992 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.992 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:253 3.2s0 3:41121c54:::smithi19639362-23:head [append 0~688128] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:53:34.992 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:29.205018+0000", 2024-02-05T04:53:34.992 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 5.7664862320000001, 2024-02-05T04:53:34.992 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0073893079999999998, 2024-02-05T04:53:34.992 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:34.992 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:34.992 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:53:34.992 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:53:34.993 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:34.993 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 253 2024-02-05T04:53:34.993 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.993 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:34.993 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.993 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:34.993 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.205018+0000", 2024-02-05T04:53:34.993 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.993 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.993 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.993 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:34.993 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.205018+0000", 2024-02-05T04:53:34.993 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.993 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.993 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.993 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:34.994 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.205019+0000", 2024-02-05T04:53:34.994 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.282e-06 2024-02-05T04:53:34.994 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.994 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.994 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:34.994 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.205752+0000", 2024-02-05T04:53:34.994 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00073318000000000005 2024-02-05T04:53:34.994 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.994 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.994 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:34.994 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.205764+0000", 2024-02-05T04:53:34.994 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1936000000000001e-05 2024-02-05T04:53:34.994 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.994 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.994 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:34.994 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.205769+0000", 2024-02-05T04:53:34.995 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.2979999999999999e-06 2024-02-05T04:53:34.995 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.995 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.995 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:34.995 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.205811+0000", 2024-02-05T04:53:34.995 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.1891000000000003e-05 2024-02-05T04:53:34.995 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.995 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.995 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:53:34.995 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.205885+0000", 2024-02-05T04:53:34.995 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.3384000000000001e-05 2024-02-05T04:53:34.995 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.995 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.995 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:34.995 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.206604+0000", 2024-02-05T04:53:34.995 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000719343 2024-02-05T04:53:34.995 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.996 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.996 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:34.996 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.210004+0000", 2024-02-05T04:53:34.996 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0034002799999999999 2024-02-05T04:53:34.996 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.996 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.996 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:53:34.996 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.212332+0000", 2024-02-05T04:53:34.996 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0023273209999999998 2024-02-05T04:53:34.996 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.996 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.996 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:34.996 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.212407+0000", 2024-02-05T04:53:34.996 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.5393000000000001e-05 2024-02-05T04:53:34.996 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.997 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:34.997 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:34.997 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.997 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.997 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:254 3.2s0 3:41121c54:::smithi19639362-23:head [append 0~376832] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:53:34.997 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:29.207235+0000", 2024-02-05T04:53:34.997 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 5.7642691920000004, 2024-02-05T04:53:34.997 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0064199799999999996, 2024-02-05T04:53:34.997 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:34.997 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:34.997 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:53:34.997 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:53:34.997 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:34.997 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 254 2024-02-05T04:53:34.997 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.997 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:34.998 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.998 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:34.998 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.207235+0000", 2024-02-05T04:53:34.998 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.998 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.998 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.998 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:34.998 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.207235+0000", 2024-02-05T04:53:34.998 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:34.998 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.998 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.998 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:34.998 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.207236+0000", 2024-02-05T04:53:34.998 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.37e-06 2024-02-05T04:53:34.998 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.999 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.999 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:34.999 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.207708+0000", 2024-02-05T04:53:34.999 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00047221200000000001 2024-02-05T04:53:34.999 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.999 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.999 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:34.999 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.207721+0000", 2024-02-05T04:53:34.999 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.2446000000000001e-05 2024-02-05T04:53:34.999 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.999 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:34.999 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:34.999 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.207730+0000", 2024-02-05T04:53:34.999 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.4870000000000008e-06 2024-02-05T04:53:34.999 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:34.999 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.000 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:35.000 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.207796+0000", 2024-02-05T04:53:35.000 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.5852999999999994e-05 2024-02-05T04:53:35.000 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.000 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.000 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:53:35.000 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.207899+0000", 2024-02-05T04:53:35.000 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00010255900000000001 2024-02-05T04:53:35.000 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.000 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.000 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:35.000 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.208541+0000", 2024-02-05T04:53:35.000 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00064210899999999997 2024-02-05T04:53:35.000 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.000 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.001 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:35.001 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.212149+0000", 2024-02-05T04:53:35.001 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0036078170000000001 2024-02-05T04:53:35.001 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.001 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.001 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:53:35.001 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.213599+0000", 2024-02-05T04:53:35.001 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.001450935 2024-02-05T04:53:35.001 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.001 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.001 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:35.001 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.213655+0000", 2024-02-05T04:53:35.001 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.5192000000000003e-05 2024-02-05T04:53:35.001 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:35.001 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:35.001 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:35.001 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.002 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.002 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:255 3.2s0 3:41121c54:::smithi19639362-23:head [setxattr _header (56) in=7b] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:53:35.002 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:29.207757+0000", 2024-02-05T04:53:35.002 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 5.7637471570000001, 2024-02-05T04:53:35.002 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0060483919999999997, 2024-02-05T04:53:35.002 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:35.002 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:35.002 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:53:35.002 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:53:35.002 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:35.002 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 255 2024-02-05T04:53:35.002 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.002 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:35.002 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.002 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:35.003 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.207757+0000", 2024-02-05T04:53:35.003 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:35.003 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.003 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.003 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:35.003 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.207757+0000", 2024-02-05T04:53:35.003 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:35.003 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.003 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.003 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:35.003 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.207758+0000", 2024-02-05T04:53:35.003 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.2200000000000003e-07 2024-02-05T04:53:35.003 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.003 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.003 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:35.003 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.207761+0000", 2024-02-05T04:53:35.004 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.422e-06 2024-02-05T04:53:35.004 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.004 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.004 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:35.004 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.207771+0000", 2024-02-05T04:53:35.004 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.6600000000000007e-06 2024-02-05T04:53:35.004 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.004 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.004 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:35.004 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.207777+0000", 2024-02-05T04:53:35.004 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.2870000000000001e-06 2024-02-05T04:53:35.004 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.004 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.004 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:35.004 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.209029+0000", 2024-02-05T04:53:35.005 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.001251699 2024-02-05T04:53:35.005 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.005 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.005 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:53:35.005 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.209123+0000", 2024-02-05T04:53:35.005 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.4444000000000006e-05 2024-02-05T04:53:35.005 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.005 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.005 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:35.005 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.209555+0000", 2024-02-05T04:53:35.005 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00043236300000000001 2024-02-05T04:53:35.005 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.005 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.005 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:35.005 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.212475+0000", 2024-02-05T04:53:35.005 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0029194709999999999 2024-02-05T04:53:35.006 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.006 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.006 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:53:35.006 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.213756+0000", 2024-02-05T04:53:35.006 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.001281194 2024-02-05T04:53:35.006 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.006 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.006 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:35.006 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.213805+0000", 2024-02-05T04:53:35.006 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.9030000000000003e-05 2024-02-05T04:53:35.006 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:35.006 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:35.006 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:35.006 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.006 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.006 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:256 3.2s0 3:41121c54:::smithi19639362-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:53:35.007 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:29.207798+0000", 2024-02-05T04:53:35.007 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 5.7637058039999998, 2024-02-05T04:53:35.007 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0068739830000000002, 2024-02-05T04:53:35.007 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:35.007 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "started", 2024-02-05T04:53:35.007 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:53:35.007 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:53:35.007 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:35.007 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 256 2024-02-05T04:53:35.007 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.007 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:35.007 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.007 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:35.007 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.207798+0000", 2024-02-05T04:53:35.007 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:35.007 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.008 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.008 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:35.008 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.207798+0000", 2024-02-05T04:53:35.008 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:35.008 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.008 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.008 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:35.008 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.207799+0000", 2024-02-05T04:53:35.008 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.5899999999999996e-07 2024-02-05T04:53:35.008 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.008 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.008 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:35.008 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.207808+0000", 2024-02-05T04:53:35.008 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.7720000000000006e-06 2024-02-05T04:53:35.008 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.009 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.009 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:35.009 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.207818+0000", 2024-02-05T04:53:35.009 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.1050000000000001e-06 2024-02-05T04:53:35.009 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.009 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.009 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:35.009 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.207823+0000", 2024-02-05T04:53:35.009 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.8409999999999996e-06 2024-02-05T04:53:35.009 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.009 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.009 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:35.009 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.210496+0000", 2024-02-05T04:53:35.009 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0026729879999999998 2024-02-05T04:53:35.010 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.010 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.010 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for rw locks", 2024-02-05T04:53:35.010 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.210593+0000", 2024-02-05T04:53:35.010 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.6873999999999994e-05 2024-02-05T04:53:35.010 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.010 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.010 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:35.010 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.213830+0000", 2024-02-05T04:53:35.010 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0032368560000000002 2024-02-05T04:53:35.010 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.010 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.010 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:53:35.010 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.213901+0000", 2024-02-05T04:53:35.010 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.0813000000000001e-05 2024-02-05T04:53:35.011 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.011 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.011 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:35.011 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.214672+0000", 2024-02-05T04:53:35.011 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00077117500000000003 2024-02-05T04:53:35.011 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:35.011 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:35.011 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:35.011 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.011 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.011 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:257 3.2s0 3:4f006f6c:::smithi19639362-3:head [stat,delete] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:53:35.011 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:29.207834+0000", 2024-02-05T04:53:35.011 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 5.7636697649999995, 2024-02-05T04:53:35.011 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0085459170000000001, 2024-02-05T04:53:35.011 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:35.012 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:35.012 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:53:35.012 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:53:35.012 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:35.012 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 257 2024-02-05T04:53:35.012 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.012 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:35.012 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.012 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:35.012 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.207834+0000", 2024-02-05T04:53:35.012 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:35.012 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.012 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.012 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:35.012 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.207834+0000", 2024-02-05T04:53:35.012 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:35.013 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.013 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.013 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:35.013 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.207835+0000", 2024-02-05T04:53:35.013 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.13e-07 2024-02-05T04:53:35.013 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.013 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.013 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:35.013 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.207838+0000", 2024-02-05T04:53:35.013 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.96e-06 2024-02-05T04:53:35.013 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.013 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.013 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:35.013 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.207845+0000", 2024-02-05T04:53:35.013 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.6699999999999994e-06 2024-02-05T04:53:35.013 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.013 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.014 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:35.014 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.207851+0000", 2024-02-05T04:53:35.014 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.896e-06 2024-02-05T04:53:35.014 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.014 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.014 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:35.014 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.210632+0000", 2024-02-05T04:53:35.014 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00278087 2024-02-05T04:53:35.014 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.014 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.014 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:53:35.014 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.210723+0000", 2024-02-05T04:53:35.014 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.1265000000000001e-05 2024-02-05T04:53:35.014 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.014 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.015 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:35.015 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.211433+0000", 2024-02-05T04:53:35.015 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00070996899999999996 2024-02-05T04:53:35.015 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.015 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.015 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:35.015 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.214530+0000", 2024-02-05T04:53:35.015 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.003096634 2024-02-05T04:53:35.015 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.015 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.015 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:53:35.015 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.216308+0000", 2024-02-05T04:53:35.015 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00177809 2024-02-05T04:53:35.015 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.015 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.015 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:35.015 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.216380+0000", 2024-02-05T04:53:35.016 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.1950000000000001e-05 2024-02-05T04:53:35.016 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:35.016 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:35.016 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:35.016 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.016 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.016 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.6s1 38/31 ECSubWrite(tid=90, reqid=osd.13.0:90, at_version=38'11, trim_to=0'0, roll_forward_to=37'8))", 2024-02-05T04:53:35.016 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:30.141755+0000", 2024-02-05T04:53:35.016 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 4.8297484209999997, 2024-02-05T04:53:35.016 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0044243779999999996, 2024-02-05T04:53:35.016 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:35.016 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:35.016 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:35.016 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.016 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:35.017 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.141755+0000", 2024-02-05T04:53:35.017 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:35.017 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.017 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.017 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:35.017 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.141755+0000", 2024-02-05T04:53:35.017 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:35.017 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.017 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.017 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:35.017 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.141756+0000", 2024-02-05T04:53:35.017 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.2099999999999996e-07 2024-02-05T04:53:35.017 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.017 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.017 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:35.017 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.141777+0000", 2024-02-05T04:53:35.018 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.1135999999999999e-05 2024-02-05T04:53:35.018 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.018 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.018 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:35.018 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.141789+0000", 2024-02-05T04:53:35.018 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1562999999999999e-05 2024-02-05T04:53:35.018 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.018 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.018 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:35.018 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.141797+0000", 2024-02-05T04:53:35.018 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.5920000000000001e-06 2024-02-05T04:53:35.018 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.018 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.018 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:35.018 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.142042+0000", 2024-02-05T04:53:35.018 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00024489899999999997 2024-02-05T04:53:35.019 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.019 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.019 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:35.019 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.142054+0000", 2024-02-05T04:53:35.019 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1482e-05 2024-02-05T04:53:35.019 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.019 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.019 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:35.019 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.146133+0000", 2024-02-05T04:53:35.019 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.004079348 2024-02-05T04:53:35.019 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.019 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.019 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:35.019 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.146180+0000", 2024-02-05T04:53:35.019 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.6637000000000001e-05 2024-02-05T04:53:35.020 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:35.020 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:35.020 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:35.020 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.020 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.020 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.7s2 38/31 ECSubWrite(tid=92, reqid=osd.13.0:92, at_version=38'8, trim_to=0'0, roll_forward_to=37'6))", 2024-02-05T04:53:35.020 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:30.149790+0000", 2024-02-05T04:53:35.020 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 4.8217138159999999, 2024-02-05T04:53:35.020 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0040142550000000004, 2024-02-05T04:53:35.020 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:35.020 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:35.020 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:35.020 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.020 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:35.020 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.149790+0000", 2024-02-05T04:53:35.021 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:35.021 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.021 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.021 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:35.021 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.149790+0000", 2024-02-05T04:53:35.021 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:35.021 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.021 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.021 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:35.021 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.149791+0000", 2024-02-05T04:53:35.021 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.7000000000000003e-07 2024-02-05T04:53:35.021 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.021 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.021 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:35.021 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.149813+0000", 2024-02-05T04:53:35.021 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.1571000000000001e-05 2024-02-05T04:53:35.021 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.021 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.022 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:35.022 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.149824+0000", 2024-02-05T04:53:35.022 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1051e-05 2024-02-05T04:53:35.022 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.022 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.022 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:35.022 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.149831+0000", 2024-02-05T04:53:35.022 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.3749999999999997e-06 2024-02-05T04:53:35.022 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.022 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.022 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:35.022 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.149897+0000", 2024-02-05T04:53:35.022 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.5648999999999994e-05 2024-02-05T04:53:35.022 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.022 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.022 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:35.023 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.149906+0000", 2024-02-05T04:53:35.023 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.5980000000000003e-06 2024-02-05T04:53:35.023 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.023 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.023 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:35.023 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.153758+0000", 2024-02-05T04:53:35.023 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0038518269999999999 2024-02-05T04:53:35.023 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.023 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.023 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:35.023 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.153804+0000", 2024-02-05T04:53:35.023 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.6313999999999997e-05 2024-02-05T04:53:35.023 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:35.023 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:35.023 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:35.023 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.024 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.024 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.6s1 38/31 ECSubWrite(tid=93, reqid=osd.13.0:90, at_version=0'0, trim_to=0'0, roll_forward_to=38'11))", 2024-02-05T04:53:35.024 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:30.151613+0000", 2024-02-05T04:53:35.024 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 4.8198910179999999, 2024-02-05T04:53:35.024 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0026098330000000002, 2024-02-05T04:53:35.024 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:35.024 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:35.024 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:35.024 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.024 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:35.024 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.151613+0000", 2024-02-05T04:53:35.024 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:35.024 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.024 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.025 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:35.025 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.151613+0000", 2024-02-05T04:53:35.025 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:35.025 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.025 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.025 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:35.025 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.151614+0000", 2024-02-05T04:53:35.025 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.0699999999999996e-07 2024-02-05T04:53:35.025 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.025 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.025 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:35.025 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.151625+0000", 2024-02-05T04:53:35.025 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0943e-05 2024-02-05T04:53:35.025 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.025 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.025 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:35.026 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.151639+0000", 2024-02-05T04:53:35.026 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.4515000000000001e-05 2024-02-05T04:53:35.026 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.026 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.026 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:35.026 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.151649+0000", 2024-02-05T04:53:35.026 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0214000000000001e-05 2024-02-05T04:53:35.026 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.026 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.026 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:35.026 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.151724+0000", 2024-02-05T04:53:35.026 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.4826000000000005e-05 2024-02-05T04:53:35.026 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.026 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.026 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:35.027 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.151736+0000", 2024-02-05T04:53:35.027 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1467999999999999e-05 2024-02-05T04:53:35.027 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.027 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.027 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:35.027 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.154197+0000", 2024-02-05T04:53:35.027 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0024616540000000002 2024-02-05T04:53:35.027 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.027 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.027 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:35.027 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.154223+0000", 2024-02-05T04:53:35.027 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.5505999999999999e-05 2024-02-05T04:53:35.027 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:35.027 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:35.027 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:35.028 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.028 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.028 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.7s2 38/31 ECSubWrite(tid=94, reqid=osd.13.0:92, at_version=0'0, trim_to=0'0, roll_forward_to=38'8))", 2024-02-05T04:53:35.028 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:30.154165+0000", 2024-02-05T04:53:35.028 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 4.8173393610000002, 2024-02-05T04:53:35.028 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0021419030000000001, 2024-02-05T04:53:35.028 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:35.028 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:35.028 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:35.028 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.028 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:35.028 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.154165+0000", 2024-02-05T04:53:35.028 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:35.028 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.028 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.029 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:35.029 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.154165+0000", 2024-02-05T04:53:35.029 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:35.029 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.029 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.029 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:35.029 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.154165+0000", 2024-02-05T04:53:35.029 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.5899999999999995e-07 2024-02-05T04:53:35.029 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.029 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.029 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:35.029 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.154176+0000", 2024-02-05T04:53:35.029 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0878e-05 2024-02-05T04:53:35.029 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.029 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.029 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:35.030 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.154189+0000", 2024-02-05T04:53:35.030 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.2466000000000001e-05 2024-02-05T04:53:35.030 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.030 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.030 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:35.030 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.154196+0000", 2024-02-05T04:53:35.030 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.024e-06 2024-02-05T04:53:35.030 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.030 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.030 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:35.030 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.154264+0000", 2024-02-05T04:53:35.030 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.8654999999999999e-05 2024-02-05T04:53:35.030 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.030 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.030 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:35.031 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.154274+0000", 2024-02-05T04:53:35.031 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.6420000000000002e-06 2024-02-05T04:53:35.031 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.031 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.031 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:35.031 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.156271+0000", 2024-02-05T04:53:35.031 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0019967380000000001 2024-02-05T04:53:35.031 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.031 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.031 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:35.031 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.156306+0000", 2024-02-05T04:53:35.031 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.5741000000000002e-05 2024-02-05T04:53:35.031 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:35.031 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:35.031 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:35.031 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.032 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.032 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:279 3.2s0 3:41121c54:::smithi19639362-23:head [read 0~1441792,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-02-05T04:53:35.032 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:30.195246+0000", 2024-02-05T04:53:35.032 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 4.7762579220000001, 2024-02-05T04:53:35.032 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.004417514, 2024-02-05T04:53:35.032 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:35.032 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "started", 2024-02-05T04:53:35.032 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:53:35.032 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:53:35.032 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:35.032 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 279 2024-02-05T04:53:35.032 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.032 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:35.032 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.032 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:35.033 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.195246+0000", 2024-02-05T04:53:35.033 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:35.033 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.033 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.033 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:35.033 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.195246+0000", 2024-02-05T04:53:35.033 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:35.033 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.033 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.033 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:35.033 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.195247+0000", 2024-02-05T04:53:35.033 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.449e-06 2024-02-05T04:53:35.033 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.033 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.033 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:35.033 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.195253+0000", 2024-02-05T04:53:35.034 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.7819999999999999e-06 2024-02-05T04:53:35.034 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.034 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.034 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:35.034 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.195268+0000", 2024-02-05T04:53:35.034 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.4514e-05 2024-02-05T04:53:35.034 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.034 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.034 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:35.034 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.195276+0000", 2024-02-05T04:53:35.034 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.2390000000000002e-06 2024-02-05T04:53:35.034 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.034 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.034 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:35.034 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.195352+0000", 2024-02-05T04:53:35.034 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.5665000000000001e-05 2024-02-05T04:53:35.035 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.035 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.035 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:53:35.035 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.195419+0000", 2024-02-05T04:53:35.035 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.7486999999999999e-05 2024-02-05T04:53:35.035 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.035 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.035 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:35.035 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.199663+0000", 2024-02-05T04:53:35.035 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.004244378 2024-02-05T04:53:35.035 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:35.035 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:35.035 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:35.035 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.035 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.035 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:280 3.2s0 3:41121c54:::smithi19639362-23:head [read 0~1441792,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-02-05T04:53:35.036 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:30.195290+0000", 2024-02-05T04:53:35.036 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 4.7762137820000001, 2024-02-05T04:53:35.036 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0053070849999999996, 2024-02-05T04:53:35.036 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:35.036 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "started", 2024-02-05T04:53:35.036 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:53:35.036 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:53:35.036 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:35.036 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 280 2024-02-05T04:53:35.036 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.036 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:35.036 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.036 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:35.036 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.195290+0000", 2024-02-05T04:53:35.036 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:35.037 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.037 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.037 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:35.037 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.195290+0000", 2024-02-05T04:53:35.037 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:35.037 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.037 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.037 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:35.037 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.195290+0000", 2024-02-05T04:53:35.037 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.4799999999999999e-07 2024-02-05T04:53:35.037 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.037 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.037 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:35.037 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.195294+0000", 2024-02-05T04:53:35.037 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.2160000000000002e-06 2024-02-05T04:53:35.038 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.038 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.038 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:35.038 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.195299+0000", 2024-02-05T04:53:35.038 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.3449999999999996e-06 2024-02-05T04:53:35.038 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.038 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.038 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:35.038 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.195302+0000", 2024-02-05T04:53:35.038 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.4249999999999998e-06 2024-02-05T04:53:35.038 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.038 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.038 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:35.038 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.195655+0000", 2024-02-05T04:53:35.038 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00035282899999999998 2024-02-05T04:53:35.039 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.039 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.039 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:53:35.039 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.195704+0000", 2024-02-05T04:53:35.039 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.8402999999999997e-05 2024-02-05T04:53:35.039 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.039 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.039 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:35.039 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.200597+0000", 2024-02-05T04:53:35.039 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0048935189999999998 2024-02-05T04:53:35.039 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:35.039 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:35.039 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:35.039 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.039 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.040 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:281 3.2s0 3:41121c54:::smithi19639362-23:head [read 0~1441792,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-02-05T04:53:35.040 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:30.195307+0000", 2024-02-05T04:53:35.040 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 4.7761965990000004, 2024-02-05T04:53:35.040 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0063372669999999997, 2024-02-05T04:53:35.040 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:35.040 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "started", 2024-02-05T04:53:35.040 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:53:35.040 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:53:35.040 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:35.040 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 281 2024-02-05T04:53:35.040 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.040 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:35.040 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.040 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:35.040 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.195307+0000", 2024-02-05T04:53:35.041 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:35.041 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.041 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.041 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:35.041 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.195307+0000", 2024-02-05T04:53:35.041 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:35.041 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.041 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.041 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:35.041 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.195308+0000", 2024-02-05T04:53:35.041 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.41e-07 2024-02-05T04:53:35.041 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.041 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.041 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:35.041 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.195309+0000", 2024-02-05T04:53:35.041 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.5659999999999999e-06 2024-02-05T04:53:35.041 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.042 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.042 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:35.042 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.195312+0000", 2024-02-05T04:53:35.042 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.242e-06 2024-02-05T04:53:35.042 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.042 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.042 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:35.042 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.195316+0000", 2024-02-05T04:53:35.042 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.2559999999999998e-06 2024-02-05T04:53:35.042 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.042 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.042 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:35.042 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.195897+0000", 2024-02-05T04:53:35.042 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00058133200000000005 2024-02-05T04:53:35.042 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.043 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.043 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:53:35.043 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.195943+0000", 2024-02-05T04:53:35.043 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.6452999999999997e-05 2024-02-05T04:53:35.043 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.043 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.043 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:35.043 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.201645+0000", 2024-02-05T04:53:35.043 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.005701177 2024-02-05T04:53:35.043 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:35.043 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:35.043 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:35.043 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.043 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.043 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpReadReply(3.2s0 38/31 ECSubReadReply(tid=32, attrs_read=0))", 2024-02-05T04:53:35.043 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:30.197060+0000", 2024-02-05T04:53:35.044 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 4.7744436769999998, 2024-02-05T04:53:35.044 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.002647595, 2024-02-05T04:53:35.044 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:35.044 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:35.044 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:35.044 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.044 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:35.044 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.197060+0000", 2024-02-05T04:53:35.044 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:35.044 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.044 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.044 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:35.044 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.197060+0000", 2024-02-05T04:53:35.044 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:35.044 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.045 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.045 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:35.045 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.197061+0000", 2024-02-05T04:53:35.045 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.1900000000000002e-07 2024-02-05T04:53:35.045 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.045 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.045 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:35.045 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.198833+0000", 2024-02-05T04:53:35.045 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.001772236 2024-02-05T04:53:35.045 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.045 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.045 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:35.045 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.198840+0000", 2024-02-05T04:53:35.045 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.3270000000000003e-06 2024-02-05T04:53:35.045 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.045 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.045 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:35.046 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.198844+0000", 2024-02-05T04:53:35.046 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.9920000000000001e-06 2024-02-05T04:53:35.046 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.046 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.046 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:35.046 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.198931+0000", 2024-02-05T04:53:35.046 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.6965000000000005e-05 2024-02-05T04:53:35.046 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.046 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.046 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:35.046 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.199708+0000", 2024-02-05T04:53:35.046 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00077645600000000002 2024-02-05T04:53:35.046 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:35.046 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:35.046 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:35.047 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.047 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.047 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpReadReply(3.2s0 38/31 ECSubReadReply(tid=34, attrs_read=0))", 2024-02-05T04:53:35.047 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:30.199334+0000", 2024-02-05T04:53:35.047 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 4.7721700069999997, 2024-02-05T04:53:35.047 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0023414999999999998, 2024-02-05T04:53:35.047 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:35.047 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:35.047 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:35.047 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.047 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:35.047 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.199334+0000", 2024-02-05T04:53:35.047 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:35.047 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.047 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.048 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:35.048 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.199334+0000", 2024-02-05T04:53:35.048 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:35.048 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.048 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.048 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:35.048 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.199334+0000", 2024-02-05T04:53:35.048 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.5699999999999998e-07 2024-02-05T04:53:35.048 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.048 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.048 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:35.048 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.199752+0000", 2024-02-05T04:53:35.048 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00041769700000000002 2024-02-05T04:53:35.048 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.048 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.049 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:35.049 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.199759+0000", 2024-02-05T04:53:35.049 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.7109999999999998e-06 2024-02-05T04:53:35.049 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.049 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.049 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:35.049 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.199764+0000", 2024-02-05T04:53:35.049 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.989e-06 2024-02-05T04:53:35.049 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.049 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.049 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:35.049 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.200750+0000", 2024-02-05T04:53:35.049 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00098594000000000004 2024-02-05T04:53:35.049 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.049 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.050 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:35.050 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.201675+0000", 2024-02-05T04:53:35.050 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00092580600000000005 2024-02-05T04:53:35.050 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:35.050 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:35.050 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:35.050 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.050 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.050 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:381 3.2s0 3:4d05e2f1:::smithi19639362-34:head [delete] snapc 7=[7,6,5,4] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e42)", 2024-02-05T04:53:35.050 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:34.158968+0000", 2024-02-05T04:53:35.050 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 0.81253610200000004, 2024-02-05T04:53:35.050 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0029229970000000001, 2024-02-05T04:53:35.050 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:35.050 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:35.050 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:53:35.051 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:53:35.051 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:35.051 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 381 2024-02-05T04:53:35.051 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.051 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:35.051 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.051 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:35.051 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.158968+0000", 2024-02-05T04:53:35.051 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:35.051 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.051 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.051 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:35.051 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.158968+0000", 2024-02-05T04:53:35.051 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:35.051 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.051 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.052 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:35.052 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.158970+0000", 2024-02-05T04:53:35.052 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.1430000000000001e-06 2024-02-05T04:53:35.052 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.052 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.052 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:35.052 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.158980+0000", 2024-02-05T04:53:35.052 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.6870000000000006e-06 2024-02-05T04:53:35.052 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.052 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.052 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:35.052 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.159000+0000", 2024-02-05T04:53:35.052 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.0292000000000002e-05 2024-02-05T04:53:35.052 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.052 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.053 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:35.053 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.159013+0000", 2024-02-05T04:53:35.053 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.2772000000000001e-05 2024-02-05T04:53:35.053 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.053 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.053 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:35.053 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.159085+0000", 2024-02-05T04:53:35.053 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.1940999999999994e-05 2024-02-05T04:53:35.053 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.053 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.053 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:53:35.053 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.159193+0000", 2024-02-05T04:53:35.053 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00010877700000000001 2024-02-05T04:53:35.053 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.053 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.053 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:35.054 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.159968+0000", 2024-02-05T04:53:35.054 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00077457399999999999 2024-02-05T04:53:35.054 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.054 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.054 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:35.054 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.161034+0000", 2024-02-05T04:53:35.054 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.001065954 2024-02-05T04:53:35.054 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.054 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.054 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:53:35.054 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.161810+0000", 2024-02-05T04:53:35.054 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00077623600000000003 2024-02-05T04:53:35.054 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.054 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.054 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:35.054 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.161891+0000", 2024-02-05T04:53:35.055 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.0620999999999999e-05 2024-02-05T04:53:35.055 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:35.055 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:35.055 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:35.055 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.055 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.055 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpRead(3.7s2 42/31 ECSubRead(tid=134, to_read={3:e8f6ce71:::smithi19639362-28:head=0,114688,0}, subchunks={3:e8f6ce71:::smithi19639362-28:head=[0,1]}, attrs_to_read=))", 2024-02-05T04:53:35.055 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:34.162533+0000", 2024-02-05T04:53:35.055 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 0.80897130100000003, 2024-02-05T04:53:35.055 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0025809499999999998, 2024-02-05T04:53:35.055 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:35.055 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:35.055 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:35.055 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.055 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:35.056 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.162533+0000", 2024-02-05T04:53:35.056 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:35.056 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.056 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.056 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:35.056 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.162533+0000", 2024-02-05T04:53:35.056 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:35.056 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.056 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.056 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:35.056 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.162533+0000", 2024-02-05T04:53:35.056 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.4999999999999999e-07 2024-02-05T04:53:35.056 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.056 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.056 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:35.057 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.162538+0000", 2024-02-05T04:53:35.057 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.5770000000000001e-06 2024-02-05T04:53:35.057 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.057 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.057 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:35.057 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.162552+0000", 2024-02-05T04:53:35.057 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.3429e-05 2024-02-05T04:53:35.057 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.057 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.057 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:35.057 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.162562+0000", 2024-02-05T04:53:35.057 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0098999999999999e-05 2024-02-05T04:53:35.057 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.057 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.057 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:35.057 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.162632+0000", 2024-02-05T04:53:35.058 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.9578000000000004e-05 2024-02-05T04:53:35.058 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:35.058 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:35.058 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:35.058 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.165114+0000", 2024-02-05T04:53:35.058 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.002482017 2024-02-05T04:53:35.058 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:35.058 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:35.058 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:35.058 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:35.058 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:35.058 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-02-05T04:53:35.060 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-02-05T04:53:35.060 INFO:tasks.ceph.osd.7.smithi196.stderr:2024-02-05T04:53:35.059+0000 7fb281419640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34606) UID: 0 2024-02-05T04:53:35.158 INFO:teuthology.orchestra.run.smithi062.stderr:reweighted osd.14 to 1 (10000) 2024-02-05T04:53:35.160 INFO:tasks.ceph.osd.1.smithi091.stderr:2024-02-05T04:53:35.159+0000 7ff9dc283640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34275) UID: 0 2024-02-05T04:53:35.161 INFO:tasks.rados.rados.0.smithi196.stdout:145: done (4 left) 2024-02-05T04:53:35.162 INFO:tasks.rados.rados.0.smithi196.stdout:146: delete oid 34 current snap is 6 2024-02-05T04:53:35.162 INFO:tasks.rados.rados.0.smithi196.stdout:138: finishing write tid 1 to smithi19639362-19 2024-02-05T04:53:35.162 INFO:tasks.rados.rados.0.smithi196.stdout:138: finishing write tid 2 to smithi19639362-19 2024-02-05T04:53:35.162 INFO:tasks.rados.rados.0.smithi196.stdout:138: finishing write tid 3 to smithi19639362-19 2024-02-05T04:53:35.162 INFO:tasks.rados.rados.0.smithi196.stdout:138: finishing write tid 4 to smithi19639362-19 2024-02-05T04:53:35.162 INFO:tasks.rados.rados.0.smithi196.stdout:138: finishing write tid 5 to smithi19639362-19 2024-02-05T04:53:35.162 INFO:tasks.rados.rados.0.smithi196.stdout:138: oid 19 updating version 0 to 28 2024-02-05T04:53:35.162 INFO:tasks.rados.rados.0.smithi196.stdout:138: oid 19 updating version 28 to 30 2024-02-05T04:53:35.162 INFO:tasks.rados.rados.0.smithi196.stdout:138: oid 19 version 30 is already newer than 27 2024-02-05T04:53:35.162 INFO:tasks.rados.rados.0.smithi196.stdout:138: oid 19 version 30 is already newer than 29 2024-02-05T04:53:35.162 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 19 v 30 (ObjNum 84 snap 6 seq_num 84) dirty exists 2024-02-05T04:53:35.162 INFO:tasks.rados.rados.0.smithi196.stdout:138: left oid 19 (ObjNum 84 snap 6 seq_num 84) 2024-02-05T04:53:35.162 INFO:tasks.rados.rados.0.smithi196.stdout:139: expect (ObjNum 38 snap 0 seq_num 38) 2024-02-05T04:53:35.169 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-02-05T04:53:35.174 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 15 1 2024-02-05T04:53:35.182 INFO:tasks.rados.rados.0.smithi196.stdout:142: finishing copy_from racing read to smithi19639362-30 2024-02-05T04:53:35.182 INFO:tasks.rados.rados.0.smithi196.stdout:142: finishing copy_from to smithi19639362-30 2024-02-05T04:53:35.182 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 30 v 20 (ObjNum 60 snap 2 seq_num 60) dirty exists 2024-02-05T04:53:35.182 INFO:tasks.rados.rados.0.smithi196.stdout:144: finishing write tid 1 to smithi19639362-9 2024-02-05T04:53:35.182 INFO:tasks.rados.rados.0.smithi196.stdout:144: finishing write tid 2 to smithi19639362-9 2024-02-05T04:53:35.182 INFO:tasks.rados.rados.0.smithi196.stdout:144: finishing write tid 3 to smithi19639362-9 2024-02-05T04:53:35.182 INFO:tasks.rados.rados.0.smithi196.stdout:144: finishing write tid 4 to smithi19639362-9 2024-02-05T04:53:35.183 INFO:tasks.rados.rados.0.smithi196.stdout:144: oid 9 updating version 0 to 25 2024-02-05T04:53:35.183 INFO:tasks.rados.rados.0.smithi196.stdout:144: oid 9 updating version 25 to 26 2024-02-05T04:53:35.183 INFO:tasks.rados.rados.0.smithi196.stdout:144: oid 9 updating version 26 to 27 2024-02-05T04:53:35.183 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 9 v 27 (ObjNum 85 snap 6 seq_num 85) dirty exists 2024-02-05T04:53:35.183 INFO:tasks.rados.rados.0.smithi196.stdout:144: left oid 9 (ObjNum 85 snap 6 seq_num 85) 2024-02-05T04:53:35.183 INFO:tasks.rados.rados.0.smithi196.stdout:138: done (4 left) 2024-02-05T04:53:35.183 INFO:tasks.rados.rados.0.smithi196.stdout:139: done (3 left) 2024-02-05T04:53:35.183 INFO:teuthology.orchestra.run.smithi091.stdout:{ 2024-02-05T04:53:35.184 INFO:teuthology.orchestra.run.smithi091.stdout: "ops": [ 2024-02-05T04:53:35.184 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.184 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.3s2 43/31 ECSubWrite(tid=65, reqid=osd.4.0:63, at_version=0'0, trim_to=0'0, roll_forward_to=43'27))", 2024-02-05T04:53:35.184 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:35.182050+0000", 2024-02-05T04:53:35.184 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 0.00080653799999999998, 2024-02-05T04:53:35.184 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00082805499999999996, 2024-02-05T04:53:35.184 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:35.184 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:35.184 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:35.184 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.184 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:35.184 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.182050+0000", 2024-02-05T04:53:35.184 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:35.185 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.185 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.185 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:35.185 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.182050+0000", 2024-02-05T04:53:35.185 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:35.185 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.185 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.185 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:35.185 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.182050+0000", 2024-02-05T04:53:35.185 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.5899999999999996e-07 2024-02-05T04:53:35.185 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.185 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.185 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:35.185 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.182060+0000", 2024-02-05T04:53:35.185 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.1549999999999996e-06 2024-02-05T04:53:35.186 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.186 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.186 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:35.186 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.182071+0000", 2024-02-05T04:53:35.186 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.1411e-05 2024-02-05T04:53:35.186 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.186 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.186 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:35.186 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.182079+0000", 2024-02-05T04:53:35.186 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.1510000000000004e-06 2024-02-05T04:53:35.186 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.186 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.186 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:35.186 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.182189+0000", 2024-02-05T04:53:35.186 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.000109514 2024-02-05T04:53:35.186 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.187 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.187 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:35.187 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.182197+0000", 2024-02-05T04:53:35.187 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.1799999999999996e-06 2024-02-05T04:53:35.187 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:35.187 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:35.187 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:35.187 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.187 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.187 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.6s4 43/31 ECSubWrite(tid=141, reqid=osd.13.0:138, at_version=0'0, trim_to=0'0, roll_forward_to=43'24))", 2024-02-05T04:53:35.187 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:35.182634+0000", 2024-02-05T04:53:35.187 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 0.00022261399999999999, 2024-02-05T04:53:35.187 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00030832800000000002, 2024-02-05T04:53:35.187 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:35.187 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:35.187 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:35.187 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.188 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:35.188 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.182634+0000", 2024-02-05T04:53:35.188 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:35.188 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.188 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.188 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:35.188 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.182634+0000", 2024-02-05T04:53:35.188 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:35.188 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.188 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.188 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:35.188 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.182634+0000", 2024-02-05T04:53:35.188 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.0999999999999998e-07 2024-02-05T04:53:35.188 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.189 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.189 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:35.189 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.182644+0000", 2024-02-05T04:53:35.189 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.5170000000000005e-06 2024-02-05T04:53:35.189 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.189 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.189 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:35.189 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.182656+0000", 2024-02-05T04:53:35.189 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.1623999999999999e-05 2024-02-05T04:53:35.189 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.189 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.189 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:35.189 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.182663+0000", 2024-02-05T04:53:35.189 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.5870000000000004e-06 2024-02-05T04:53:35.189 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.189 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.190 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:35.190 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.182729+0000", 2024-02-05T04:53:35.190 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.5804999999999998e-05 2024-02-05T04:53:35.190 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.190 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.190 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:35.190 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.182742+0000", 2024-02-05T04:53:35.190 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.2612000000000001e-05 2024-02-05T04:53:35.190 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:35.190 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:35.190 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:35.190 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:35.190 INFO:teuthology.orchestra.run.smithi091.stdout: ], 2024-02-05T04:53:35.190 INFO:teuthology.orchestra.run.smithi091.stdout: "num_ops": 2 2024-02-05T04:53:35.190 INFO:teuthology.orchestra.run.smithi091.stdout:} 2024-02-05T04:53:35.191 INFO:tasks.rados.rados.0.smithi196.stdout:142: done (2 left) 2024-02-05T04:53:35.191 INFO:tasks.rados.rados.0.smithi196.stdout:144: done (1 left) 2024-02-05T04:53:35.191 INFO:tasks.rados.rados.0.smithi196.stdout:146: done (0 left) 2024-02-05T04:53:35.191 INFO:tasks.rados.rados.0.smithi196.stdout:147: read oid 8 snap -1 2024-02-05T04:53:35.191 INFO:tasks.rados.rados.0.smithi196.stdout:147: expect (ObjNum 43 snap 0 seq_num 43) 2024-02-05T04:53:35.191 INFO:tasks.rados.rados.0.smithi196.stdout:148: delete oid 50 current snap is 6 2024-02-05T04:53:35.191 INFO:tasks.rados.rados.0.smithi196.stdout:148: done (1 left) 2024-02-05T04:53:35.191 INFO:tasks.rados.rados.0.smithi196.stdout:149: snap_remove snap 2 2024-02-05T04:53:35.195 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-02-05T04:53:35.261 INFO:tasks.ceph.osd.12.smithi062.stderr:2024-02-05T04:53:35.261+0000 7f5e5847e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 34695) UID: 0 2024-02-05T04:53:35.339 INFO:teuthology.orchestra.run.smithi091.stdout:{ 2024-02-05T04:53:35.339 INFO:teuthology.orchestra.run.smithi091.stdout: "ops": [], 2024-02-05T04:53:35.339 INFO:teuthology.orchestra.run.smithi091.stdout: "complaint_time": 30, 2024-02-05T04:53:35.339 INFO:teuthology.orchestra.run.smithi091.stdout: "num_blocked_ops": 0 2024-02-05T04:53:35.340 INFO:teuthology.orchestra.run.smithi091.stdout:} 2024-02-05T04:53:35.348 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-02-05T04:53:35.362 INFO:tasks.ceph.osd.3.smithi196.stderr:2024-02-05T04:53:35.362+0000 7fc8817c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34605) UID: 0 2024-02-05T04:53:35.454 INFO:teuthology.orchestra.run.smithi091.stdout:{ 2024-02-05T04:53:35.455 INFO:teuthology.orchestra.run.smithi091.stdout: "size": 20, 2024-02-05T04:53:35.455 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 600, 2024-02-05T04:53:35.455 INFO:teuthology.orchestra.run.smithi091.stdout: "ops": [ 2024-02-05T04:53:35.455 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.455 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:216 3.4s0 3:25248d28:::smithi19639362-18:head [read 0~294912,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-02-05T04:53:35.455 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:28.122406+0000", 2024-02-05T04:53:35.455 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 7.330385325, 2024-02-05T04:53:35.455 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0056680589999999996, 2024-02-05T04:53:35.455 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:35.455 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "started", 2024-02-05T04:53:35.455 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:53:35.455 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:53:35.455 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:35.456 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 216 2024-02-05T04:53:35.456 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.456 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:35.456 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.456 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:35.456 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.122406+0000", 2024-02-05T04:53:35.456 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:35.456 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.456 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.456 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:35.456 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.122406+0000", 2024-02-05T04:53:35.456 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:35.456 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.456 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.456 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:35.457 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.122406+0000", 2024-02-05T04:53:35.457 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.8000000000000001e-07 2024-02-05T04:53:35.457 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.457 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.457 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:35.457 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.122408+0000", 2024-02-05T04:53:35.457 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.4600000000000002e-06 2024-02-05T04:53:35.457 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.457 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.457 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:35.457 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.122413+0000", 2024-02-05T04:53:35.457 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.0209999999999999e-06 2024-02-05T04:53:35.457 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.457 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.457 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:35.458 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.122417+0000", 2024-02-05T04:53:35.458 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.6239999999999999e-06 2024-02-05T04:53:35.458 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.458 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.458 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:35.458 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.122541+0000", 2024-02-05T04:53:35.458 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00012391100000000001 2024-02-05T04:53:35.458 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.458 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.458 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "waiting_for_map not empty", 2024-02-05T04:53:35.458 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.122547+0000", 2024-02-05T04:53:35.458 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.682e-06 2024-02-05T04:53:35.458 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.458 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.458 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:35.459 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.127053+0000", 2024-02-05T04:53:35.459 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0045067170000000004 2024-02-05T04:53:35.459 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.459 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.459 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:53:35.459 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.127105+0000", 2024-02-05T04:53:35.459 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.1458000000000001e-05 2024-02-05T04:53:35.459 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.459 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.459 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:35.459 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.128074+0000", 2024-02-05T04:53:35.459 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00096880600000000001 2024-02-05T04:53:35.459 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:35.459 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:35.459 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:35.460 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.460 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.460 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.3s2 37/31 ECSubWrite(tid=30, reqid=client.4634.0:219, at_version=37'13, trim_to=0'0, roll_forward_to=37'10))", 2024-02-05T04:53:35.460 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:29.136955+0000", 2024-02-05T04:53:35.460 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 6.3158355039999998, 2024-02-05T04:53:35.460 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0062065059999999997, 2024-02-05T04:53:35.460 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:35.460 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:35.460 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:35.460 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.460 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:35.460 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.136955+0000", 2024-02-05T04:53:35.460 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:35.460 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.461 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.461 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:35.461 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.136955+0000", 2024-02-05T04:53:35.461 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:35.461 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.461 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.461 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:35.461 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.136956+0000", 2024-02-05T04:53:35.461 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.8299999999999996e-07 2024-02-05T04:53:35.461 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.461 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.461 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:35.461 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.137016+0000", 2024-02-05T04:53:35.461 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.961e-05 2024-02-05T04:53:35.461 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.461 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.462 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:35.462 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.137030+0000", 2024-02-05T04:53:35.462 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.4121000000000001e-05 2024-02-05T04:53:35.462 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.462 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.462 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:35.462 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.137040+0000", 2024-02-05T04:53:35.462 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.7350000000000008e-06 2024-02-05T04:53:35.462 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.462 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.462 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:35.462 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.137102+0000", 2024-02-05T04:53:35.462 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.1977000000000006e-05 2024-02-05T04:53:35.462 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.462 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.463 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:35.463 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.137111+0000", 2024-02-05T04:53:35.463 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.7690000000000008e-06 2024-02-05T04:53:35.463 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.463 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.463 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:35.463 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.143129+0000", 2024-02-05T04:53:35.463 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0060175410000000004 2024-02-05T04:53:35.463 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.463 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.463 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:35.463 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.143162+0000", 2024-02-05T04:53:35.463 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.307e-05 2024-02-05T04:53:35.463 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:35.463 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:35.463 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:35.464 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.464 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.464 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.3s2 37/31 ECSubWrite(tid=31, reqid=client.4634.0:220, at_version=37'14, trim_to=0'0, roll_forward_to=37'10))", 2024-02-05T04:53:35.464 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:29.138096+0000", 2024-02-05T04:53:35.464 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 6.3146948319999998, 2024-02-05T04:53:35.464 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0050892680000000001, 2024-02-05T04:53:35.464 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:35.464 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:35.464 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:35.464 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.464 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:35.464 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.138096+0000", 2024-02-05T04:53:35.464 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:35.464 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.464 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.464 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:35.465 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.138096+0000", 2024-02-05T04:53:35.465 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:35.465 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.465 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.465 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:35.465 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.138097+0000", 2024-02-05T04:53:35.465 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.3000000000000001e-07 2024-02-05T04:53:35.465 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.465 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.465 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:35.465 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.138111+0000", 2024-02-05T04:53:35.465 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.4810999999999999e-05 2024-02-05T04:53:35.465 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.465 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.465 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:35.466 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.138121+0000", 2024-02-05T04:53:35.466 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.9769999999999995e-06 2024-02-05T04:53:35.466 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.466 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.466 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:35.466 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.138129+0000", 2024-02-05T04:53:35.466 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.3370000000000002e-06 2024-02-05T04:53:35.466 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.466 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.466 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:35.466 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.138177+0000", 2024-02-05T04:53:35.466 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.8383000000000001e-05 2024-02-05T04:53:35.466 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.466 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.466 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:35.466 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.138182+0000", 2024-02-05T04:53:35.467 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.3249999999999998e-06 2024-02-05T04:53:35.467 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.467 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.467 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:35.467 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.143166+0000", 2024-02-05T04:53:35.467 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0049834340000000001 2024-02-05T04:53:35.467 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.467 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.467 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:35.467 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.143185+0000", 2024-02-05T04:53:35.467 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.9471e-05 2024-02-05T04:53:35.467 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:35.467 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:35.467 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:35.467 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.468 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.468 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.6s4 37/31 ECSubWrite(tid=75, reqid=client.4634.0:228, at_version=37'7, trim_to=0'0, roll_forward_to=34'4))", 2024-02-05T04:53:35.468 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:29.149273+0000", 2024-02-05T04:53:35.468 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 6.303518349, 2024-02-05T04:53:35.468 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0048548230000000003, 2024-02-05T04:53:35.468 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:35.468 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:35.468 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:35.468 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.468 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:35.468 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.149273+0000", 2024-02-05T04:53:35.468 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:35.468 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.468 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.469 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:35.469 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.149273+0000", 2024-02-05T04:53:35.469 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:35.469 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.469 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.469 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:35.469 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.149273+0000", 2024-02-05T04:53:35.469 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.5600000000000005e-07 2024-02-05T04:53:35.469 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.469 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.469 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:35.469 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.149289+0000", 2024-02-05T04:53:35.469 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.5872999999999999e-05 2024-02-05T04:53:35.469 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.469 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.470 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:35.470 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.149304+0000", 2024-02-05T04:53:35.470 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.5031999999999999e-05 2024-02-05T04:53:35.470 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.470 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.470 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:35.470 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.149315+0000", 2024-02-05T04:53:35.470 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.0455000000000001e-05 2024-02-05T04:53:35.470 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.470 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.470 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:35.470 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.149384+0000", 2024-02-05T04:53:35.470 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.9482999999999999e-05 2024-02-05T04:53:35.470 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.470 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.470 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:35.471 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.149394+0000", 2024-02-05T04:53:35.471 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.0243e-05 2024-02-05T04:53:35.471 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.471 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.471 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:35.471 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.154092+0000", 2024-02-05T04:53:35.471 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0046977429999999999 2024-02-05T04:53:35.471 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.471 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.471 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:35.471 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.154127+0000", 2024-02-05T04:53:35.471 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.5337999999999999e-05 2024-02-05T04:53:35.471 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:35.471 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:35.471 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:35.472 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.472 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.472 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:239 3.4s0 3:2c242b83:::smithi19639362-9:head [append 0~655360] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:53:35.472 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:29.155211+0000", 2024-02-05T04:53:35.472 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 6.297580172, 2024-02-05T04:53:35.472 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0067722930000000004, 2024-02-05T04:53:35.472 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:35.472 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:35.472 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:53:35.472 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:53:35.472 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:35.472 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 239 2024-02-05T04:53:35.472 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.472 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:35.472 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.473 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:35.473 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.155211+0000", 2024-02-05T04:53:35.473 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:35.473 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.473 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.473 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:35.473 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.155211+0000", 2024-02-05T04:53:35.473 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:35.473 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.473 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.473 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:35.473 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.155212+0000", 2024-02-05T04:53:35.473 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.302e-06 2024-02-05T04:53:35.473 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.473 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.474 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:35.474 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.156080+0000", 2024-02-05T04:53:35.474 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00086819000000000002 2024-02-05T04:53:35.474 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.474 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.474 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:35.474 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.156094+0000", 2024-02-05T04:53:35.474 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.3637e-05 2024-02-05T04:53:35.474 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.474 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.474 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:35.474 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.156101+0000", 2024-02-05T04:53:35.474 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.5619999999999998e-06 2024-02-05T04:53:35.474 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.474 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.475 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:35.475 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.156158+0000", 2024-02-05T04:53:35.475 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.6595999999999998e-05 2024-02-05T04:53:35.475 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.475 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.475 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:53:35.475 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.156235+0000", 2024-02-05T04:53:35.475 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.7181999999999994e-05 2024-02-05T04:53:35.475 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.475 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.475 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:35.475 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.157325+0000", 2024-02-05T04:53:35.475 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.001089558 2024-02-05T04:53:35.475 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.475 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.475 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:35.476 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.160456+0000", 2024-02-05T04:53:35.476 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.003131694 2024-02-05T04:53:35.476 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.476 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.476 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "commit_sent", 2024-02-05T04:53:35.476 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.161914+0000", 2024-02-05T04:53:35.476 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.001457814 2024-02-05T04:53:35.476 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.476 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.476 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:35.476 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.161983+0000", 2024-02-05T04:53:35.476 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.8757999999999995e-05 2024-02-05T04:53:35.476 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:35.476 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:35.476 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:35.477 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.477 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.477 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:240 3.4s0 3:2c242b83:::smithi19639362-9:head [append 0~376832] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:53:35.477 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:29.157254+0000", 2024-02-05T04:53:35.477 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 6.2955374050000001, 2024-02-05T04:53:35.477 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.005019819, 2024-02-05T04:53:35.477 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:35.477 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:35.477 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:53:35.477 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:53:35.477 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:35.477 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 240 2024-02-05T04:53:35.477 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.477 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:35.477 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.477 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:35.478 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.157254+0000", 2024-02-05T04:53:35.478 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:35.478 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.478 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.478 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:35.478 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.157254+0000", 2024-02-05T04:53:35.478 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:35.478 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.478 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.478 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:35.478 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.157255+0000", 2024-02-05T04:53:35.478 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.102e-06 2024-02-05T04:53:35.478 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.478 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.478 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:35.478 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.158415+0000", 2024-02-05T04:53:35.479 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0011606349999999999 2024-02-05T04:53:35.479 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.479 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.479 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:35.479 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.158433+0000", 2024-02-05T04:53:35.479 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.7288000000000001e-05 2024-02-05T04:53:35.479 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.479 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.479 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:35.479 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.158441+0000", 2024-02-05T04:53:35.479 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.6780000000000006e-06 2024-02-05T04:53:35.479 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.479 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.479 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:35.479 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.158499+0000", 2024-02-05T04:53:35.479 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.8145e-05 2024-02-05T04:53:35.480 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.480 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.480 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:53:35.480 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.158585+0000", 2024-02-05T04:53:35.480 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.5371999999999995e-05 2024-02-05T04:53:35.480 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.480 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.480 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:35.480 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.159118+0000", 2024-02-05T04:53:35.480 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00053369599999999995 2024-02-05T04:53:35.480 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.480 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.480 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:35.480 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.160458+0000", 2024-02-05T04:53:35.480 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0013391639999999999 2024-02-05T04:53:35.481 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.481 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.481 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "commit_sent", 2024-02-05T04:53:35.481 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.162238+0000", 2024-02-05T04:53:35.481 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.001780429 2024-02-05T04:53:35.481 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.481 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.481 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:35.481 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.162273+0000", 2024-02-05T04:53:35.481 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.5309999999999999e-05 2024-02-05T04:53:35.481 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:35.481 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:35.481 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:35.481 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.481 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.482 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:242 3.4s0 3:2c242b83:::smithi19639362-9:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:53:35.482 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:29.158493+0000", 2024-02-05T04:53:35.482 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 6.2942979399999999, 2024-02-05T04:53:35.482 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0052039679999999998, 2024-02-05T04:53:35.482 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:35.482 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "started", 2024-02-05T04:53:35.482 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:53:35.482 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:53:35.482 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:35.482 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 242 2024-02-05T04:53:35.482 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.482 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:35.482 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.482 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:35.482 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.158493+0000", 2024-02-05T04:53:35.483 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:35.483 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.483 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.483 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:35.483 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.158493+0000", 2024-02-05T04:53:35.483 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:35.483 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.483 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.483 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:35.483 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.158494+0000", 2024-02-05T04:53:35.483 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.06e-07 2024-02-05T04:53:35.483 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.483 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.484 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:35.484 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.158496+0000", 2024-02-05T04:53:35.484 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.3709999999999998e-06 2024-02-05T04:53:35.484 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.484 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.484 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:35.484 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.158503+0000", 2024-02-05T04:53:35.484 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.1459999999999999e-06 2024-02-05T04:53:35.484 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.484 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.484 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:35.484 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.158509+0000", 2024-02-05T04:53:35.484 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.7960000000000001e-06 2024-02-05T04:53:35.484 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.485 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.485 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:35.485 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.160639+0000", 2024-02-05T04:53:35.485 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0021304269999999998 2024-02-05T04:53:35.485 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.485 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.485 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "waiting for rw locks", 2024-02-05T04:53:35.485 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.160718+0000", 2024-02-05T04:53:35.485 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.8591999999999996e-05 2024-02-05T04:53:35.485 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.485 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.485 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:35.485 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.162682+0000", 2024-02-05T04:53:35.485 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.001964155 2024-02-05T04:53:35.485 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.485 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.486 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:53:35.486 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.162771+0000", 2024-02-05T04:53:35.486 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.8739999999999994e-05 2024-02-05T04:53:35.486 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.486 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.486 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:35.486 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.163697+0000", 2024-02-05T04:53:35.486 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00092623499999999997 2024-02-05T04:53:35.486 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:35.486 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:35.486 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:35.486 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.486 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.486 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:278 3.4s0 3:25248d28:::smithi19639362-18:head [create,setxattr Zk\\P^SegWYU[`WTPeUNnQmiTaa_jWqlbUlOOo_`Q (807) in=40b,setxattr WQnfRYk\\Wee]jkWpPVU\\b\\^O`d_aqhhsrNa\\q]` (1683) in=39b,setxattr isjYpf]SpaTmbRX[lPfZYd`iRg^qpPtXgta\\YSeP (1578) in=40b,setxattr Rj`[raddO^W[eaYZe\\p]n (1683) in=21b,setxattr \\b (1683) in=2b,setxattr SUU_aQmekRQTrtncbjnO]YVtnYUiSmRYkZdfVWjn (9612) in=40b,setxattr htY\\NVO]cOXO_S_laj[Tb^UlYeRVeTdgtRihsQNV (1717) in=40b,setxattr cNlYV\\rZSe_tflYeOk_TblX`eVOcfgno_ioWgrZY (875) in=40b,setxattr qS]Oimh`cRqmTYb[qlU[TqXhleWb\\ (1683) in=29b,setxattr NUkmOR`_Ui`rla]jt_Ql[Tfra[NmmgYhj_PTghpt (911) in=40b,setxattr e[kalNoWQnaXoX_kOZap_kXQahW]d[gPb_OgoTtr (1542) in=40b,setxattr rXW]_VkTh]^hcoaOsfUSdi_i[q\\oYfihbhS\\ooV] (210) in=40b,setxattr Na\\ch (1683) in=5b,setxattr f`WaS^^g\\Zc]]\\fgOeYOa_cN_^[lUa[YT`aNlZep (876) in=40b,setxattr ][ebaQ\\NOOfnlZQmROWj_mXld\\XTdfe^bZ (1683) in=34b,setxattr [\\hq^][jRsSfi\\SoVV\\VfOrrdbanYccVh`b`Pmhh (1789) in=40b,setxattr _Zkp`Xt_VWpQVOPk[tlZaRs`[rbSgWjNXP[k\\_O] (15648) in=40b,setxattr \\PnkdXaP]^o_orX[hlrOSrRRPVfReZparfiseVeR (13964) in=40b,setxattr PS]d`lagRXOkQsXaRYVtOYmbonZdcZtmTbXtThNa (1367) in=40b,setxattr VijqchiV`trXRPoVjQV^^XR\\qPPRNWct[RN]sgse (491) in=40b] snapc 3=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-02-05T04:53:35.486 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:30.194977+0000", 2024-02-05T04:53:35.487 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 5.2578141800000004, 2024-02-05T04:53:35.487 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0050184410000000002, 2024-02-05T04:53:35.487 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:35.487 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:35.487 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:53:35.487 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:53:35.487 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:35.487 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 278 2024-02-05T04:53:35.487 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.487 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:35.487 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.487 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:35.487 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.194977+0000", 2024-02-05T04:53:35.487 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:35.487 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.487 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.488 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:35.488 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.194977+0000", 2024-02-05T04:53:35.488 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:35.488 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.488 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.488 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:35.488 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.194978+0000", 2024-02-05T04:53:35.488 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.5999999999999999e-06 2024-02-05T04:53:35.488 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.488 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.488 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:35.488 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.195135+0000", 2024-02-05T04:53:35.488 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.000156253 2024-02-05T04:53:35.488 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.488 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.489 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:35.489 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.195151+0000", 2024-02-05T04:53:35.489 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.6273000000000002e-05 2024-02-05T04:53:35.489 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.489 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.489 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:35.489 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.195159+0000", 2024-02-05T04:53:35.489 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.4659999999999999e-06 2024-02-05T04:53:35.489 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.489 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.489 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:35.489 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.195219+0000", 2024-02-05T04:53:35.489 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.9545e-05 2024-02-05T04:53:35.489 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.489 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.489 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:53:35.490 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.195345+0000", 2024-02-05T04:53:35.490 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00012637400000000001 2024-02-05T04:53:35.490 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.490 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.490 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:35.490 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.195978+0000", 2024-02-05T04:53:35.490 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00063293500000000005 2024-02-05T04:53:35.490 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.490 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.490 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:35.490 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.196930+0000", 2024-02-05T04:53:35.490 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00095194300000000002 2024-02-05T04:53:35.490 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.490 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.490 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "commit_sent", 2024-02-05T04:53:35.491 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.199944+0000", 2024-02-05T04:53:35.491 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0030134179999999999 2024-02-05T04:53:35.491 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.491 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.491 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:35.491 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.199995+0000", 2024-02-05T04:53:35.491 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.1634000000000001e-05 2024-02-05T04:53:35.491 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:35.491 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:35.491 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:35.491 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.491 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.491 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:284 3.4s0 3:264f0b4b:::smithi19639362-35:head [setxattr _header (56) in=7b] snapc 3=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-02-05T04:53:35.491 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:30.205620+0000", 2024-02-05T04:53:35.491 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 5.2471712249999998, 2024-02-05T04:53:35.492 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0067013009999999998, 2024-02-05T04:53:35.492 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:35.492 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:35.492 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:53:35.492 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:53:35.492 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:35.492 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 284 2024-02-05T04:53:35.492 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.492 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:35.492 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.492 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:35.492 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.205620+0000", 2024-02-05T04:53:35.492 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:35.492 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.492 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.493 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:35.493 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.205620+0000", 2024-02-05T04:53:35.493 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:35.493 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.493 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.493 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:35.493 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.205621+0000", 2024-02-05T04:53:35.493 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.8800000000000001e-07 2024-02-05T04:53:35.493 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.493 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.493 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:35.493 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.205626+0000", 2024-02-05T04:53:35.493 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.028e-06 2024-02-05T04:53:35.493 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.493 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.493 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:35.494 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.205634+0000", 2024-02-05T04:53:35.494 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.8820000000000001e-06 2024-02-05T04:53:35.494 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.494 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.494 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:35.494 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.205639+0000", 2024-02-05T04:53:35.494 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.6670000000000001e-06 2024-02-05T04:53:35.494 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.494 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.494 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:35.494 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.207319+0000", 2024-02-05T04:53:35.494 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.001680029 2024-02-05T04:53:35.494 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.494 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.494 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:53:35.494 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.207416+0000", 2024-02-05T04:53:35.495 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.6576000000000005e-05 2024-02-05T04:53:35.495 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.495 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.495 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:35.495 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.207826+0000", 2024-02-05T04:53:35.495 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00041016499999999999 2024-02-05T04:53:35.495 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.495 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.495 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:35.495 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.208446+0000", 2024-02-05T04:53:35.495 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00061980300000000004 2024-02-05T04:53:35.495 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.495 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.495 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "commit_sent", 2024-02-05T04:53:35.495 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.212268+0000", 2024-02-05T04:53:35.496 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0038218890000000002 2024-02-05T04:53:35.496 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.496 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.496 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:35.496 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.212321+0000", 2024-02-05T04:53:35.496 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.3374000000000001e-05 2024-02-05T04:53:35.496 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:35.496 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:35.496 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:35.496 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.496 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.496 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:285 3.4s0 3:264f0b4b:::smithi19639362-35:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-02-05T04:53:35.496 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:30.205648+0000", 2024-02-05T04:53:35.496 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 5.2471430090000002, 2024-02-05T04:53:35.496 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0099691669999999993, 2024-02-05T04:53:35.496 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:35.497 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "started", 2024-02-05T04:53:35.497 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:53:35.497 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:53:35.497 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:35.497 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 285 2024-02-05T04:53:35.497 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.497 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:35.497 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.497 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:35.497 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.205648+0000", 2024-02-05T04:53:35.497 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:35.497 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.497 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.497 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:35.497 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.205648+0000", 2024-02-05T04:53:35.498 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:35.498 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.498 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.498 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:35.498 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.205648+0000", 2024-02-05T04:53:35.498 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.7100000000000002e-07 2024-02-05T04:53:35.498 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.498 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.498 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:35.498 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.205651+0000", 2024-02-05T04:53:35.498 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.277e-06 2024-02-05T04:53:35.498 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.498 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.498 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:35.498 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.205658+0000", 2024-02-05T04:53:35.498 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.1709999999999996e-06 2024-02-05T04:53:35.499 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.499 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.499 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:35.499 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.205663+0000", 2024-02-05T04:53:35.499 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.4009999999999996e-06 2024-02-05T04:53:35.499 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.499 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.499 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:35.499 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.208044+0000", 2024-02-05T04:53:35.499 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.002380957 2024-02-05T04:53:35.499 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.499 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.499 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "waiting for rw locks", 2024-02-05T04:53:35.499 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.208122+0000", 2024-02-05T04:53:35.499 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.7688999999999994e-05 2024-02-05T04:53:35.499 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.500 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.500 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:35.500 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.212367+0000", 2024-02-05T04:53:35.500 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0042449569999999997 2024-02-05T04:53:35.500 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.500 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.500 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:53:35.500 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.212502+0000", 2024-02-05T04:53:35.500 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00013503199999999999 2024-02-05T04:53:35.500 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.500 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.500 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:35.500 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.215617+0000", 2024-02-05T04:53:35.500 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0031151120000000002 2024-02-05T04:53:35.500 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:35.501 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:35.501 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:35.501 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.501 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.501 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:286 3.4s0 3:2c15cb2e:::smithi19639362-6:head [append 0~802816] snapc 3=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-02-05T04:53:35.501 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:30.209949+0000", 2024-02-05T04:53:35.501 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 5.2428422279999998, 2024-02-05T04:53:35.501 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.005149219, 2024-02-05T04:53:35.501 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:35.501 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:35.501 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:53:35.501 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:53:35.501 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:35.501 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 286 2024-02-05T04:53:35.501 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.501 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:35.502 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.502 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:35.502 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.209949+0000", 2024-02-05T04:53:35.502 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:35.502 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.502 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.502 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:35.502 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.209949+0000", 2024-02-05T04:53:35.502 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:35.502 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.502 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.502 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:35.502 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.209949+0000", 2024-02-05T04:53:35.502 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.5499999999999998e-07 2024-02-05T04:53:35.502 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.502 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.503 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:35.503 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.210754+0000", 2024-02-05T04:53:35.503 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00080471000000000004 2024-02-05T04:53:35.503 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.503 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.503 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:35.503 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.210763+0000", 2024-02-05T04:53:35.503 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.7530000000000007e-06 2024-02-05T04:53:35.503 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.503 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.503 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:35.503 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.210769+0000", 2024-02-05T04:53:35.503 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.8969999999999996e-06 2024-02-05T04:53:35.503 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.503 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.504 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:35.504 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.210846+0000", 2024-02-05T04:53:35.504 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.7012000000000001e-05 2024-02-05T04:53:35.504 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.504 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.504 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:53:35.504 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.210929+0000", 2024-02-05T04:53:35.504 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.3238999999999994e-05 2024-02-05T04:53:35.504 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.504 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.504 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:35.504 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.211572+0000", 2024-02-05T04:53:35.504 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00064284499999999996 2024-02-05T04:53:35.504 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.505 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.505 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:35.505 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.214764+0000", 2024-02-05T04:53:35.505 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.003192299 2024-02-05T04:53:35.505 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.505 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.505 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "commit_sent", 2024-02-05T04:53:35.505 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.215058+0000", 2024-02-05T04:53:35.505 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00029368999999999999 2024-02-05T04:53:35.505 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.505 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.505 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:35.505 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.215098+0000", 2024-02-05T04:53:35.505 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.0119e-05 2024-02-05T04:53:35.505 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:35.506 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:35.506 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:35.506 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.506 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.506 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:288 3.4s0 3:2c15cb2e:::smithi19639362-6:head [setxattr _header (55) in=7b] snapc 3=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-02-05T04:53:35.506 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:30.212734+0000", 2024-02-05T04:53:35.506 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 5.2400565160000001, 2024-02-05T04:53:35.506 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0048514259999999998, 2024-02-05T04:53:35.506 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:35.506 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:35.506 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:53:35.506 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:53:35.506 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:35.506 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 288 2024-02-05T04:53:35.506 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.507 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:35.507 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.507 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:35.507 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.212734+0000", 2024-02-05T04:53:35.507 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:35.507 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.507 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.507 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:35.507 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.212734+0000", 2024-02-05T04:53:35.507 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:35.507 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.507 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.507 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:35.507 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.212735+0000", 2024-02-05T04:53:35.507 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.3900000000000004e-07 2024-02-05T04:53:35.507 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.507 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.508 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:35.508 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.212738+0000", 2024-02-05T04:53:35.508 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.2100000000000002e-06 2024-02-05T04:53:35.508 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.508 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.508 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:35.508 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.212745+0000", 2024-02-05T04:53:35.508 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.6340000000000003e-06 2024-02-05T04:53:35.508 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.508 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.508 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:35.508 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.212750+0000", 2024-02-05T04:53:35.509 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.096e-06 2024-02-05T04:53:35.509 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.509 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.509 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:35.509 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.214165+0000", 2024-02-05T04:53:35.509 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.001414654 2024-02-05T04:53:35.509 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.509 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.509 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:53:35.509 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.214258+0000", 2024-02-05T04:53:35.509 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.3455999999999994e-05 2024-02-05T04:53:35.509 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.509 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.509 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:35.510 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.214578+0000", 2024-02-05T04:53:35.510 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00032020200000000001 2024-02-05T04:53:35.510 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.510 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.510 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:35.510 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.216896+0000", 2024-02-05T04:53:35.510 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0023180340000000001 2024-02-05T04:53:35.510 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.510 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.510 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "commit_sent", 2024-02-05T04:53:35.510 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.217541+0000", 2024-02-05T04:53:35.510 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00064483300000000004 2024-02-05T04:53:35.510 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.510 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.510 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:35.511 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.217586+0000", 2024-02-05T04:53:35.511 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.4668e-05 2024-02-05T04:53:35.511 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:35.511 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:35.511 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:35.511 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.511 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.511 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:289 3.4s0 3:2c15cb2e:::smithi19639362-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-02-05T04:53:35.511 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:30.212758+0000", 2024-02-05T04:53:35.511 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 5.2400333940000001, 2024-02-05T04:53:35.511 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0060231160000000002, 2024-02-05T04:53:35.511 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:35.511 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "started", 2024-02-05T04:53:35.511 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:53:35.511 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:53:35.512 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:35.512 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 289 2024-02-05T04:53:35.512 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.512 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:35.512 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.512 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:35.512 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.212758+0000", 2024-02-05T04:53:35.512 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:35.512 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.512 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.512 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:35.512 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.212758+0000", 2024-02-05T04:53:35.512 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:35.512 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.512 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.513 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:35.513 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.212758+0000", 2024-02-05T04:53:35.513 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.0400000000000002e-07 2024-02-05T04:53:35.513 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.513 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.513 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:35.513 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.212760+0000", 2024-02-05T04:53:35.513 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.8500000000000001e-06 2024-02-05T04:53:35.513 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.513 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.513 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:35.513 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.212765+0000", 2024-02-05T04:53:35.513 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.4920000000000001e-06 2024-02-05T04:53:35.513 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.513 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.514 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:35.514 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.212768+0000", 2024-02-05T04:53:35.514 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.112e-06 2024-02-05T04:53:35.514 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.514 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.514 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:35.514 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.215646+0000", 2024-02-05T04:53:35.514 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.002877395 2024-02-05T04:53:35.514 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.514 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.514 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "waiting for rw locks", 2024-02-05T04:53:35.514 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.215694+0000", 2024-02-05T04:53:35.514 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.7831999999999999e-05 2024-02-05T04:53:35.514 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.514 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.515 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:35.515 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.217839+0000", 2024-02-05T04:53:35.515 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0021454450000000002 2024-02-05T04:53:35.515 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.515 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.515 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:53:35.515 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.217915+0000", 2024-02-05T04:53:35.515 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.6347000000000006e-05 2024-02-05T04:53:35.515 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.515 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.515 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:35.515 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.218781+0000", 2024-02-05T04:53:35.515 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00086523899999999996 2024-02-05T04:53:35.515 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:35.515 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:35.516 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:35.516 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.516 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.516 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:393 3.4s0 3:2c242b83:::smithi19639362-9:head [append 0~770048] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-02-05T04:53:35.516 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:34.186553+0000", 2024-02-05T04:53:35.516 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 1.2662375479999999, 2024-02-05T04:53:35.516 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0062969649999999999, 2024-02-05T04:53:35.516 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:35.516 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:35.516 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:53:35.516 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:53:35.516 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:35.516 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 393 2024-02-05T04:53:35.516 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.516 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:35.517 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.517 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:35.517 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.186553+0000", 2024-02-05T04:53:35.517 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:35.517 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.517 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.517 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:35.517 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.186553+0000", 2024-02-05T04:53:35.517 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:35.517 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.517 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.517 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:35.517 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.186555+0000", 2024-02-05T04:53:35.517 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.5960000000000001e-06 2024-02-05T04:53:35.517 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.518 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.518 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:35.518 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.187770+0000", 2024-02-05T04:53:35.518 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0012145389999999999 2024-02-05T04:53:35.518 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.518 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.518 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:35.518 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.187786+0000", 2024-02-05T04:53:35.518 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.6719e-05 2024-02-05T04:53:35.518 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.518 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.518 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:35.518 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.187795+0000", 2024-02-05T04:53:35.518 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.1649999999999995e-06 2024-02-05T04:53:35.518 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.519 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.519 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:35.519 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.187861+0000", 2024-02-05T04:53:35.519 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.5399000000000002e-05 2024-02-05T04:53:35.519 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.519 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.519 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:53:35.519 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.187956+0000", 2024-02-05T04:53:35.519 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.4853999999999997e-05 2024-02-05T04:53:35.519 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.519 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.519 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:35.519 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.188717+0000", 2024-02-05T04:53:35.519 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00076123399999999996 2024-02-05T04:53:35.519 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.520 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.520 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:35.520 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.191130+0000", 2024-02-05T04:53:35.520 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0024133660000000001 2024-02-05T04:53:35.520 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.520 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.520 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "commit_sent", 2024-02-05T04:53:35.520 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.192780+0000", 2024-02-05T04:53:35.520 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0016500320000000001 2024-02-05T04:53:35.520 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.520 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.520 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:35.520 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.192850+0000", 2024-02-05T04:53:35.520 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.0061000000000006e-05 2024-02-05T04:53:35.520 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:35.521 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:35.521 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:35.521 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.521 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.521 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:394 3.4s0 3:2c242b83:::smithi19639362-9:head [setxattr _header (55) in=7b] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-02-05T04:53:35.521 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:34.187824+0000", 2024-02-05T04:53:35.521 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 1.264966686, 2024-02-05T04:53:35.521 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0063472709999999998, 2024-02-05T04:53:35.521 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:35.521 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:35.521 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:53:35.521 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:53:35.521 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:35.521 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 394 2024-02-05T04:53:35.521 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.521 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:35.522 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.522 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:35.522 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.187824+0000", 2024-02-05T04:53:35.522 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:35.522 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.522 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.522 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:35.522 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.187824+0000", 2024-02-05T04:53:35.522 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:35.522 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.522 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.522 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:35.522 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.187826+0000", 2024-02-05T04:53:35.522 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.342e-06 2024-02-05T04:53:35.522 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.522 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.523 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:35.523 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.187832+0000", 2024-02-05T04:53:35.523 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.2040000000000002e-06 2024-02-05T04:53:35.523 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.523 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.523 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:35.523 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.187841+0000", 2024-02-05T04:53:35.523 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.0529999999999996e-06 2024-02-05T04:53:35.523 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.523 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.523 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:35.523 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.187847+0000", 2024-02-05T04:53:35.523 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.3570000000000003e-06 2024-02-05T04:53:35.523 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.523 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.523 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:35.524 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.189963+0000", 2024-02-05T04:53:35.524 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0021155969999999999 2024-02-05T04:53:35.524 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.524 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.524 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:53:35.524 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.190114+0000", 2024-02-05T04:53:35.524 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00015165600000000001 2024-02-05T04:53:35.524 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.524 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.524 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:35.524 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.190493+0000", 2024-02-05T04:53:35.524 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00037816 2024-02-05T04:53:35.524 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.524 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.525 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:35.525 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.192865+0000", 2024-02-05T04:53:35.525 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0023725840000000001 2024-02-05T04:53:35.525 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.525 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.525 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "commit_sent", 2024-02-05T04:53:35.525 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.194107+0000", 2024-02-05T04:53:35.525 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.001242139 2024-02-05T04:53:35.525 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.525 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.525 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:35.525 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.194172+0000", 2024-02-05T04:53:35.525 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.4178999999999997e-05 2024-02-05T04:53:35.525 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:35.525 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:35.526 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:35.526 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.526 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.526 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:395 3.4s0 3:2c242b83:::smithi19639362-9:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-02-05T04:53:35.526 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:34.187868+0000", 2024-02-05T04:53:35.526 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 1.264923349, 2024-02-05T04:53:35.526 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0080281970000000008, 2024-02-05T04:53:35.526 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:35.526 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "started", 2024-02-05T04:53:35.526 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:53:35.526 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:53:35.526 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:35.526 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 395 2024-02-05T04:53:35.526 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.526 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:35.527 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.527 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:35.527 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.187868+0000", 2024-02-05T04:53:35.527 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:35.527 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.527 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.527 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:35.527 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.187868+0000", 2024-02-05T04:53:35.527 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:35.527 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.527 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.527 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:35.527 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.187868+0000", 2024-02-05T04:53:35.527 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.6499999999999999e-07 2024-02-05T04:53:35.527 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.527 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.528 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:35.528 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.187872+0000", 2024-02-05T04:53:35.528 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.9659999999999998e-06 2024-02-05T04:53:35.528 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.528 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.528 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:35.528 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.187881+0000", 2024-02-05T04:53:35.528 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.8519999999999993e-06 2024-02-05T04:53:35.528 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.528 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.528 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:35.528 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.187887+0000", 2024-02-05T04:53:35.528 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.3579999999999999e-06 2024-02-05T04:53:35.528 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.528 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.529 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:35.529 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.191261+0000", 2024-02-05T04:53:35.529 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.003373095 2024-02-05T04:53:35.529 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.529 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.529 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "waiting for rw locks", 2024-02-05T04:53:35.529 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.191336+0000", 2024-02-05T04:53:35.529 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.5731999999999999e-05 2024-02-05T04:53:35.529 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.529 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.529 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:35.529 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.194503+0000", 2024-02-05T04:53:35.529 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0031670069999999999 2024-02-05T04:53:35.529 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.530 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.530 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:53:35.530 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.194590+0000", 2024-02-05T04:53:35.530 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.6650000000000006e-05 2024-02-05T04:53:35.530 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.530 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.530 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:35.530 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.195896+0000", 2024-02-05T04:53:35.530 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.001305872 2024-02-05T04:53:35.530 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:35.530 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:35.530 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:35.530 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.530 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.530 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.6s4 43/31 ECSubWrite(tid=138, reqid=osd.13.0:138, at_version=43'24, trim_to=0'0, roll_forward_to=40'22))", 2024-02-05T04:53:35.531 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:35.172360+0000", 2024-02-05T04:53:35.531 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 0.28043114400000002, 2024-02-05T04:53:35.531 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00794804, 2024-02-05T04:53:35.531 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:35.531 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:35.531 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:35.531 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.531 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:35.531 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.172360+0000", 2024-02-05T04:53:35.531 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:35.531 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.531 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.531 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:35.531 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.172360+0000", 2024-02-05T04:53:35.531 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:35.532 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.532 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.532 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:35.532 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.172360+0000", 2024-02-05T04:53:35.532 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.9400000000000005e-07 2024-02-05T04:53:35.532 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.532 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.532 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:35.532 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.172384+0000", 2024-02-05T04:53:35.532 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.3186000000000001e-05 2024-02-05T04:53:35.532 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.532 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.532 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:35.532 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.172425+0000", 2024-02-05T04:53:35.532 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.1069000000000003e-05 2024-02-05T04:53:35.533 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.533 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.533 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:35.533 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.172435+0000", 2024-02-05T04:53:35.533 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.9150000000000008e-06 2024-02-05T04:53:35.533 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.533 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.533 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:35.533 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.172511+0000", 2024-02-05T04:53:35.533 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.6118000000000005e-05 2024-02-05T04:53:35.533 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.533 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.533 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:35.533 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.172521+0000", 2024-02-05T04:53:35.534 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.0429e-05 2024-02-05T04:53:35.534 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.534 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.534 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:35.534 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.180273+0000", 2024-02-05T04:53:35.534 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0077523109999999996 2024-02-05T04:53:35.534 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.534 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.534 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:35.534 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.180308+0000", 2024-02-05T04:53:35.534 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.4418000000000001e-05 2024-02-05T04:53:35.534 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:35.534 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:35.534 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:35.534 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.535 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.535 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.8s1 43/31 ECSubWrite(tid=62, reqid=osd.4.0:62, at_version=43'19, trim_to=0'0, roll_forward_to=42'17))", 2024-02-05T04:53:35.535 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:35.172380+0000", 2024-02-05T04:53:35.535 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 0.280411364, 2024-02-05T04:53:35.535 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0081089249999999995, 2024-02-05T04:53:35.535 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:35.535 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:35.535 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:35.535 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.535 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:35.535 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.172380+0000", 2024-02-05T04:53:35.535 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:35.535 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.535 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.535 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:35.535 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.172380+0000", 2024-02-05T04:53:35.536 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:35.536 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.536 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.536 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:35.536 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.172380+0000", 2024-02-05T04:53:35.536 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.48e-07 2024-02-05T04:53:35.536 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.536 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.536 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:35.536 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.172399+0000", 2024-02-05T04:53:35.536 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.9377000000000001e-05 2024-02-05T04:53:35.536 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.536 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.536 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:35.537 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.172407+0000", 2024-02-05T04:53:35.537 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.7729999999999999e-06 2024-02-05T04:53:35.537 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.537 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.537 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:35.537 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.172415+0000", 2024-02-05T04:53:35.537 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.8010000000000004e-06 2024-02-05T04:53:35.537 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.537 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.537 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:35.537 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.172494+0000", 2024-02-05T04:53:35.537 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.9208999999999993e-05 2024-02-05T04:53:35.537 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.537 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.537 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:35.537 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.172503+0000", 2024-02-05T04:53:35.537 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.1439999999999992e-06 2024-02-05T04:53:35.538 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.538 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.538 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:35.538 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.180452+0000", 2024-02-05T04:53:35.538 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0079488509999999998 2024-02-05T04:53:35.538 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.538 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.538 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:35.538 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.180488+0000", 2024-02-05T04:53:35.538 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.6622000000000003e-05 2024-02-05T04:53:35.538 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:35.538 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:35.538 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:35.538 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.538 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.539 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.3s2 43/31 ECSubWrite(tid=63, reqid=osd.4.0:63, at_version=43'27, trim_to=0'0, roll_forward_to=40'25))", 2024-02-05T04:53:35.539 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:35.173633+0000", 2024-02-05T04:53:35.539 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 0.279157977, 2024-02-05T04:53:35.539 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0074948189999999998, 2024-02-05T04:53:35.539 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:35.539 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:35.539 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:35.539 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.539 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:35.539 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.173633+0000", 2024-02-05T04:53:35.539 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:35.539 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.539 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.539 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:35.539 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.173633+0000", 2024-02-05T04:53:35.539 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:35.540 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.540 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.540 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:35.540 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.173634+0000", 2024-02-05T04:53:35.540 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.2099999999999996e-07 2024-02-05T04:53:35.540 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.540 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.540 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:35.540 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.173648+0000", 2024-02-05T04:53:35.540 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.3957999999999999e-05 2024-02-05T04:53:35.540 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.540 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.540 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:35.540 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.173657+0000", 2024-02-05T04:53:35.540 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.9289999999999993e-06 2024-02-05T04:53:35.541 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.541 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.541 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:35.541 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.173663+0000", 2024-02-05T04:53:35.541 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.2689999999999999e-06 2024-02-05T04:53:35.541 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.541 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.541 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:35.541 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.173709+0000", 2024-02-05T04:53:35.541 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.6238e-05 2024-02-05T04:53:35.541 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.541 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.541 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:35.541 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.173714+0000", 2024-02-05T04:53:35.541 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.4199999999999998e-06 2024-02-05T04:53:35.541 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.542 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.542 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:35.542 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.181098+0000", 2024-02-05T04:53:35.542 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0073840570000000003 2024-02-05T04:53:35.542 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.542 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.542 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:35.542 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.181128+0000", 2024-02-05T04:53:35.542 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.9326999999999999e-05 2024-02-05T04:53:35.542 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:35.542 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:35.542 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:35.542 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.542 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.542 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.6s4 43/31 ECSubWrite(tid=141, reqid=osd.13.0:138, at_version=0'0, trim_to=0'0, roll_forward_to=43'24))", 2024-02-05T04:53:35.543 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:35.182634+0000", 2024-02-05T04:53:35.543 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 0.27015715299999998, 2024-02-05T04:53:35.543 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0062931330000000002, 2024-02-05T04:53:35.543 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:35.543 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:35.543 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:35.543 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.543 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:35.543 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.182634+0000", 2024-02-05T04:53:35.543 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:35.543 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.543 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.543 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:35.543 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.182634+0000", 2024-02-05T04:53:35.543 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:35.543 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.543 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.544 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:35.544 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.182634+0000", 2024-02-05T04:53:35.544 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.0999999999999998e-07 2024-02-05T04:53:35.544 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.544 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.544 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:35.544 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.182644+0000", 2024-02-05T04:53:35.544 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.5170000000000005e-06 2024-02-05T04:53:35.544 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.544 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.544 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:35.544 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.182656+0000", 2024-02-05T04:53:35.544 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.1623999999999999e-05 2024-02-05T04:53:35.544 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.544 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.545 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:35.545 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.182663+0000", 2024-02-05T04:53:35.545 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.5870000000000004e-06 2024-02-05T04:53:35.545 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.545 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.545 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:35.545 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.182729+0000", 2024-02-05T04:53:35.545 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.5804999999999998e-05 2024-02-05T04:53:35.545 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.545 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.545 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:35.545 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.182742+0000", 2024-02-05T04:53:35.545 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.2612000000000001e-05 2024-02-05T04:53:35.545 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.545 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.546 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:35.546 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.188886+0000", 2024-02-05T04:53:35.546 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0061443640000000002 2024-02-05T04:53:35.546 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:35.546 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:35.546 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:35.546 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.188927+0000", 2024-02-05T04:53:35.546 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.1013999999999997e-05 2024-02-05T04:53:35.546 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:35.546 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:35.546 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:35.546 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:35.546 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:35.546 INFO:teuthology.orchestra.run.smithi091.stdout:} 2024-02-05T04:53:35.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-02-05T04:53:35.538+0000 7f1de1564640 0 --2- 172.21.15.62:0/3763520571 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1ddc0cf890 0x7f1ddc149210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-02-05T04:53:35.548 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-02-05T04:53:35.549 INFO:tasks.ceph.osd.1.smithi091.stderr:2024-02-05T04:53:35.546+0000 7ff9dc283640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34275) UID: 0 2024-02-05T04:53:35.648 INFO:tasks.ceph.osd.11.smithi196.stderr:2024-02-05T04:53:35.647+0000 7f85a8d86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 34609) UID: 0 2024-02-05T04:53:35.681 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T04:53:35.693 INFO:teuthology.orchestra.run.smithi091.stdout:{ 2024-02-05T04:53:35.693 INFO:teuthology.orchestra.run.smithi091.stdout: "ops": [], 2024-02-05T04:53:35.693 INFO:teuthology.orchestra.run.smithi091.stdout: "num_ops": 0 2024-02-05T04:53:35.694 INFO:teuthology.orchestra.run.smithi091.stdout:} 2024-02-05T04:53:35.702 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-02-05T04:53:35.749 INFO:tasks.ceph.osd.9.smithi091.stderr:2024-02-05T04:53:35.748+0000 7efc18827640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 34279) UID: 0 2024-02-05T04:53:35.805 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0: {} 2024-02-05T04:53:35.805 INFO:teuthology.orchestra.run.smithi062.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T04:53:35.808 INFO:teuthology.orchestra.run.smithi091.stdout:{ 2024-02-05T04:53:35.808 INFO:teuthology.orchestra.run.smithi091.stdout: "ops": [], 2024-02-05T04:53:35.808 INFO:teuthology.orchestra.run.smithi091.stdout: "complaint_time": 30, 2024-02-05T04:53:35.808 INFO:teuthology.orchestra.run.smithi091.stdout: "num_blocked_ops": 0 2024-02-05T04:53:35.808 INFO:teuthology.orchestra.run.smithi091.stdout:} 2024-02-05T04:53:35.817 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-02-05T04:53:35.825 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1: {} 2024-02-05T04:53:35.825 INFO:teuthology.orchestra.run.smithi062.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T04:53:35.843 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2: {} 2024-02-05T04:53:35.844 INFO:teuthology.orchestra.run.smithi062.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T04:53:35.850 INFO:tasks.ceph.osd.4.smithi062.stderr:2024-02-05T04:53:35.849+0000 7f708c308640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34689) UID: 0 2024-02-05T04:53:35.861 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3: {} 2024-02-05T04:53:35.862 INFO:teuthology.orchestra.run.smithi062.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T04:53:35.878 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4: {} 2024-02-05T04:53:35.878 INFO:teuthology.orchestra.run.smithi062.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T04:53:35.896 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5: {} 2024-02-05T04:53:35.896 INFO:teuthology.orchestra.run.smithi062.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T04:53:35.914 INFO:teuthology.orchestra.run.smithi062.stdout:osd.6: {} 2024-02-05T04:53:35.914 INFO:teuthology.orchestra.run.smithi062.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T04:53:35.933 INFO:teuthology.orchestra.run.smithi062.stdout:osd.7: {} 2024-02-05T04:53:35.933 INFO:teuthology.orchestra.run.smithi062.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T04:53:35.950 INFO:teuthology.orchestra.run.smithi062.stdout:osd.8: {} 2024-02-05T04:53:35.950 INFO:teuthology.orchestra.run.smithi062.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-05T04:53:35.951 INFO:tasks.ceph.osd.13.smithi091.stderr:2024-02-05T04:53:35.949+0000 7f526fd4d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 34281) UID: 0 2024-02-05T04:53:35.962 INFO:teuthology.orchestra.run.smithi091.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:35.962 INFO:teuthology.orchestra.run.smithi091.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:35.967 INFO:teuthology.orchestra.run.smithi062.stdout:osd.9: {} 2024-02-05T04:53:35.967 INFO:teuthology.orchestra.run.smithi062.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-05T04:53:35.972 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-02-05T04:53:35.985 INFO:teuthology.orchestra.run.smithi062.stdout:osd.10: {} 2024-02-05T04:53:35.986 INFO:teuthology.orchestra.run.smithi062.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-05T04:53:36.003 INFO:teuthology.orchestra.run.smithi062.stdout:osd.11: {} 2024-02-05T04:53:36.004 INFO:teuthology.orchestra.run.smithi062.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-05T04:53:36.021 INFO:teuthology.orchestra.run.smithi062.stdout:osd.12: {} 2024-02-05T04:53:36.021 INFO:teuthology.orchestra.run.smithi062.stderr:osd.12: osd_enable_op_tracker = '' 2024-02-05T04:53:36.038 INFO:teuthology.orchestra.run.smithi062.stdout:osd.13: {} 2024-02-05T04:53:36.039 INFO:teuthology.orchestra.run.smithi062.stderr:osd.13: osd_enable_op_tracker = '' 2024-02-05T04:53:36.055 INFO:tasks.ceph.osd.1.smithi091.stderr:2024-02-05T04:53:36.054+0000 7ff9dc283640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34275) UID: 0 2024-02-05T04:53:36.057 INFO:teuthology.orchestra.run.smithi062.stdout:osd.14: {} 2024-02-05T04:53:36.057 INFO:teuthology.orchestra.run.smithi062.stderr:osd.14: osd_enable_op_tracker = '' 2024-02-05T04:53:36.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-02-05T04:53:36.058+0000 7fe3f37fe640 0 -- 172.21.15.62:0/2867495498 >> [v2:172.21.15.196:6826/3187639683,v1:172.21.15.196:6827/3187639683] conn(0x7fe3f421fdf0 msgr2=0x7fe3f42221e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-02-05T04:53:36.076 INFO:teuthology.orchestra.run.smithi062.stdout:osd.15: {} 2024-02-05T04:53:36.076 INFO:teuthology.orchestra.run.smithi062.stderr:osd.15: osd_enable_op_tracker = '' 2024-02-05T04:53:36.118 INFO:teuthology.orchestra.run.smithi091.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:36.118 INFO:teuthology.orchestra.run.smithi091.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:36.127 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-02-05T04:53:36.152 INFO:tasks.ceph.osd.10.smithi150.stderr:2024-02-05T04:53:36.151+0000 7f3075a4d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 34018) UID: 0 2024-02-05T04:53:36.165 INFO:teuthology.orchestra.run.smithi062.stderr:nodeep-scrub is set 2024-02-05T04:53:36.166 INFO:teuthology.orchestra.run.smithi062.stderr:reweighted osd.15 to 1 (10000) 2024-02-05T04:53:36.174 INFO:tasks.rados.rados.0.smithi196.stdout:149: done (1 left) 2024-02-05T04:53:36.174 INFO:tasks.rados.rados.0.smithi196.stdout:150: snap_remove snap 4 2024-02-05T04:53:36.186 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-02-05T04:53:36.186 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-05T04:53:36.271 INFO:tasks.ceph.osd.4.smithi062.stderr:2024-02-05T04:53:36.270+0000 7f708c308640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34689) UID: 0 2024-02-05T04:53:36.279 INFO:teuthology.orchestra.run.smithi091.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:36.279 INFO:teuthology.orchestra.run.smithi091.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:36.287 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-02-05T04:53:36.354 INFO:tasks.ceph.osd.6.smithi150.stderr:2024-02-05T04:53:36.353+0000 7f6bea9f0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34015) UID: 0 2024-02-05T04:53:36.393 INFO:teuthology.orchestra.run.smithi091.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:36.393 INFO:teuthology.orchestra.run.smithi091.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:36.401 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-02-05T04:53:36.454 INFO:tasks.ceph.osd.7.smithi196.stderr:2024-02-05T04:53:36.453+0000 7fb281419640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34606) UID: 0 2024-02-05T04:53:36.508 INFO:teuthology.orchestra.run.smithi091.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:36.508 INFO:teuthology.orchestra.run.smithi091.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:36.516 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-02-05T04:53:36.519 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:53:36.519 INFO:teuthology.orchestra.run.smithi062.stderr:dumped all 2024-02-05T04:53:36.531 INFO:teuthology.orchestra.run.smithi062.stdout:{"pg_ready":true,"pg_map":{"version":46,"stamp":"2024-02-05T04:53:36.166793+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":47464992,"num_objects":60,"num_object_clones":12,"num_object_copies":410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":335,"num_read_kb":24721,"num_write":473,"num_write_kb":53044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":432,"ondisk_log_size":432,"up":130,"acting":130,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":130,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":525372,"kb_used_data":88832,"kb_used_omap":24,"kb_used_meta":436263,"kb_avail":1509424068,"statfs":{"total":1546188226560,"available":1545650245632,"internally_reserved":0,"allocated":90963968,"data_stored":102094606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25520,"internal_metadata":446733392},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19464192,"num_objects":8,"num_object_clones":10,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":60,"num_read_kb":12874,"num_write":114,"num_write_kb":20752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.028395"},"pg_stats":[{"pgid":"3.f","version":"39'25","reported_seq":78,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-02-05T04:53:32.143864+0000","last_change":"2024-02-05T04:53:30.137756+0000","last_active":"2024-02-05T04:53:32.143864+0000","last_peered":"2024-02-05T04:53:32.143864+0000","last_clean":"2024-02-05T04:53:32.143864+0000","last_became_active":"2024-02-05T04:53:24.723119+0000","last_became_peered":"2024-02-05T04:53:24.723119+0000","last_unstale":"2024-02-05T04:53:32.143864+0000","last_undegraded":"2024-02-05T04:53:32.143864+0000","last_fullsized":"2024-02-05T04:53:32.143864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:23.066897+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:50.375007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064357800000000001,"stat_sum":{"num_bytes":5292032,"num_objects":8,"num_object_clones":1,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":2156,"num_write":24,"num_write_kb":5664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,14,4,0,10,1],"acting":[11,2,14,4,0,10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"3.e","version":"40'25","reported_seq":66,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-02-05T04:53:33.691007+0000","last_change":"2024-02-05T04:53:30.149178+0000","last_active":"2024-02-05T04:53:33.691007+0000","last_peered":"2024-02-05T04:53:33.691007+0000","last_clean":"2024-02-05T04:53:33.691007+0000","last_became_active":"2024-02-05T04:53:24.641093+0000","last_became_peered":"2024-02-05T04:53:24.641093+0000","last_unstale":"2024-02-05T04:53:33.691007+0000","last_undegraded":"2024-02-05T04:53:33.691007+0000","last_fullsized":"2024-02-05T04:53:33.691007+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:23.066897+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:42.302165+0000","scrub_duration":0,"objects_trimmed":3,"snaptrim_duration":0.016258147000000001,"stat_sum":{"num_bytes":3063808,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":36,"num_write_kb":3744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,8,11,13,6,5],"acting":[2,4,8,11,13,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"3.d","version":"38'22","reported_seq":72,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-02-05T04:53:33.155264+0000","last_change":"2024-02-05T04:53:30.142199+0000","last_active":"2024-02-05T04:53:33.155264+0000","last_peered":"2024-02-05T04:53:33.155264+0000","last_clean":"2024-02-05T04:53:33.155264+0000","last_became_active":"2024-02-05T04:53:24.723128+0000","last_became_peered":"2024-02-05T04:53:24.723128+0000","last_unstale":"2024-02-05T04:53:33.155264+0000","last_undegraded":"2024-02-05T04:53:33.155264+0000","last_fullsized":"2024-02-05T04:53:33.155264+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:23.066897+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:49.868795+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.0086066909999999996,"stat_sum":{"num_bytes":6062080,"num_objects":4,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":1880,"num_write":19,"num_write_kb":6320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,14,10,9,13,5],"acting":[7,3,14,10,9,13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"3.c","version":"38'5","reported_seq":46,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-02-05T04:53:32.143784+0000","last_change":"2024-02-05T04:53:30.137652+0000","last_active":"2024-02-05T04:53:32.143784+0000","last_peered":"2024-02-05T04:53:32.143784+0000","last_clean":"2024-02-05T04:53:32.143784+0000","last_became_active":"2024-02-05T04:53:24.723258+0000","last_became_peered":"2024-02-05T04:53:24.723258+0000","last_unstale":"2024-02-05T04:53:32.143784+0000","last_undegraded":"2024-02-05T04:53:32.143784+0000","last_fullsized":"2024-02-05T04:53:32.143784+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:23.066897+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:49.363891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072003400000000004,"stat_sum":{"num_bytes":1196032,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":195,"num_write":5,"num_write_kb":1168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,13,5,10,6,1,9],"acting":[11,13,5,10,6,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"3.b","version":"40'24","reported_seq":82,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-02-05T04:53:32.222229+0000","last_change":"2024-02-05T04:53:30.139740+0000","last_active":"2024-02-05T04:53:32.222229+0000","last_peered":"2024-02-05T04:53:32.222229+0000","last_clean":"2024-02-05T04:53:32.222229+0000","last_became_active":"2024-02-05T04:53:24.486317+0000","last_became_peered":"2024-02-05T04:53:24.486317+0000","last_unstale":"2024-02-05T04:53:32.222229+0000","last_undegraded":"2024-02-05T04:53:32.222229+0000","last_fullsized":"2024-02-05T04:53:32.222229+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:23.066897+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:52.949763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080546700000000003,"stat_sum":{"num_bytes":6176768,"num_objects":8,"num_object_clones":2,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":35,"num_read_kb":7262,"num_write":22,"num_write_kb":6032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,1,8,0,6,7,12],"acting":[13,1,8,0,6,7,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"3.a","version":"0'0","reported_seq":34,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-02-05T04:53:31.155660+0000","last_change":"2024-02-05T04:53:30.136931+0000","last_active":"2024-02-05T04:53:31.155660+0000","last_peered":"2024-02-05T04:53:31.155660+0000","last_clean":"2024-02-05T04:53:31.155660+0000","last_became_active":"2024-02-05T04:53:24.488548+0000","last_became_peered":"2024-02-05T04:53:24.488548+0000","last_unstale":"2024-02-05T04:53:31.155660+0000","last_undegraded":"2024-02-05T04:53:31.155660+0000","last_fullsized":"2024-02-05T04:53:31.155660+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:23.066897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:40.240039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066884300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,8,9,14,6,11],"acting":[3,7,8,9,14,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"3.9","version":"34'11","reported_seq":56,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-02-05T04:53:32.162176+0000","last_change":"2024-02-05T04:53:30.144729+0000","last_active":"2024-02-05T04:53:32.162176+0000","last_peered":"2024-02-05T04:53:32.162176+0000","last_clean":"2024-02-05T04:53:32.162176+0000","last_became_active":"2024-02-05T04:53:24.722954+0000","last_became_peered":"2024-02-05T04:53:24.722954+0000","last_unstale":"2024-02-05T04:53:32.162176+0000","last_undegraded":"2024-02-05T04:53:32.162176+0000","last_fullsized":"2024-02-05T04:53:32.162176+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:23.066897+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:45.999739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052870900000000004,"stat_sum":{"num_bytes":2424832,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":13,"num_read_kb":3125,"num_write":11,"num_write_kb":3152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0,13,3,15,9,10],"acting":[14,0,13,3,15,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"3.8","version":"37'10","reported_seq":45,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-02-05T04:53:31.130630+0000","last_change":"2024-02-05T04:53:30.130865+0000","last_active":"2024-02-05T04:53:31.130630+0000","last_peered":"2024-02-05T04:53:31.130630+0000","last_clean":"2024-02-05T04:53:31.130630+0000","last_became_active":"2024-02-05T04:53:24.486354+0000","last_became_peered":"2024-02-05T04:53:24.486354+0000","last_unstale":"2024-02-05T04:53:31.130630+0000","last_undegraded":"2024-02-05T04:53:31.130630+0000","last_fullsized":"2024-02-05T04:53:31.130630+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:23.066897+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:41.034180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000602909,"stat_sum":{"num_bytes":573440,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":6,"num_write_kb":560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,0,14,15,6],"acting":[4,1,2,0,14,15,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"1.0","version":"26'157","reported_seq":223,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-02-05T04:53:32.161103+0000","last_change":"2024-02-05T04:53:09.625781+0000","last_active":"2024-02-05T04:53:32.161103+0000","last_peered":"2024-02-05T04:53:32.161103+0000","last_clean":"2024-02-05T04:53:32.161103+0000","last_became_active":"2024-02-05T04:53:09.624695+0000","last_became_peered":"2024-02-05T04:53:09.624695+0000","last_unstale":"2024-02-05T04:53:32.161103+0000","last_undegraded":"2024-02-05T04:53:32.161103+0000","last_fullsized":"2024-02-05T04:53:32.161103+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:07.900741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:07.900741+0000","last_clean_scrub_stamp":"2024-02-05T04:53:07.900741+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:23.258506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2],"acting":[8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":20,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-05T04:53:30.130395+0000","last_change":"2024-02-05T04:53:12.028466+0000","last_active":"2024-02-05T04:53:30.130395+0000","last_peered":"2024-02-05T04:53:30.130395+0000","last_clean":"2024-02-05T04:53:30.130395+0000","last_became_active":"2024-02-05T04:53:12.028138+0000","last_became_peered":"2024-02-05T04:53:12.028138+0000","last_unstale":"2024-02-05T04:53:30.130395+0000","last_undegraded":"2024-02-05T04:53:30.130395+0000","last_fullsized":"2024-02-05T04:53:30.130395+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_clean_scrub_stamp":"2024-02-05T04:53:11.017155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:38.427375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,15],"acting":[4,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"39'33","reported_seq":75,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-02-05T04:53:33.157166+0000","last_change":"2024-02-05T04:53:30.160725+0000","last_active":"2024-02-05T04:53:33.157166+0000","last_peered":"2024-02-05T04:53:33.157166+0000","last_clean":"2024-02-05T04:53:33.157166+0000","last_became_active":"2024-02-05T04:53:24.486494+0000","last_became_peered":"2024-02-05T04:53:24.486494+0000","last_unstale":"2024-02-05T04:53:33.157166+0000","last_undegraded":"2024-02-05T04:53:33.157166+0000","last_fullsized":"2024-02-05T04:53:33.157166+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:23.066897+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:29.148681+0000","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.018221533000000002,"stat_sum":{"num_bytes":3440640,"num_objects":5,"num_object_clones":2,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":4856,"num_write":20,"num_write_kb":3408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,2,3,6,15,8,11],"acting":[12,2,3,6,15,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-05T04:53:30.136078+0000","last_change":"2024-02-05T04:53:12.040500+0000","last_active":"2024-02-05T04:53:30.136078+0000","last_peered":"2024-02-05T04:53:30.136078+0000","last_clean":"2024-02-05T04:53:30.136078+0000","last_became_active":"2024-02-05T04:53:12.040297+0000","last_became_peered":"2024-02-05T04:53:12.040297+0000","last_unstale":"2024-02-05T04:53:30.136078+0000","last_undegraded":"2024-02-05T04:53:30.136078+0000","last_fullsized":"2024-02-05T04:53:30.136078+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_clean_scrub_stamp":"2024-02-05T04:53:11.017155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:20.053075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"40'25","reported_seq":64,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-02-05T04:53:32.197654+0000","last_change":"2024-02-05T04:53:30.149067+0000","last_active":"2024-02-05T04:53:32.197654+0000","last_peered":"2024-02-05T04:53:32.197654+0000","last_clean":"2024-02-05T04:53:32.197654+0000","last_became_active":"2024-02-05T04:53:24.723508+0000","last_became_peered":"2024-02-05T04:53:24.723508+0000","last_unstale":"2024-02-05T04:53:32.197654+0000","last_undegraded":"2024-02-05T04:53:32.197654+0000","last_fullsized":"2024-02-05T04:53:32.197654+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:23.066897+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:40.903038+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.010166428999999999,"stat_sum":{"num_bytes":2932736,"num_objects":5,"num_object_clones":2,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":5,"num_write":18,"num_write_kb":3216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,11,4,8,10,2,14],"acting":[13,11,4,8,10,2,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"2.1","version":"0'0","reported_seq":24,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-02-05T04:53:32.161171+0000","last_change":"2024-02-05T04:53:12.033593+0000","last_active":"2024-02-05T04:53:32.161171+0000","last_peered":"2024-02-05T04:53:32.161171+0000","last_clean":"2024-02-05T04:53:32.161171+0000","last_became_active":"2024-02-05T04:53:12.033337+0000","last_became_peered":"2024-02-05T04:53:12.033337+0000","last_unstale":"2024-02-05T04:53:32.161171+0000","last_undegraded":"2024-02-05T04:53:32.161171+0000","last_fullsized":"2024-02-05T04:53:32.161171+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_clean_scrub_stamp":"2024-02-05T04:53:11.017155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:14.948579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4],"acting":[8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.0","version":"34'5","reported_seq":44,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-02-05T04:53:32.161705+0000","last_change":"2024-02-05T04:53:30.144407+0000","last_active":"2024-02-05T04:53:32.161705+0000","last_peered":"2024-02-05T04:53:32.161705+0000","last_clean":"2024-02-05T04:53:32.161705+0000","last_became_active":"2024-02-05T04:53:24.485082+0000","last_became_peered":"2024-02-05T04:53:24.485082+0000","last_unstale":"2024-02-05T04:53:32.161705+0000","last_undegraded":"2024-02-05T04:53:32.161705+0000","last_fullsized":"2024-02-05T04:53:32.161705+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:23.066897+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:33.698562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074980399999999997,"stat_sum":{"num_bytes":114688,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":5,"num_write_kb":432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,2,15,0,12,8,9],"acting":[14,2,15,0,12,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"2.2","version":"0'0","reported_seq":20,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-05T04:53:30.136259+0000","last_change":"2024-02-05T04:53:12.035901+0000","last_active":"2024-02-05T04:53:30.136259+0000","last_peered":"2024-02-05T04:53:30.136259+0000","last_clean":"2024-02-05T04:53:30.136259+0000","last_became_active":"2024-02-05T04:53:12.035739+0000","last_became_peered":"2024-02-05T04:53:12.035739+0000","last_unstale":"2024-02-05T04:53:30.136259+0000","last_undegraded":"2024-02-05T04:53:30.136259+0000","last_fullsized":"2024-02-05T04:53:30.136259+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_clean_scrub_stamp":"2024-02-05T04:53:11.017155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:14.062467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9],"acting":[3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"39'20","reported_seq":59,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-02-05T04:53:31.187602+0000","last_change":"2024-02-05T04:53:30.139074+0000","last_active":"2024-02-05T04:53:31.187602+0000","last_peered":"2024-02-05T04:53:31.187602+0000","last_clean":"2024-02-05T04:53:31.187602+0000","last_became_active":"2024-02-05T04:53:24.641280+0000","last_became_peered":"2024-02-05T04:53:24.641280+0000","last_unstale":"2024-02-05T04:53:31.187602+0000","last_undegraded":"2024-02-05T04:53:31.187602+0000","last_fullsized":"2024-02-05T04:53:31.187602+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:23.066897+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:50.543393+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.0087663459999999995,"stat_sum":{"num_bytes":1916928,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":12,"num_read_kb":1877,"num_write":12,"num_write_kb":2112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1,15,2,5,3],"acting":[4,7,1,15,2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"3.5","version":"39'17","reported_seq":60,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-02-05T04:53:32.160351+0000","last_change":"2024-02-05T04:53:30.129958+0000","last_active":"2024-02-05T04:53:32.160351+0000","last_peered":"2024-02-05T04:53:32.160351+0000","last_clean":"2024-02-05T04:53:32.160351+0000","last_became_active":"2024-02-05T04:53:24.643048+0000","last_became_peered":"2024-02-05T04:53:24.643048+0000","last_unstale":"2024-02-05T04:53:32.160351+0000","last_undegraded":"2024-02-05T04:53:32.160351+0000","last_fullsized":"2024-02-05T04:53:32.160351+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:23.066897+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:27.951270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045887599999999999,"stat_sum":{"num_bytes":4161536,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":1830,"num_write":13,"num_write_kb":4064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13,5,8,7,1,6],"acting":[15,13,5,8,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-02-05T04:53:31.139606+0000","last_change":"2024-02-05T04:53:12.032853+0000","last_active":"2024-02-05T04:53:31.139606+0000","last_peered":"2024-02-05T04:53:31.139606+0000","last_clean":"2024-02-05T04:53:31.139606+0000","last_became_active":"2024-02-05T04:53:12.032664+0000","last_became_peered":"2024-02-05T04:53:12.032664+0000","last_unstale":"2024-02-05T04:53:31.139606+0000","last_undegraded":"2024-02-05T04:53:31.139606+0000","last_fullsized":"2024-02-05T04:53:31.139606+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_clean_scrub_stamp":"2024-02-05T04:53:11.017155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:14.638056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"38'23","reported_seq":71,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-02-05T04:53:32.163546+0000","last_change":"2024-02-05T04:53:30.139523+0000","last_active":"2024-02-05T04:53:32.163546+0000","last_peered":"2024-02-05T04:53:32.163546+0000","last_clean":"2024-02-05T04:53:32.163546+0000","last_became_active":"2024-02-05T04:53:24.643568+0000","last_became_peered":"2024-02-05T04:53:24.643568+0000","last_unstale":"2024-02-05T04:53:32.163546+0000","last_undegraded":"2024-02-05T04:53:32.163546+0000","last_fullsized":"2024-02-05T04:53:32.163546+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:23.066897+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:37.484371+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.0064274600000000003,"stat_sum":{"num_bytes":5242880,"num_objects":5,"num_object_clones":0,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":17,"num_read_kb":873,"num_write":39,"num_write_kb":5120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5,8,9,4,13],"acting":[1,2,5,8,9,4,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"2.5","version":"0'0","reported_seq":21,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-05T04:53:30.135515+0000","last_change":"2024-02-05T04:53:12.753233+0000","last_active":"2024-02-05T04:53:30.135515+0000","last_peered":"2024-02-05T04:53:30.135515+0000","last_clean":"2024-02-05T04:53:30.135515+0000","last_became_active":"2024-02-05T04:53:12.752915+0000","last_became_peered":"2024-02-05T04:53:12.752915+0000","last_unstale":"2024-02-05T04:53:30.135515+0000","last_undegraded":"2024-02-05T04:53:30.135515+0000","last_fullsized":"2024-02-05T04:53:30.135515+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_clean_scrub_stamp":"2024-02-05T04:53:11.017155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:18.391150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"38'8","reported_seq":45,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-02-05T04:53:32.157158+0000","last_change":"2024-02-05T04:53:30.154505+0000","last_active":"2024-02-05T04:53:32.157158+0000","last_peered":"2024-02-05T04:53:32.157158+0000","last_clean":"2024-02-05T04:53:32.157158+0000","last_became_active":"2024-02-05T04:53:24.641453+0000","last_became_peered":"2024-02-05T04:53:24.641453+0000","last_unstale":"2024-02-05T04:53:32.157158+0000","last_undegraded":"2024-02-05T04:53:32.157158+0000","last_fullsized":"2024-02-05T04:53:32.157158+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:23.066897+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:29.454665+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.015508259999999999,"stat_sum":{"num_bytes":901120,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":24,"num_write_kb":880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,6,12,1,4,5,11],"acting":[13,6,12,1,4,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"2.6","version":"0'0","reported_seq":22,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-02-05T04:53:31.224237+0000","last_change":"2024-02-05T04:53:12.037950+0000","last_active":"2024-02-05T04:53:31.224237+0000","last_peered":"2024-02-05T04:53:31.224237+0000","last_clean":"2024-02-05T04:53:31.224237+0000","last_became_active":"2024-02-05T04:53:12.037750+0000","last_became_peered":"2024-02-05T04:53:12.037750+0000","last_unstale":"2024-02-05T04:53:31.224237+0000","last_undegraded":"2024-02-05T04:53:31.224237+0000","last_fullsized":"2024-02-05T04:53:31.224237+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_clean_scrub_stamp":"2024-02-05T04:53:11.017155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:26.129079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.6","version":"40'22","reported_seq":63,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-02-05T04:53:32.167908+0000","last_change":"2024-02-05T04:53:30.151858+0000","last_active":"2024-02-05T04:53:32.167908+0000","last_peered":"2024-02-05T04:53:32.167908+0000","last_clean":"2024-02-05T04:53:32.167908+0000","last_became_active":"2024-02-05T04:53:24.485438+0000","last_became_peered":"2024-02-05T04:53:24.485438+0000","last_unstale":"2024-02-05T04:53:32.167908+0000","last_undegraded":"2024-02-05T04:53:32.167908+0000","last_fullsized":"2024-02-05T04:53:32.167908+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:23.066897+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:42.314856+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.012872903999999999,"stat_sum":{"num_bytes":3375104,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":11,"num_read_kb":533,"num_write":16,"num_write_kb":3712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,12,4,15,1,7,8],"acting":[13,12,4,15,1,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"2.7","version":"0'0","reported_seq":21,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-02-05T04:53:31.130847+0000","last_change":"2024-02-05T04:53:12.781269+0000","last_active":"2024-02-05T04:53:31.130847+0000","last_peered":"2024-02-05T04:53:31.130847+0000","last_clean":"2024-02-05T04:53:31.130847+0000","last_became_active":"2024-02-05T04:53:12.780970+0000","last_became_peered":"2024-02-05T04:53:12.780970+0000","last_unstale":"2024-02-05T04:53:31.130847+0000","last_undegraded":"2024-02-05T04:53:31.130847+0000","last_fullsized":"2024-02-05T04:53:31.130847+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_clean_scrub_stamp":"2024-02-05T04:53:11.017155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:36.119048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,0],"acting":[15,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":46874624,"num_objects":58,"num_object_clones":12,"num_object_copies":406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":0,"num_read":199,"num_read_kb":24603,"num_write":270,"num_write_kb":49584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":83410944,"data_stored":99414016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":275,"ondisk_log_size":275,"up":112,"acting":112,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":12,"up_from":23,"seq":98784247816,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30944,"kb_used_data":3916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340896,"statfs":{"total":96636764160,"available":96605077504,"internally_reserved":0,"allocated":4009984,"data_stored":4846822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,8,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28899999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57499999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.252}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35199999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51700000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36499999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.313}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.377}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60199999999999998}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47499999999999998}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.443}]}]},{"osd":11,"up_from":23,"seq":98784247818,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31612,"kb_used_data":4588,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340228,"statfs":{"total":96636764160,"available":96604393472,"internally_reserved":0,"allocated":4698112,"data_stored":5795109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,4,5,6,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40500000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.312}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59399999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53200000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57599999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58699999999999997}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71999999999999997}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73399999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66000000000000003}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67200000000000004}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88800000000000001}]}]},{"osd":10,"up_from":23,"seq":98784247818,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31876,"kb_used_data":4852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339964,"statfs":{"total":96636764160,"available":96604123136,"internally_reserved":0,"allocated":4968448,"data_stored":5209381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36699999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32400000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60299999999999998}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.496}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41299999999999998}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68600000000000005}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59099999999999997}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46999999999999997}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57899999999999996}]}]},{"osd":3,"up_from":21,"seq":90194313225,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30868,"kb_used_data":3844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340972,"statfs":{"total":96636764160,"available":96605155328,"internally_reserved":0,"allocated":3936256,"data_stored":4511076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,4,6,7,8,9,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69399999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60399999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45800000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.622}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40699999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29799999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65400000000000003}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49299999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59699999999999998}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52900000000000003}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69799999999999995}]}]},{"osd":4,"up_from":21,"seq":90194313224,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32532,"kb_used_data":5512,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339308,"statfs":{"total":96636764160,"available":96603451392,"internally_reserved":0,"allocated":5644288,"data_stored":5727588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,9,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.245}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41599999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.31900000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52400000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53700000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.501}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33500000000000002}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64800000000000002}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60799999999999998}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.443}]}]},{"osd":5,"up_from":21,"seq":90194313224,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32668,"kb_used_data":5644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339172,"statfs":{"total":96636764160,"available":96603312128,"internally_reserved":0,"allocated":5779456,"data_stored":6006116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,6,7,9,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29099999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57999999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33800000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60899999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.378}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47499999999999998}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50800000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59299999999999997}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]}]},{"osd":6,"up_from":21,"seq":90194313225,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31592,"kb_used_data":4568,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340248,"statfs":{"total":96636764160,"available":96604413952,"internally_reserved":0,"allocated":4677632,"data_stored":5092708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,3,4,5,7,8,9,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59699999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42399999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41199999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52700000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39400000000000002}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36599999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57699999999999996}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48799999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47499999999999998}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55700000000000005}]}]},{"osd":1,"up_from":22,"seq":94489280521,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33872,"kb_used_data":6852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337968,"statfs":{"total":96636764160,"available":96602079232,"internally_reserved":0,"allocated":7016448,"data_stored":7259492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,4,5,8,9,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50900000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72799999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35799999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.626}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64100000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67800000000000005}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57199999999999995}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54900000000000004}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69299999999999995}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78300000000000003}]}]},{"osd":14,"up_from":22,"seq":94489280520,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32084,"kb_used_data":5060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339756,"statfs":{"total":96636764160,"available":96603910144,"internally_reserved":0,"allocated":5181440,"data_stored":5545253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.37}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63300000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.308}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54900000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39000000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68200000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34100000000000003}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40100000000000002}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.501}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29099999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48699999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51300000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52500000000000002}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53600000000000003}]}]},{"osd":9,"up_from":22,"seq":94489280520,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31176,"kb_used_data":4152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340664,"statfs":{"total":96636764160,"available":96604839936,"internally_reserved":0,"allocated":4251648,"data_stored":4001061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,3,4,5,6,8,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28499999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77500000000000002}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.753}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61199999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86699999999999999}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69799999999999995}]}]},{"osd":13,"up_from":22,"seq":94489280519,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36180,"kb_used_data":9156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94335660,"statfs":{"total":96636764160,"available":96599715840,"internally_reserved":0,"allocated":9375744,"data_stored":10710309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,4,5,6,7,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40600000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59799999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64200000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49299999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64200000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.622}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47699999999999998}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.505}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55200000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63300000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60999999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67100000000000004}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74099999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247817,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31416,"kb_used_data":4388,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340424,"statfs":{"total":96636764160,"available":96604594176,"internally_reserved":0,"allocated":4493312,"data_stored":4642022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41299999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.628}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.503}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57499999999999996}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67200000000000004}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68200000000000005}]}]},{"osd":15,"up_from":22,"seq":94489280521,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32072,"kb_used_data":5048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339768,"statfs":{"total":96636764160,"available":96603922432,"internally_reserved":0,"allocated":5169152,"data_stored":6491429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,4,5,6,7,8,9,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73699999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69399999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58899999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57899999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68400000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61199999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.628}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79800000000000004}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67100000000000004}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86299999999999999}]}]},{"osd":2,"up_from":23,"seq":98784247818,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38352,"kb_used_data":7172,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94333488,"statfs":{"total":96636764160,"available":96597491712,"internally_reserved":0,"allocated":7344128,"data_stored":9039685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,11,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56599999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39500000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28000000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58299999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30499999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40899999999999997}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.434}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60199999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44600000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64000000000000001}]}]},{"osd":7,"up_from":23,"seq":98784247818,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33128,"kb_used_data":6104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338712,"statfs":{"total":96636764160,"available":96602841088,"internally_reserved":0,"allocated":6250496,"data_stored":7224613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,2,3,5,6,8,9,10,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.378}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61099999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35299999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41199999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46400000000000002}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70099999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85199999999999998}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67600000000000005}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47799999999999998}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65300000000000002}]}]},{"osd":8,"up_from":23,"seq":98784247815,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35000,"kb_used_data":7976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94336840,"statfs":{"total":96636764160,"available":96600924160,"internally_reserved":0,"allocated":8167424,"data_stored":9991942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,4,6,7,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26300000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.318}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47699999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39500000000000002}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44700000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40500000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.628}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54300000000000004}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68999999999999995}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57299999999999995}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72499999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3985408,"data_stored":4390912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":7884800,"data_stored":9240576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6320128,"data_stored":8269824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3461120,"data_stored":4280320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":5169152,"data_stored":5496832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":5304320,"data_stored":5775360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4202496,"data_stored":4861952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":5758976,"data_stored":6983680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":7127040,"data_stored":9211904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":3760128,"data_stored":3760128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4476928,"data_stored":4968448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4206592,"data_stored":5554176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":3502080,"data_stored":4595712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":8884224,"data_stored":10469376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":4689920,"data_stored":5304320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":4677632,"data_stored":6250496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T04:53:36.533 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-05T04:53:36.562 INFO:tasks.ceph.osd.1.smithi091.stderr:2024-02-05T04:53:36.561+0000 7ff9dc283640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34275) UID: 0 2024-02-05T04:53:36.622 INFO:teuthology.orchestra.run.smithi091.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:36.623 INFO:teuthology.orchestra.run.smithi091.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:36.631 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-02-05T04:53:36.673 INFO:tasks.ceph.osd.13.smithi091.stderr:2024-02-05T04:53:36.671+0000 7f526fd4d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 34281) UID: 0 2024-02-05T04:53:36.756 INFO:tasks.ceph.osd.4.smithi062.stderr:2024-02-05T04:53:36.755+0000 7f708c308640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34689) UID: 0 2024-02-05T04:53:36.776 INFO:teuthology.orchestra.run.smithi091.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:36.776 INFO:teuthology.orchestra.run.smithi091.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:36.785 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-02-05T04:53:36.857 INFO:tasks.ceph.osd.8.smithi062.stderr:2024-02-05T04:53:36.856+0000 7f9385bf2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 34693) UID: 0 2024-02-05T04:53:36.863 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:53:36.864 INFO:teuthology.orchestra.run.smithi062.stderr:dumped all 2024-02-05T04:53:36.875 INFO:teuthology.orchestra.run.smithi062.stdout:{"pg_ready":true,"pg_map":{"version":46,"stamp":"2024-02-05T04:53:36.166793+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":47464992,"num_objects":60,"num_object_clones":12,"num_object_copies":410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":335,"num_read_kb":24721,"num_write":473,"num_write_kb":53044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":432,"ondisk_log_size":432,"up":130,"acting":130,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":130,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":525372,"kb_used_data":88832,"kb_used_omap":24,"kb_used_meta":436263,"kb_avail":1509424068,"statfs":{"total":1546188226560,"available":1545650245632,"internally_reserved":0,"allocated":90963968,"data_stored":102094606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25520,"internal_metadata":446733392},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19464192,"num_objects":8,"num_object_clones":10,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":60,"num_read_kb":12874,"num_write":114,"num_write_kb":20752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.028395"},"pg_stats":[{"pgid":"3.f","version":"39'25","reported_seq":78,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-02-05T04:53:32.143864+0000","last_change":"2024-02-05T04:53:30.137756+0000","last_active":"2024-02-05T04:53:32.143864+0000","last_peered":"2024-02-05T04:53:32.143864+0000","last_clean":"2024-02-05T04:53:32.143864+0000","last_became_active":"2024-02-05T04:53:24.723119+0000","last_became_peered":"2024-02-05T04:53:24.723119+0000","last_unstale":"2024-02-05T04:53:32.143864+0000","last_undegraded":"2024-02-05T04:53:32.143864+0000","last_fullsized":"2024-02-05T04:53:32.143864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:23.066897+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:50.375007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064357800000000001,"stat_sum":{"num_bytes":5292032,"num_objects":8,"num_object_clones":1,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":2156,"num_write":24,"num_write_kb":5664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,14,4,0,10,1],"acting":[11,2,14,4,0,10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"3.e","version":"40'25","reported_seq":66,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-02-05T04:53:33.691007+0000","last_change":"2024-02-05T04:53:30.149178+0000","last_active":"2024-02-05T04:53:33.691007+0000","last_peered":"2024-02-05T04:53:33.691007+0000","last_clean":"2024-02-05T04:53:33.691007+0000","last_became_active":"2024-02-05T04:53:24.641093+0000","last_became_peered":"2024-02-05T04:53:24.641093+0000","last_unstale":"2024-02-05T04:53:33.691007+0000","last_undegraded":"2024-02-05T04:53:33.691007+0000","last_fullsized":"2024-02-05T04:53:33.691007+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:23.066897+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:42.302165+0000","scrub_duration":0,"objects_trimmed":3,"snaptrim_duration":0.016258147000000001,"stat_sum":{"num_bytes":3063808,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":36,"num_write_kb":3744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,8,11,13,6,5],"acting":[2,4,8,11,13,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"3.d","version":"38'22","reported_seq":72,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-02-05T04:53:33.155264+0000","last_change":"2024-02-05T04:53:30.142199+0000","last_active":"2024-02-05T04:53:33.155264+0000","last_peered":"2024-02-05T04:53:33.155264+0000","last_clean":"2024-02-05T04:53:33.155264+0000","last_became_active":"2024-02-05T04:53:24.723128+0000","last_became_peered":"2024-02-05T04:53:24.723128+0000","last_unstale":"2024-02-05T04:53:33.155264+0000","last_undegraded":"2024-02-05T04:53:33.155264+0000","last_fullsized":"2024-02-05T04:53:33.155264+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:23.066897+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:49.868795+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.0086066909999999996,"stat_sum":{"num_bytes":6062080,"num_objects":4,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":1880,"num_write":19,"num_write_kb":6320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,14,10,9,13,5],"acting":[7,3,14,10,9,13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"3.c","version":"38'5","reported_seq":46,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-02-05T04:53:32.143784+0000","last_change":"2024-02-05T04:53:30.137652+0000","last_active":"2024-02-05T04:53:32.143784+0000","last_peered":"2024-02-05T04:53:32.143784+0000","last_clean":"2024-02-05T04:53:32.143784+0000","last_became_active":"2024-02-05T04:53:24.723258+0000","last_became_peered":"2024-02-05T04:53:24.723258+0000","last_unstale":"2024-02-05T04:53:32.143784+0000","last_undegraded":"2024-02-05T04:53:32.143784+0000","last_fullsized":"2024-02-05T04:53:32.143784+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:23.066897+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:49.363891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072003400000000004,"stat_sum":{"num_bytes":1196032,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":195,"num_write":5,"num_write_kb":1168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,13,5,10,6,1,9],"acting":[11,13,5,10,6,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"3.b","version":"40'24","reported_seq":82,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-02-05T04:53:32.222229+0000","last_change":"2024-02-05T04:53:30.139740+0000","last_active":"2024-02-05T04:53:32.222229+0000","last_peered":"2024-02-05T04:53:32.222229+0000","last_clean":"2024-02-05T04:53:32.222229+0000","last_became_active":"2024-02-05T04:53:24.486317+0000","last_became_peered":"2024-02-05T04:53:24.486317+0000","last_unstale":"2024-02-05T04:53:32.222229+0000","last_undegraded":"2024-02-05T04:53:32.222229+0000","last_fullsized":"2024-02-05T04:53:32.222229+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:23.066897+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:52.949763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080546700000000003,"stat_sum":{"num_bytes":6176768,"num_objects":8,"num_object_clones":2,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":35,"num_read_kb":7262,"num_write":22,"num_write_kb":6032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,1,8,0,6,7,12],"acting":[13,1,8,0,6,7,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"3.a","version":"0'0","reported_seq":34,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-02-05T04:53:31.155660+0000","last_change":"2024-02-05T04:53:30.136931+0000","last_active":"2024-02-05T04:53:31.155660+0000","last_peered":"2024-02-05T04:53:31.155660+0000","last_clean":"2024-02-05T04:53:31.155660+0000","last_became_active":"2024-02-05T04:53:24.488548+0000","last_became_peered":"2024-02-05T04:53:24.488548+0000","last_unstale":"2024-02-05T04:53:31.155660+0000","last_undegraded":"2024-02-05T04:53:31.155660+0000","last_fullsized":"2024-02-05T04:53:31.155660+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:23.066897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:40.240039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066884300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,8,9,14,6,11],"acting":[3,7,8,9,14,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"3.9","version":"34'11","reported_seq":56,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-02-05T04:53:32.162176+0000","last_change":"2024-02-05T04:53:30.144729+0000","last_active":"2024-02-05T04:53:32.162176+0000","last_peered":"2024-02-05T04:53:32.162176+0000","last_clean":"2024-02-05T04:53:32.162176+0000","last_became_active":"2024-02-05T04:53:24.722954+0000","last_became_peered":"2024-02-05T04:53:24.722954+0000","last_unstale":"2024-02-05T04:53:32.162176+0000","last_undegraded":"2024-02-05T04:53:32.162176+0000","last_fullsized":"2024-02-05T04:53:32.162176+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:23.066897+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:45.999739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052870900000000004,"stat_sum":{"num_bytes":2424832,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":13,"num_read_kb":3125,"num_write":11,"num_write_kb":3152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0,13,3,15,9,10],"acting":[14,0,13,3,15,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"3.8","version":"37'10","reported_seq":45,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-02-05T04:53:31.130630+0000","last_change":"2024-02-05T04:53:30.130865+0000","last_active":"2024-02-05T04:53:31.130630+0000","last_peered":"2024-02-05T04:53:31.130630+0000","last_clean":"2024-02-05T04:53:31.130630+0000","last_became_active":"2024-02-05T04:53:24.486354+0000","last_became_peered":"2024-02-05T04:53:24.486354+0000","last_unstale":"2024-02-05T04:53:31.130630+0000","last_undegraded":"2024-02-05T04:53:31.130630+0000","last_fullsized":"2024-02-05T04:53:31.130630+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:23.066897+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:41.034180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000602909,"stat_sum":{"num_bytes":573440,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":6,"num_write_kb":560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,0,14,15,6],"acting":[4,1,2,0,14,15,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"1.0","version":"26'157","reported_seq":223,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-02-05T04:53:32.161103+0000","last_change":"2024-02-05T04:53:09.625781+0000","last_active":"2024-02-05T04:53:32.161103+0000","last_peered":"2024-02-05T04:53:32.161103+0000","last_clean":"2024-02-05T04:53:32.161103+0000","last_became_active":"2024-02-05T04:53:09.624695+0000","last_became_peered":"2024-02-05T04:53:09.624695+0000","last_unstale":"2024-02-05T04:53:32.161103+0000","last_undegraded":"2024-02-05T04:53:32.161103+0000","last_fullsized":"2024-02-05T04:53:32.161103+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:07.900741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:07.900741+0000","last_clean_scrub_stamp":"2024-02-05T04:53:07.900741+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:23.258506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2],"acting":[8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":20,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-05T04:53:30.130395+0000","last_change":"2024-02-05T04:53:12.028466+0000","last_active":"2024-02-05T04:53:30.130395+0000","last_peered":"2024-02-05T04:53:30.130395+0000","last_clean":"2024-02-05T04:53:30.130395+0000","last_became_active":"2024-02-05T04:53:12.028138+0000","last_became_peered":"2024-02-05T04:53:12.028138+0000","last_unstale":"2024-02-05T04:53:30.130395+0000","last_undegraded":"2024-02-05T04:53:30.130395+0000","last_fullsized":"2024-02-05T04:53:30.130395+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_clean_scrub_stamp":"2024-02-05T04:53:11.017155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:38.427375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,15],"acting":[4,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"39'33","reported_seq":75,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-02-05T04:53:33.157166+0000","last_change":"2024-02-05T04:53:30.160725+0000","last_active":"2024-02-05T04:53:33.157166+0000","last_peered":"2024-02-05T04:53:33.157166+0000","last_clean":"2024-02-05T04:53:33.157166+0000","last_became_active":"2024-02-05T04:53:24.486494+0000","last_became_peered":"2024-02-05T04:53:24.486494+0000","last_unstale":"2024-02-05T04:53:33.157166+0000","last_undegraded":"2024-02-05T04:53:33.157166+0000","last_fullsized":"2024-02-05T04:53:33.157166+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:23.066897+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:29.148681+0000","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.018221533000000002,"stat_sum":{"num_bytes":3440640,"num_objects":5,"num_object_clones":2,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":4856,"num_write":20,"num_write_kb":3408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,2,3,6,15,8,11],"acting":[12,2,3,6,15,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-05T04:53:30.136078+0000","last_change":"2024-02-05T04:53:12.040500+0000","last_active":"2024-02-05T04:53:30.136078+0000","last_peered":"2024-02-05T04:53:30.136078+0000","last_clean":"2024-02-05T04:53:30.136078+0000","last_became_active":"2024-02-05T04:53:12.040297+0000","last_became_peered":"2024-02-05T04:53:12.040297+0000","last_unstale":"2024-02-05T04:53:30.136078+0000","last_undegraded":"2024-02-05T04:53:30.136078+0000","last_fullsized":"2024-02-05T04:53:30.136078+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_clean_scrub_stamp":"2024-02-05T04:53:11.017155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:20.053075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"40'25","reported_seq":64,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-02-05T04:53:32.197654+0000","last_change":"2024-02-05T04:53:30.149067+0000","last_active":"2024-02-05T04:53:32.197654+0000","last_peered":"2024-02-05T04:53:32.197654+0000","last_clean":"2024-02-05T04:53:32.197654+0000","last_became_active":"2024-02-05T04:53:24.723508+0000","last_became_peered":"2024-02-05T04:53:24.723508+0000","last_unstale":"2024-02-05T04:53:32.197654+0000","last_undegraded":"2024-02-05T04:53:32.197654+0000","last_fullsized":"2024-02-05T04:53:32.197654+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:23.066897+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:40.903038+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.010166428999999999,"stat_sum":{"num_bytes":2932736,"num_objects":5,"num_object_clones":2,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":5,"num_write":18,"num_write_kb":3216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,11,4,8,10,2,14],"acting":[13,11,4,8,10,2,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"2.1","version":"0'0","reported_seq":24,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-02-05T04:53:32.161171+0000","last_change":"2024-02-05T04:53:12.033593+0000","last_active":"2024-02-05T04:53:32.161171+0000","last_peered":"2024-02-05T04:53:32.161171+0000","last_clean":"2024-02-05T04:53:32.161171+0000","last_became_active":"2024-02-05T04:53:12.033337+0000","last_became_peered":"2024-02-05T04:53:12.033337+0000","last_unstale":"2024-02-05T04:53:32.161171+0000","last_undegraded":"2024-02-05T04:53:32.161171+0000","last_fullsized":"2024-02-05T04:53:32.161171+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_clean_scrub_stamp":"2024-02-05T04:53:11.017155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:14.948579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4],"acting":[8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.0","version":"34'5","reported_seq":44,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-02-05T04:53:32.161705+0000","last_change":"2024-02-05T04:53:30.144407+0000","last_active":"2024-02-05T04:53:32.161705+0000","last_peered":"2024-02-05T04:53:32.161705+0000","last_clean":"2024-02-05T04:53:32.161705+0000","last_became_active":"2024-02-05T04:53:24.485082+0000","last_became_peered":"2024-02-05T04:53:24.485082+0000","last_unstale":"2024-02-05T04:53:32.161705+0000","last_undegraded":"2024-02-05T04:53:32.161705+0000","last_fullsized":"2024-02-05T04:53:32.161705+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:23.066897+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:33.698562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074980399999999997,"stat_sum":{"num_bytes":114688,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":5,"num_write_kb":432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,2,15,0,12,8,9],"acting":[14,2,15,0,12,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"2.2","version":"0'0","reported_seq":20,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-05T04:53:30.136259+0000","last_change":"2024-02-05T04:53:12.035901+0000","last_active":"2024-02-05T04:53:30.136259+0000","last_peered":"2024-02-05T04:53:30.136259+0000","last_clean":"2024-02-05T04:53:30.136259+0000","last_became_active":"2024-02-05T04:53:12.035739+0000","last_became_peered":"2024-02-05T04:53:12.035739+0000","last_unstale":"2024-02-05T04:53:30.136259+0000","last_undegraded":"2024-02-05T04:53:30.136259+0000","last_fullsized":"2024-02-05T04:53:30.136259+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_clean_scrub_stamp":"2024-02-05T04:53:11.017155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:14.062467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9],"acting":[3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"39'20","reported_seq":59,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-02-05T04:53:31.187602+0000","last_change":"2024-02-05T04:53:30.139074+0000","last_active":"2024-02-05T04:53:31.187602+0000","last_peered":"2024-02-05T04:53:31.187602+0000","last_clean":"2024-02-05T04:53:31.187602+0000","last_became_active":"2024-02-05T04:53:24.641280+0000","last_became_peered":"2024-02-05T04:53:24.641280+0000","last_unstale":"2024-02-05T04:53:31.187602+0000","last_undegraded":"2024-02-05T04:53:31.187602+0000","last_fullsized":"2024-02-05T04:53:31.187602+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:23.066897+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:50.543393+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.0087663459999999995,"stat_sum":{"num_bytes":1916928,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":12,"num_read_kb":1877,"num_write":12,"num_write_kb":2112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1,15,2,5,3],"acting":[4,7,1,15,2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"3.5","version":"39'17","reported_seq":60,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-02-05T04:53:32.160351+0000","last_change":"2024-02-05T04:53:30.129958+0000","last_active":"2024-02-05T04:53:32.160351+0000","last_peered":"2024-02-05T04:53:32.160351+0000","last_clean":"2024-02-05T04:53:32.160351+0000","last_became_active":"2024-02-05T04:53:24.643048+0000","last_became_peered":"2024-02-05T04:53:24.643048+0000","last_unstale":"2024-02-05T04:53:32.160351+0000","last_undegraded":"2024-02-05T04:53:32.160351+0000","last_fullsized":"2024-02-05T04:53:32.160351+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:23.066897+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:27.951270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045887599999999999,"stat_sum":{"num_bytes":4161536,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":1830,"num_write":13,"num_write_kb":4064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13,5,8,7,1,6],"acting":[15,13,5,8,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-02-05T04:53:31.139606+0000","last_change":"2024-02-05T04:53:12.032853+0000","last_active":"2024-02-05T04:53:31.139606+0000","last_peered":"2024-02-05T04:53:31.139606+0000","last_clean":"2024-02-05T04:53:31.139606+0000","last_became_active":"2024-02-05T04:53:12.032664+0000","last_became_peered":"2024-02-05T04:53:12.032664+0000","last_unstale":"2024-02-05T04:53:31.139606+0000","last_undegraded":"2024-02-05T04:53:31.139606+0000","last_fullsized":"2024-02-05T04:53:31.139606+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_clean_scrub_stamp":"2024-02-05T04:53:11.017155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:14.638056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"38'23","reported_seq":71,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-02-05T04:53:32.163546+0000","last_change":"2024-02-05T04:53:30.139523+0000","last_active":"2024-02-05T04:53:32.163546+0000","last_peered":"2024-02-05T04:53:32.163546+0000","last_clean":"2024-02-05T04:53:32.163546+0000","last_became_active":"2024-02-05T04:53:24.643568+0000","last_became_peered":"2024-02-05T04:53:24.643568+0000","last_unstale":"2024-02-05T04:53:32.163546+0000","last_undegraded":"2024-02-05T04:53:32.163546+0000","last_fullsized":"2024-02-05T04:53:32.163546+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:23.066897+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:37.484371+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.0064274600000000003,"stat_sum":{"num_bytes":5242880,"num_objects":5,"num_object_clones":0,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":17,"num_read_kb":873,"num_write":39,"num_write_kb":5120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5,8,9,4,13],"acting":[1,2,5,8,9,4,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"2.5","version":"0'0","reported_seq":21,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-05T04:53:30.135515+0000","last_change":"2024-02-05T04:53:12.753233+0000","last_active":"2024-02-05T04:53:30.135515+0000","last_peered":"2024-02-05T04:53:30.135515+0000","last_clean":"2024-02-05T04:53:30.135515+0000","last_became_active":"2024-02-05T04:53:12.752915+0000","last_became_peered":"2024-02-05T04:53:12.752915+0000","last_unstale":"2024-02-05T04:53:30.135515+0000","last_undegraded":"2024-02-05T04:53:30.135515+0000","last_fullsized":"2024-02-05T04:53:30.135515+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_clean_scrub_stamp":"2024-02-05T04:53:11.017155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:18.391150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"38'8","reported_seq":45,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-02-05T04:53:32.157158+0000","last_change":"2024-02-05T04:53:30.154505+0000","last_active":"2024-02-05T04:53:32.157158+0000","last_peered":"2024-02-05T04:53:32.157158+0000","last_clean":"2024-02-05T04:53:32.157158+0000","last_became_active":"2024-02-05T04:53:24.641453+0000","last_became_peered":"2024-02-05T04:53:24.641453+0000","last_unstale":"2024-02-05T04:53:32.157158+0000","last_undegraded":"2024-02-05T04:53:32.157158+0000","last_fullsized":"2024-02-05T04:53:32.157158+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:23.066897+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:29.454665+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.015508259999999999,"stat_sum":{"num_bytes":901120,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":24,"num_write_kb":880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,6,12,1,4,5,11],"acting":[13,6,12,1,4,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"2.6","version":"0'0","reported_seq":22,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-02-05T04:53:31.224237+0000","last_change":"2024-02-05T04:53:12.037950+0000","last_active":"2024-02-05T04:53:31.224237+0000","last_peered":"2024-02-05T04:53:31.224237+0000","last_clean":"2024-02-05T04:53:31.224237+0000","last_became_active":"2024-02-05T04:53:12.037750+0000","last_became_peered":"2024-02-05T04:53:12.037750+0000","last_unstale":"2024-02-05T04:53:31.224237+0000","last_undegraded":"2024-02-05T04:53:31.224237+0000","last_fullsized":"2024-02-05T04:53:31.224237+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_clean_scrub_stamp":"2024-02-05T04:53:11.017155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:26.129079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.6","version":"40'22","reported_seq":63,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-02-05T04:53:32.167908+0000","last_change":"2024-02-05T04:53:30.151858+0000","last_active":"2024-02-05T04:53:32.167908+0000","last_peered":"2024-02-05T04:53:32.167908+0000","last_clean":"2024-02-05T04:53:32.167908+0000","last_became_active":"2024-02-05T04:53:24.485438+0000","last_became_peered":"2024-02-05T04:53:24.485438+0000","last_unstale":"2024-02-05T04:53:32.167908+0000","last_undegraded":"2024-02-05T04:53:32.167908+0000","last_fullsized":"2024-02-05T04:53:32.167908+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:23.066897+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:42.314856+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.012872903999999999,"stat_sum":{"num_bytes":3375104,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":11,"num_read_kb":533,"num_write":16,"num_write_kb":3712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,12,4,15,1,7,8],"acting":[13,12,4,15,1,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"2.7","version":"0'0","reported_seq":21,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-02-05T04:53:31.130847+0000","last_change":"2024-02-05T04:53:12.781269+0000","last_active":"2024-02-05T04:53:31.130847+0000","last_peered":"2024-02-05T04:53:31.130847+0000","last_clean":"2024-02-05T04:53:31.130847+0000","last_became_active":"2024-02-05T04:53:12.780970+0000","last_became_peered":"2024-02-05T04:53:12.780970+0000","last_unstale":"2024-02-05T04:53:31.130847+0000","last_undegraded":"2024-02-05T04:53:31.130847+0000","last_fullsized":"2024-02-05T04:53:31.130847+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_clean_scrub_stamp":"2024-02-05T04:53:11.017155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:36.119048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,0],"acting":[15,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":46874624,"num_objects":58,"num_object_clones":12,"num_object_copies":406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":0,"num_read":199,"num_read_kb":24603,"num_write":270,"num_write_kb":49584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":83410944,"data_stored":99414016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":275,"ondisk_log_size":275,"up":112,"acting":112,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":12,"up_from":23,"seq":98784247816,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30944,"kb_used_data":3916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340896,"statfs":{"total":96636764160,"available":96605077504,"internally_reserved":0,"allocated":4009984,"data_stored":4846822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,8,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28899999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57499999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.252}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35199999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51700000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36499999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.313}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.377}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60199999999999998}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47499999999999998}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.443}]}]},{"osd":11,"up_from":23,"seq":98784247818,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31612,"kb_used_data":4588,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340228,"statfs":{"total":96636764160,"available":96604393472,"internally_reserved":0,"allocated":4698112,"data_stored":5795109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,4,5,6,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40500000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.312}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59399999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53200000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57599999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58699999999999997}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71999999999999997}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73399999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66000000000000003}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67200000000000004}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88800000000000001}]}]},{"osd":10,"up_from":23,"seq":98784247818,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31876,"kb_used_data":4852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339964,"statfs":{"total":96636764160,"available":96604123136,"internally_reserved":0,"allocated":4968448,"data_stored":5209381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36699999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32400000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60299999999999998}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.496}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41299999999999998}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68600000000000005}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59099999999999997}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46999999999999997}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57899999999999996}]}]},{"osd":3,"up_from":21,"seq":90194313225,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30868,"kb_used_data":3844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340972,"statfs":{"total":96636764160,"available":96605155328,"internally_reserved":0,"allocated":3936256,"data_stored":4511076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,4,6,7,8,9,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69399999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60399999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45800000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.622}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40699999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29799999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65400000000000003}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49299999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59699999999999998}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52900000000000003}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69799999999999995}]}]},{"osd":4,"up_from":21,"seq":90194313224,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32532,"kb_used_data":5512,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339308,"statfs":{"total":96636764160,"available":96603451392,"internally_reserved":0,"allocated":5644288,"data_stored":5727588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,9,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.245}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41599999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.31900000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52400000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53700000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.501}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33500000000000002}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64800000000000002}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60799999999999998}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.443}]}]},{"osd":5,"up_from":21,"seq":90194313224,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32668,"kb_used_data":5644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339172,"statfs":{"total":96636764160,"available":96603312128,"internally_reserved":0,"allocated":5779456,"data_stored":6006116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,6,7,9,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29099999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57999999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33800000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60899999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.378}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47499999999999998}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50800000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59299999999999997}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]}]},{"osd":6,"up_from":21,"seq":90194313225,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31592,"kb_used_data":4568,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340248,"statfs":{"total":96636764160,"available":96604413952,"internally_reserved":0,"allocated":4677632,"data_stored":5092708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,3,4,5,7,8,9,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59699999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42399999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41199999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52700000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39400000000000002}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36599999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57699999999999996}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48799999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47499999999999998}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55700000000000005}]}]},{"osd":1,"up_from":22,"seq":94489280521,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33872,"kb_used_data":6852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337968,"statfs":{"total":96636764160,"available":96602079232,"internally_reserved":0,"allocated":7016448,"data_stored":7259492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,4,5,8,9,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50900000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72799999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35799999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.626}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64100000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67800000000000005}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57199999999999995}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54900000000000004}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69299999999999995}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78300000000000003}]}]},{"osd":14,"up_from":22,"seq":94489280520,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32084,"kb_used_data":5060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339756,"statfs":{"total":96636764160,"available":96603910144,"internally_reserved":0,"allocated":5181440,"data_stored":5545253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.37}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63300000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.308}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54900000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39000000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68200000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34100000000000003}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40100000000000002}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.501}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29099999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48699999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51300000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52500000000000002}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53600000000000003}]}]},{"osd":9,"up_from":22,"seq":94489280520,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31176,"kb_used_data":4152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340664,"statfs":{"total":96636764160,"available":96604839936,"internally_reserved":0,"allocated":4251648,"data_stored":4001061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,3,4,5,6,8,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28499999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77500000000000002}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.753}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61199999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86699999999999999}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69799999999999995}]}]},{"osd":13,"up_from":22,"seq":94489280519,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36180,"kb_used_data":9156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94335660,"statfs":{"total":96636764160,"available":96599715840,"internally_reserved":0,"allocated":9375744,"data_stored":10710309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,4,5,6,7,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40600000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59799999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64200000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49299999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64200000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.622}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47699999999999998}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.505}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55200000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63300000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60999999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67100000000000004}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74099999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247817,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31416,"kb_used_data":4388,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340424,"statfs":{"total":96636764160,"available":96604594176,"internally_reserved":0,"allocated":4493312,"data_stored":4642022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41299999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.628}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.503}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57499999999999996}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67200000000000004}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68200000000000005}]}]},{"osd":15,"up_from":22,"seq":94489280521,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32072,"kb_used_data":5048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339768,"statfs":{"total":96636764160,"available":96603922432,"internally_reserved":0,"allocated":5169152,"data_stored":6491429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,4,5,6,7,8,9,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73699999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69399999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58899999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57899999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68400000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61199999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.628}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79800000000000004}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67100000000000004}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86299999999999999}]}]},{"osd":2,"up_from":23,"seq":98784247818,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38352,"kb_used_data":7172,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94333488,"statfs":{"total":96636764160,"available":96597491712,"internally_reserved":0,"allocated":7344128,"data_stored":9039685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,11,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56599999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39500000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28000000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58299999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30499999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40899999999999997}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.434}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60199999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44600000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64000000000000001}]}]},{"osd":7,"up_from":23,"seq":98784247818,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33128,"kb_used_data":6104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338712,"statfs":{"total":96636764160,"available":96602841088,"internally_reserved":0,"allocated":6250496,"data_stored":7224613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,2,3,5,6,8,9,10,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.378}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61099999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35299999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41199999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46400000000000002}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70099999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85199999999999998}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67600000000000005}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47799999999999998}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65300000000000002}]}]},{"osd":8,"up_from":23,"seq":98784247815,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35000,"kb_used_data":7976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94336840,"statfs":{"total":96636764160,"available":96600924160,"internally_reserved":0,"allocated":8167424,"data_stored":9991942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,4,6,7,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26300000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.318}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47699999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39500000000000002}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44700000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40500000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.628}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54300000000000004}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68999999999999995}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57299999999999995}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72499999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3985408,"data_stored":4390912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":7884800,"data_stored":9240576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6320128,"data_stored":8269824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3461120,"data_stored":4280320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":5169152,"data_stored":5496832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":5304320,"data_stored":5775360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4202496,"data_stored":4861952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":5758976,"data_stored":6983680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":7127040,"data_stored":9211904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":3760128,"data_stored":3760128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4476928,"data_stored":4968448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4206592,"data_stored":5554176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":3502080,"data_stored":4595712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":8884224,"data_stored":10469376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":4689920,"data_stored":5304320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":4677632,"data_stored":6250496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T04:53:36.878 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-02-05T04:53:36.878 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T04:53:36.879 INFO:tasks.thrashosds.thrasher:Setting full ratio to .001 2024-02-05T04:53:36.879 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set-full-ratio .001 2024-02-05T04:53:36.902 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:36.902 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:36.912 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-02-05T04:53:36.958 INFO:tasks.ceph.osd.13.smithi091.stderr:2024-02-05T04:53:36.957+0000 7f526fd4d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 34281) UID: 0 2024-02-05T04:53:37.018 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:37.018 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:37.027 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-02-05T04:53:37.059 INFO:tasks.ceph.osd.9.smithi091.stderr:2024-02-05T04:53:37.057+0000 7efc18827640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 34279) UID: 0 2024-02-05T04:53:37.133 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:37.133 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:37.144 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-02-05T04:53:37.159 INFO:tasks.ceph.osd.8.smithi062.stderr:2024-02-05T04:53:37.158+0000 7f9385bf2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 34693) UID: 0 2024-02-05T04:53:37.172 INFO:tasks.rados.rados.0.smithi196.stdout:150: done (1 left) 2024-02-05T04:53:37.172 INFO:tasks.rados.rados.0.smithi196.stdout:151: delete oid 19 current snap is 6 2024-02-05T04:53:37.172 INFO:tasks.rados.rados.0.smithi196.stdout:147: expect (ObjNum 43 snap 0 seq_num 43) 2024-02-05T04:53:37.196 INFO:tasks.rados.rados.0.smithi196.stdout:147: done (1 left) 2024-02-05T04:53:37.197 INFO:tasks.rados.rados.0.smithi196.stdout:151: done (0 left) 2024-02-05T04:53:37.197 INFO:tasks.rados.rados.0.smithi196.stdout:152: rollback oid 12 current snap is 6 2024-02-05T04:53:37.197 INFO:tasks.rados.rados.0.smithi196.stdout:rollback oid 12 to 5 2024-02-05T04:53:37.197 INFO:tasks.rados.rados.0.smithi196.stdout:153: rmattr oid 42 current snap is 6 2024-02-05T04:53:37.197 INFO:tasks.rados.rados.0.smithi196.stdout:153: done (1 left) 2024-02-05T04:53:37.197 INFO:tasks.rados.rados.0.smithi196.stdout:154: rmattr oid 11 current snap is 6 2024-02-05T04:53:37.197 INFO:tasks.rados.rados.0.smithi196.stdout:154: done (1 left) 2024-02-05T04:53:37.197 INFO:tasks.rados.rados.0.smithi196.stdout:155: read oid 6 snap -1 2024-02-05T04:53:37.197 INFO:tasks.rados.rados.0.smithi196.stdout:155: expect (ObjNum 70 snap 2 seq_num 70) 2024-02-05T04:53:37.197 INFO:tasks.rados.rados.0.smithi196.stdout:156: snap_create 2024-02-05T04:53:37.197 INFO:tasks.rados.rados.0.smithi196.stdout:152: finishing rollback tid 1 to smithi19639362-12 2024-02-05T04:53:37.197 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 12 v 0 (ObjNum 177 snap 0 seq_num 3192036480) dirty dne 2024-02-05T04:53:37.203 INFO:tasks.rados.rados.0.smithi196.stdout:155: expect (ObjNum 70 snap 2 seq_num 70) 2024-02-05T04:53:37.255 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:37.255 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:37.260 INFO:tasks.ceph.osd.7.smithi196.stderr:2024-02-05T04:53:37.259+0000 7fb281419640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34606) UID: 0 2024-02-05T04:53:37.263 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-02-05T04:53:37.362 INFO:tasks.ceph.osd.12.smithi062.stderr:2024-02-05T04:53:37.361+0000 7f5e5847e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 34695) UID: 0 2024-02-05T04:53:37.368 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:37.368 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:37.377 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-02-05T04:53:37.463 INFO:tasks.ceph.osd.3.smithi196.stderr:2024-02-05T04:53:37.461+0000 7fc8817c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34605) UID: 0 2024-02-05T04:53:37.482 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:37.483 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:37.491 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-02-05T04:53:37.564 INFO:tasks.ceph.osd.1.smithi091.stderr:2024-02-05T04:53:37.563+0000 7ff9dc283640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34275) UID: 0 2024-02-05T04:53:37.597 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:37.597 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:37.606 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-02-05T04:53:37.664 INFO:tasks.ceph.osd.3.smithi196.stderr:2024-02-05T04:53:37.663+0000 7fc8817c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34605) UID: 0 2024-02-05T04:53:37.711 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:37.711 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:37.720 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-02-05T04:53:37.766 INFO:tasks.ceph.osd.5.smithi091.stderr:2024-02-05T04:53:37.764+0000 7f63983b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34277) UID: 0 2024-02-05T04:53:37.826 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:37.826 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:37.834 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-02-05T04:53:37.878 INFO:tasks.ceph.osd.14.smithi150.stderr:2024-02-05T04:53:37.877+0000 7f5f81083640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 34020) UID: 0 2024-02-05T04:53:37.939 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:37.940 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:37.948 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-02-05T04:53:37.968 INFO:tasks.ceph.osd.15.smithi196.stderr:2024-02-05T04:53:37.967+0000 7f1facc45640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 34611) UID: 0 2024-02-05T04:53:38.068 INFO:tasks.ceph.osd.4.smithi062.stderr:2024-02-05T04:53:38.068+0000 7f708c308640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34689) UID: 0 2024-02-05T04:53:38.088 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T04:53:38.094 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:38.094 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:38.103 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-02-05T04:53:38.169 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-02-05T04:53:38.168+0000 7fe601a80640 -1 log_channel(cluster) log [ERR] : Health check failed: full ratio(s) out of order (OSD_OUT_OF_ORDER_FULL) 2024-02-05T04:53:38.170 INFO:tasks.ceph.osd.6.smithi150.stderr:2024-02-05T04:53:38.168+0000 7f6bea9f0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34015) UID: 0 2024-02-05T04:53:38.173 INFO:teuthology.orchestra.run.smithi062.stderr:osd set-full-ratio 0.001 2024-02-05T04:53:38.177 INFO:tasks.rados.rados.0.smithi196.stdout:152: done (2 left) 2024-02-05T04:53:38.177 INFO:tasks.rados.rados.0.smithi196.stdout:155: done (1 left) 2024-02-05T04:53:38.177 INFO:tasks.rados.rados.0.smithi196.stdout:156: done (0 left) 2024-02-05T04:53:38.177 INFO:tasks.rados.rados.0.smithi196.stdout:append oid 45 current snap is 7 2024-02-05T04:53:38.177 INFO:tasks.rados.rados.0.smithi196.stdout:157: seq_num 89 ranges {753664=737280,1490944=573440,2064384=196608} 2024-02-05T04:53:38.185 INFO:tasks.rados.rados.0.smithi196.stdout:157: writing smithi19639362-45 from 753664 to 1490944 tid 1 2024-02-05T04:53:38.188 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-02-05T04:53:38.189 INFO:tasks.rados.rados.0.smithi196.stdout:157: writing smithi19639362-45 from 1490944 to 2064384 tid 2 2024-02-05T04:53:38.189 INFO:tasks.rados.rados.0.smithi196.stdout:157: writing smithi19639362-45 from 2064384 to 2260992 tid 3 2024-02-05T04:53:38.190 INFO:tasks.rados.rados.0.smithi196.stdout:158: snap_create 2024-02-05T04:53:38.198 INFO:tasks.rados.rados.0.smithi196.stdout:157: finishing write tid 1 to smithi19639362-45 2024-02-05T04:53:38.198 INFO:tasks.rados.rados.0.smithi196.stdout:157: finishing write tid 2 to smithi19639362-45 2024-02-05T04:53:38.198 INFO:tasks.rados.rados.0.smithi196.stdout:157: finishing write tid 3 to smithi19639362-45 2024-02-05T04:53:38.198 INFO:tasks.rados.rados.0.smithi196.stdout:157: finishing write tid 4 to smithi19639362-45 2024-02-05T04:53:38.199 INFO:tasks.rados.rados.0.smithi196.stdout:157: finishing write tid 5 to smithi19639362-45 2024-02-05T04:53:38.199 INFO:tasks.rados.rados.0.smithi196.stdout:157: oid 45 updating version 0 to 36 2024-02-05T04:53:38.199 INFO:tasks.rados.rados.0.smithi196.stdout:157: oid 45 updating version 36 to 38 2024-02-05T04:53:38.199 INFO:tasks.rados.rados.0.smithi196.stdout:157: oid 45 version 38 is already newer than 35 2024-02-05T04:53:38.200 INFO:tasks.rados.rados.0.smithi196.stdout:157: oid 45 version 38 is already newer than 37 2024-02-05T04:53:38.200 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 45 v 38 (ObjNum 89 snap 7 seq_num 89) dirty exists 2024-02-05T04:53:38.200 INFO:tasks.rados.rados.0.smithi196.stdout:157: left oid 45 (ObjNum 89 snap 7 seq_num 89) 2024-02-05T04:53:38.211 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:38.211 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:38.220 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-02-05T04:53:38.231 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0: {} 2024-02-05T04:53:38.231 INFO:teuthology.orchestra.run.smithi062.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T04:53:38.249 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1: {} 2024-02-05T04:53:38.250 INFO:teuthology.orchestra.run.smithi062.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T04:53:38.268 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2: {} 2024-02-05T04:53:38.268 INFO:teuthology.orchestra.run.smithi062.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T04:53:38.270 INFO:tasks.ceph.osd.2.smithi150.stderr:2024-02-05T04:53:38.269+0000 7f7337bf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 34014) UID: 0 2024-02-05T04:53:38.285 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3: {} 2024-02-05T04:53:38.286 INFO:teuthology.orchestra.run.smithi062.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T04:53:38.303 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4: {} 2024-02-05T04:53:38.303 INFO:teuthology.orchestra.run.smithi062.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T04:53:38.321 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5: {} 2024-02-05T04:53:38.321 INFO:teuthology.orchestra.run.smithi062.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T04:53:38.335 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T04:53:38.335 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-02-05T04:53:38.335 INFO:teuthology.orchestra.run.smithi196.stdout: "num_ops": 0 2024-02-05T04:53:38.335 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T04:53:38.340 INFO:teuthology.orchestra.run.smithi062.stdout:osd.6: {} 2024-02-05T04:53:38.340 INFO:teuthology.orchestra.run.smithi062.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T04:53:38.344 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-02-05T04:53:38.357 INFO:teuthology.orchestra.run.smithi062.stdout:osd.7: {} 2024-02-05T04:53:38.358 INFO:teuthology.orchestra.run.smithi062.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T04:53:38.371 INFO:tasks.ceph.osd.6.smithi150.stderr:2024-02-05T04:53:38.370+0000 7f6bea9f0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34015) UID: 0 2024-02-05T04:53:38.374 INFO:teuthology.orchestra.run.smithi062.stdout:osd.8: {} 2024-02-05T04:53:38.375 INFO:teuthology.orchestra.run.smithi062.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-05T04:53:38.392 INFO:teuthology.orchestra.run.smithi062.stdout:osd.9: {} 2024-02-05T04:53:38.393 INFO:teuthology.orchestra.run.smithi062.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-05T04:53:38.410 INFO:teuthology.orchestra.run.smithi062.stdout:osd.10: {} 2024-02-05T04:53:38.410 INFO:teuthology.orchestra.run.smithi062.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-05T04:53:38.428 INFO:teuthology.orchestra.run.smithi062.stdout:osd.11: {} 2024-02-05T04:53:38.428 INFO:teuthology.orchestra.run.smithi062.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-05T04:53:38.445 INFO:teuthology.orchestra.run.smithi062.stdout:osd.12: {} 2024-02-05T04:53:38.445 INFO:teuthology.orchestra.run.smithi062.stderr:osd.12: osd_enable_op_tracker = '' 2024-02-05T04:53:38.451 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T04:53:38.451 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-02-05T04:53:38.451 INFO:teuthology.orchestra.run.smithi196.stdout: "complaint_time": 30, 2024-02-05T04:53:38.451 INFO:teuthology.orchestra.run.smithi196.stdout: "num_blocked_ops": 0 2024-02-05T04:53:38.451 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T04:53:38.460 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-02-05T04:53:38.472 INFO:tasks.ceph.osd.10.smithi150.stderr:2024-02-05T04:53:38.470+0000 7f3075a4d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 34018) UID: 0 2024-02-05T04:53:38.568 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T04:53:38.568 INFO:teuthology.orchestra.run.smithi196.stdout: "size": 20, 2024-02-05T04:53:38.568 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 600, 2024-02-05T04:53:38.568 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-02-05T04:53:38.568 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.568 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.3s6 37/31 ECSubWrite(tid=26, reqid=osd.4.0:26, at_version=37'10, trim_to=0'0, roll_forward_to=35'8))", 2024-02-05T04:53:38.568 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:29.129001+0000", 2024-02-05T04:53:38.568 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 9.4369139769999997, 2024-02-05T04:53:38.568 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001632823, 2024-02-05T04:53:38.569 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:38.569 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:38.569 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:38.569 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.569 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:38.569 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.129001+0000", 2024-02-05T04:53:38.569 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:38.569 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.569 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.569 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:38.569 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.129001+0000", 2024-02-05T04:53:38.569 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:38.569 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.569 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.569 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:38.570 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.129001+0000", 2024-02-05T04:53:38.570 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.4900000000000002e-07 2024-02-05T04:53:38.570 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.570 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.570 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:38.570 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.129062+0000", 2024-02-05T04:53:38.570 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0878000000000002e-05 2024-02-05T04:53:38.570 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.570 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.570 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:38.570 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.129078+0000", 2024-02-05T04:53:38.570 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5316999999999999e-05 2024-02-05T04:53:38.571 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.571 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.571 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:38.571 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.129090+0000", 2024-02-05T04:53:38.571 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2199e-05 2024-02-05T04:53:38.571 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.571 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.572 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:38.572 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.129183+0000", 2024-02-05T04:53:38.572 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.3323000000000001e-05 2024-02-05T04:53:38.572 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.573 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.573 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:38.573 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.129196+0000", 2024-02-05T04:53:38.573 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2816000000000001e-05 2024-02-05T04:53:38.573 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.573 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.574 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:38.574 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.130587+0000", 2024-02-05T04:53:38.574 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0013914800000000001 2024-02-05T04:53:38.574 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.574 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.574 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:38.574 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.130634+0000", 2024-02-05T04:53:38.575 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6560999999999997e-05 2024-02-05T04:53:38.575 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:38.575 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:38.575 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:38.575 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.575 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.575 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.3s6 37/31 ECSubWrite(tid=27, reqid=osd.4.0:26, at_version=0'0, trim_to=0'0, roll_forward_to=37'10))", 2024-02-05T04:53:38.576 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:29.131110+0000", 2024-02-05T04:53:38.576 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 9.4348052960000004, 2024-02-05T04:53:38.576 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002311582, 2024-02-05T04:53:38.576 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:38.576 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:38.576 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:38.577 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.577 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:38.577 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.131110+0000", 2024-02-05T04:53:38.577 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:38.577 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.577 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.578 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:38.578 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.131110+0000", 2024-02-05T04:53:38.578 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:38.578 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.578 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.578 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:38.578 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.131111+0000", 2024-02-05T04:53:38.579 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6099999999999999e-07 2024-02-05T04:53:38.579 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.579 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.579 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:38.579 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.131285+0000", 2024-02-05T04:53:38.579 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00017457799999999999 2024-02-05T04:53:38.580 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.580 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.580 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:38.580 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.131298+0000", 2024-02-05T04:53:38.580 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2500999999999999e-05 2024-02-05T04:53:38.580 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.580 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.581 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:38.581 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.131307+0000", 2024-02-05T04:53:38.581 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1109999999999997e-06 2024-02-05T04:53:38.581 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.581 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.581 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:38.582 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.131396+0000", 2024-02-05T04:53:38.582 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.9377999999999997e-05 2024-02-05T04:53:38.582 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.582 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.582 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:38.582 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.131405+0000", 2024-02-05T04:53:38.582 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4090000000000003e-06 2024-02-05T04:53:38.583 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.583 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.583 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:38.583 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.133378+0000", 2024-02-05T04:53:38.583 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0019734090000000002 2024-02-05T04:53:38.583 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.584 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.584 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:38.584 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.133421+0000", 2024-02-05T04:53:38.584 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3235000000000002e-05 2024-02-05T04:53:38.584 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:38.584 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:38.585 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:38.585 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.585 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.585 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.3s6 37/31 ECSubWrite(tid=29, reqid=client.4634.0:218, at_version=37'12, trim_to=0'0, roll_forward_to=37'10))", 2024-02-05T04:53:38.585 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:29.134419+0000", 2024-02-05T04:53:38.585 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 9.4314959740000006, 2024-02-05T04:53:38.586 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0029388539999999999, 2024-02-05T04:53:38.586 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:38.586 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:38.586 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:38.586 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.586 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:38.586 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.134419+0000", 2024-02-05T04:53:38.587 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:38.587 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.587 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.587 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:38.587 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.134419+0000", 2024-02-05T04:53:38.587 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:38.588 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.588 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.588 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:38.588 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.134420+0000", 2024-02-05T04:53:38.588 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2900000000000003e-07 2024-02-05T04:53:38.588 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.589 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.589 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:38.589 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135211+0000", 2024-02-05T04:53:38.589 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00079167699999999996 2024-02-05T04:53:38.589 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.589 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.590 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:38.590 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135234+0000", 2024-02-05T04:53:38.590 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2869000000000001e-05 2024-02-05T04:53:38.590 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.590 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.590 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:38.591 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135287+0000", 2024-02-05T04:53:38.591 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2948999999999997e-05 2024-02-05T04:53:38.591 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.591 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.591 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:38.591 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135370+0000", 2024-02-05T04:53:38.592 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.2416000000000005e-05 2024-02-05T04:53:38.592 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.592 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.592 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:38.592 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135380+0000", 2024-02-05T04:53:38.592 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.9250000000000007e-06 2024-02-05T04:53:38.593 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.593 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.593 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:38.593 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.137316+0000", 2024-02-05T04:53:38.593 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0019360930000000001 2024-02-05T04:53:38.593 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.594 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.594 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:38.594 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.137358+0000", 2024-02-05T04:53:38.594 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2196000000000001e-05 2024-02-05T04:53:38.594 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:38.594 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:38.595 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:38.595 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.595 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.595 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.3s6 37/31 ECSubWrite(tid=30, reqid=client.4634.0:219, at_version=37'13, trim_to=0'0, roll_forward_to=37'10))", 2024-02-05T04:53:38.595 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:29.137475+0000", 2024-02-05T04:53:38.595 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 9.4284398790000008, 2024-02-05T04:53:38.595 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0022173890000000002, 2024-02-05T04:53:38.596 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:38.596 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:38.596 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:38.596 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.596 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:38.597 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.137475+0000", 2024-02-05T04:53:38.597 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:38.597 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.597 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.597 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:38.597 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.137475+0000", 2024-02-05T04:53:38.598 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:38.598 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.598 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.598 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:38.598 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.137476+0000", 2024-02-05T04:53:38.598 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0729999999999999e-06 2024-02-05T04:53:38.599 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.599 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.599 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:38.599 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.137869+0000", 2024-02-05T04:53:38.599 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00039313999999999999 2024-02-05T04:53:38.599 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.600 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.600 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:38.600 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.137912+0000", 2024-02-05T04:53:38.600 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2379999999999997e-05 2024-02-05T04:53:38.600 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.600 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.601 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:38.601 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.137925+0000", 2024-02-05T04:53:38.601 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2948999999999999e-05 2024-02-05T04:53:38.601 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.601 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.601 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:38.602 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.137998+0000", 2024-02-05T04:53:38.602 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2995000000000002e-05 2024-02-05T04:53:38.602 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.602 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.602 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:38.602 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.138008+0000", 2024-02-05T04:53:38.603 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0104e-05 2024-02-05T04:53:38.603 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.603 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.603 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:38.603 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.139653+0000", 2024-02-05T04:53:38.603 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0016451009999999999 2024-02-05T04:53:38.603 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.604 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.604 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:38.604 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.139693+0000", 2024-02-05T04:53:38.604 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9647000000000002e-05 2024-02-05T04:53:38.604 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:38.604 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:38.605 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:38.605 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.605 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.605 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.3s6 37/31 ECSubWrite(tid=31, reqid=client.4634.0:220, at_version=37'14, trim_to=0'0, roll_forward_to=37'10))", 2024-02-05T04:53:38.605 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:29.138137+0000", 2024-02-05T04:53:38.605 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 9.4277779820000003, 2024-02-05T04:53:38.606 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0041763130000000001, 2024-02-05T04:53:38.606 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:38.606 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:38.606 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:38.606 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.606 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:38.607 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.138137+0000", 2024-02-05T04:53:38.607 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:38.607 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.607 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.607 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:38.607 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.138137+0000", 2024-02-05T04:53:38.607 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:38.608 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.608 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.608 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:38.608 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.138138+0000", 2024-02-05T04:53:38.608 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.37e-07 2024-02-05T04:53:38.608 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.608 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.608 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:38.608 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.138156+0000", 2024-02-05T04:53:38.609 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8448999999999999e-05 2024-02-05T04:53:38.609 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.609 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.609 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:38.609 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.138167+0000", 2024-02-05T04:53:38.609 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0930999999999999e-05 2024-02-05T04:53:38.609 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.609 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.609 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:38.610 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.138176+0000", 2024-02-05T04:53:38.610 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1600000000000004e-06 2024-02-05T04:53:38.610 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.610 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.610 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:38.610 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.138835+0000", 2024-02-05T04:53:38.610 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00065867499999999995 2024-02-05T04:53:38.610 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.610 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.611 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:38.611 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.138845+0000", 2024-02-05T04:53:38.611 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8109999999999997e-06 2024-02-05T04:53:38.611 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.611 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.611 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:38.611 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.142268+0000", 2024-02-05T04:53:38.612 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.003423059 2024-02-05T04:53:38.612 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.612 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.612 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:38.612 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.142313+0000", 2024-02-05T04:53:38.612 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5491000000000002e-05 2024-02-05T04:53:38.612 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:38.612 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:38.612 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:38.612 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.612 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.612 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.ds1 37/31 ECSubWrite(tid=23, reqid=client.4634.0:225, at_version=37'14, trim_to=0'0, roll_forward_to=34'12))", 2024-02-05T04:53:38.612 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:29.146330+0000", 2024-02-05T04:53:38.612 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 9.4195850369999992, 2024-02-05T04:53:38.613 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0019783069999999999, 2024-02-05T04:53:38.613 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:38.613 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:38.613 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:38.613 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.613 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:38.613 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.146330+0000", 2024-02-05T04:53:38.613 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:38.613 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.613 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.613 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:38.613 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.146330+0000", 2024-02-05T04:53:38.613 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:38.613 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.613 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.614 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:38.614 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.146330+0000", 2024-02-05T04:53:38.614 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2100000000000001e-07 2024-02-05T04:53:38.614 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.614 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.614 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:38.614 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.146490+0000", 2024-02-05T04:53:38.614 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00016007900000000001 2024-02-05T04:53:38.614 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.614 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.614 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:38.614 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.146512+0000", 2024-02-05T04:53:38.614 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2036999999999999e-05 2024-02-05T04:53:38.614 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.614 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.614 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:38.615 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.146524+0000", 2024-02-05T04:53:38.615 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2051e-05 2024-02-05T04:53:38.615 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.615 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.615 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:38.615 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.146629+0000", 2024-02-05T04:53:38.615 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010429699999999999 2024-02-05T04:53:38.615 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.615 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.615 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:38.615 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.146640+0000", 2024-02-05T04:53:38.615 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1396000000000001e-05 2024-02-05T04:53:38.615 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.615 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.615 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:38.616 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.148265+0000", 2024-02-05T04:53:38.616 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001624489 2024-02-05T04:53:38.616 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.616 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.616 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:38.616 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.148308+0000", 2024-02-05T04:53:38.616 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3736999999999998e-05 2024-02-05T04:53:38.616 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:38.616 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:38.616 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:38.616 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.616 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.616 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.2s2 37/31 ECSubWrite(tid=22, reqid=client.4634.0:254, at_version=37'15, trim_to=0'0, roll_forward_to=37'10))", 2024-02-05T04:53:38.617 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:29.208534+0000", 2024-02-05T04:53:38.617 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 9.3573808449999998, 2024-02-05T04:53:38.617 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0022982129999999999, 2024-02-05T04:53:38.617 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:38.617 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:38.617 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:38.617 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.617 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:38.617 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.208534+0000", 2024-02-05T04:53:38.617 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:38.617 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.617 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.617 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:38.617 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.208534+0000", 2024-02-05T04:53:38.617 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:38.617 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.618 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.618 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:38.618 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.208535+0000", 2024-02-05T04:53:38.618 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6599999999999996e-07 2024-02-05T04:53:38.618 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.618 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.618 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:38.618 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.208746+0000", 2024-02-05T04:53:38.618 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000211418 2024-02-05T04:53:38.618 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.618 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.618 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:38.618 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.208757+0000", 2024-02-05T04:53:38.618 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0878e-05 2024-02-05T04:53:38.619 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.619 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.619 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:38.619 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.208763+0000", 2024-02-05T04:53:38.619 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.75e-06 2024-02-05T04:53:38.619 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.619 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.619 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:38.619 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.208815+0000", 2024-02-05T04:53:38.619 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2169999999999997e-05 2024-02-05T04:53:38.619 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.619 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.619 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:38.619 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.208825+0000", 2024-02-05T04:53:38.619 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8120000000000002e-06 2024-02-05T04:53:38.619 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.620 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.620 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:38.620 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.210797+0000", 2024-02-05T04:53:38.620 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0019724500000000002 2024-02-05T04:53:38.620 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.620 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.620 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:38.620 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.210832+0000", 2024-02-05T04:53:38.620 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5169000000000002e-05 2024-02-05T04:53:38.620 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:38.620 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:38.621 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:38.621 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.621 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.621 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.2s2 37/31 ECSubWrite(tid=24, reqid=client.4634.0:257, at_version=37'18, trim_to=0'0, roll_forward_to=37'12))", 2024-02-05T04:53:38.621 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:29.211387+0000", 2024-02-05T04:53:38.621 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 9.3545277680000005, 2024-02-05T04:53:38.621 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0043575100000000002, 2024-02-05T04:53:38.621 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:38.621 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:38.621 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:38.621 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.621 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:38.621 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.211387+0000", 2024-02-05T04:53:38.621 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:38.621 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.622 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.622 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:38.622 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.211387+0000", 2024-02-05T04:53:38.622 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:38.622 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.622 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.622 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:38.622 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.211388+0000", 2024-02-05T04:53:38.622 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2200000000000004e-07 2024-02-05T04:53:38.622 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.622 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.622 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:38.622 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.211406+0000", 2024-02-05T04:53:38.622 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7796999999999999e-05 2024-02-05T04:53:38.622 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.622 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.623 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:38.623 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.211416+0000", 2024-02-05T04:53:38.623 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0234000000000001e-05 2024-02-05T04:53:38.623 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.623 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.623 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:38.623 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.211421+0000", 2024-02-05T04:53:38.623 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.609e-06 2024-02-05T04:53:38.623 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.623 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.623 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:38.623 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.211467+0000", 2024-02-05T04:53:38.623 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5271000000000001e-05 2024-02-05T04:53:38.623 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.623 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.624 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:38.624 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.211473+0000", 2024-02-05T04:53:38.624 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5549999999999999e-06 2024-02-05T04:53:38.624 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.624 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.624 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:38.624 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.215717+0000", 2024-02-05T04:53:38.624 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.004243592 2024-02-05T04:53:38.624 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.624 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.624 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:38.624 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.215745+0000", 2024-02-05T04:53:38.624 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.7929999999999999e-05 2024-02-05T04:53:38.624 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:38.625 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:38.625 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:38.625 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.625 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.625 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.2s2 37/31 ECSubWrite(tid=26, reqid=client.4634.0:257, at_version=0'0, trim_to=0'0, roll_forward_to=37'18))", 2024-02-05T04:53:38.625 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:29.216477+0000", 2024-02-05T04:53:38.625 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 9.3494380859999993, 2024-02-05T04:53:38.625 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0023123649999999998, 2024-02-05T04:53:38.625 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:38.625 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:38.625 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:38.625 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.625 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:38.626 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.216477+0000", 2024-02-05T04:53:38.626 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:38.626 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.626 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.626 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:38.626 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.216477+0000", 2024-02-05T04:53:38.626 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:38.626 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.626 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.626 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:38.626 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.216478+0000", 2024-02-05T04:53:38.626 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6499999999999999e-07 2024-02-05T04:53:38.626 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.626 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.626 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:38.627 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.216485+0000", 2024-02-05T04:53:38.627 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9529999999999999e-06 2024-02-05T04:53:38.627 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.627 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.627 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:38.627 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.216495+0000", 2024-02-05T04:53:38.627 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.9620000000000005e-06 2024-02-05T04:53:38.627 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.627 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.627 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:38.627 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.216504+0000", 2024-02-05T04:53:38.627 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.5639999999999997e-06 2024-02-05T04:53:38.627 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.627 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.627 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:38.627 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.216562+0000", 2024-02-05T04:53:38.628 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7837999999999997e-05 2024-02-05T04:53:38.628 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.628 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.628 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:38.628 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.216570+0000", 2024-02-05T04:53:38.628 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.1510000000000004e-06 2024-02-05T04:53:38.628 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.628 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.628 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:38.628 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.218758+0000", 2024-02-05T04:53:38.628 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0021876690000000002 2024-02-05T04:53:38.628 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.628 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.628 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:38.629 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.218789+0000", 2024-02-05T04:53:38.629 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.1662999999999998e-05 2024-02-05T04:53:38.629 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:38.629 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:38.629 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:38.629 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.629 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.629 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.ds1 38/31 ECSubWrite(tid=26, reqid=osd.7.0:26, at_version=38'16, trim_to=0'0, roll_forward_to=37'14))", 2024-02-05T04:53:38.629 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:30.134683+0000", 2024-02-05T04:53:38.629 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 8.4312324029999992, 2024-02-05T04:53:38.629 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0018417500000000001, 2024-02-05T04:53:38.629 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:38.629 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:38.629 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:38.629 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.630 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:38.630 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.134683+0000", 2024-02-05T04:53:38.630 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:38.630 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.630 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.630 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:38.630 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.134683+0000", 2024-02-05T04:53:38.630 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:38.630 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.630 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.630 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:38.630 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.134683+0000", 2024-02-05T04:53:38.630 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9799999999999999e-07 2024-02-05T04:53:38.630 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.630 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.630 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:38.631 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.134707+0000", 2024-02-05T04:53:38.631 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.4162e-05 2024-02-05T04:53:38.631 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.631 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.631 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:38.631 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.134720+0000", 2024-02-05T04:53:38.631 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2847000000000001e-05 2024-02-05T04:53:38.631 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.632 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.632 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:38.632 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.134741+0000", 2024-02-05T04:53:38.632 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1331e-05 2024-02-05T04:53:38.632 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.632 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.632 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:38.633 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.134833+0000", 2024-02-05T04:53:38.633 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1556000000000002e-05 2024-02-05T04:53:38.633 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.633 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.633 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:38.633 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.134860+0000", 2024-02-05T04:53:38.633 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.7073999999999999e-05 2024-02-05T04:53:38.634 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.634 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.634 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:38.634 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.136485+0000", 2024-02-05T04:53:38.634 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001624732 2024-02-05T04:53:38.634 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.634 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.635 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:38.635 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.136524+0000", 2024-02-05T04:53:38.635 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9650000000000002e-05 2024-02-05T04:53:38.635 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:38.635 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:38.635 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:38.635 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.636 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.636 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.2s2 38/31 ECSubWrite(tid=27, reqid=osd.12.0:27, at_version=38'20, trim_to=0'0, roll_forward_to=37'18))", 2024-02-05T04:53:38.636 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:30.143602+0000", 2024-02-05T04:53:38.636 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 8.4223131050000006, 2024-02-05T04:53:38.636 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0021805349999999999, 2024-02-05T04:53:38.636 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:38.636 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:38.637 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:38.637 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.637 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:38.637 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.143602+0000", 2024-02-05T04:53:38.637 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:38.637 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.637 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.637 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:38.637 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.143602+0000", 2024-02-05T04:53:38.638 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:38.638 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.638 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.638 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:38.638 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.143602+0000", 2024-02-05T04:53:38.638 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2399999999999999e-07 2024-02-05T04:53:38.639 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.639 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.639 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:38.639 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.143626+0000", 2024-02-05T04:53:38.639 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3947e-05 2024-02-05T04:53:38.639 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.639 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.640 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:38.640 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.143640+0000", 2024-02-05T04:53:38.640 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3679999999999999e-05 2024-02-05T04:53:38.640 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.640 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.640 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:38.640 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.143648+0000", 2024-02-05T04:53:38.641 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0919999999999998e-06 2024-02-05T04:53:38.641 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.641 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.641 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:38.641 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.143722+0000", 2024-02-05T04:53:38.641 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4177999999999994e-05 2024-02-05T04:53:38.641 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.641 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.641 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:38.641 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.143733+0000", 2024-02-05T04:53:38.641 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0596e-05 2024-02-05T04:53:38.641 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.642 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.642 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:38.642 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.145745+0000", 2024-02-05T04:53:38.642 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0020118879999999999 2024-02-05T04:53:38.642 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.642 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.642 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:38.642 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.145782+0000", 2024-02-05T04:53:38.642 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7929999999999998e-05 2024-02-05T04:53:38.642 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:38.642 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:38.642 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:38.642 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.642 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.642 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.2s2 38/31 ECSubWrite(tid=28, reqid=osd.12.0:28, at_version=38'22, trim_to=0'0, roll_forward_to=37'18))", 2024-02-05T04:53:38.642 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:30.144310+0000", 2024-02-05T04:53:38.643 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 8.4216046389999999, 2024-02-05T04:53:38.643 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0042662150000000003, 2024-02-05T04:53:38.643 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:38.643 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:38.643 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:38.643 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.643 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:38.643 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.144310+0000", 2024-02-05T04:53:38.643 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:38.643 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.643 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.643 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:38.643 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.144310+0000", 2024-02-05T04:53:38.643 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:38.643 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.644 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.644 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:38.644 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.144311+0000", 2024-02-05T04:53:38.644 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4499999999999996e-07 2024-02-05T04:53:38.644 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.644 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.644 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:38.644 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.144332+0000", 2024-02-05T04:53:38.644 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0954999999999998e-05 2024-02-05T04:53:38.644 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.644 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.644 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:38.644 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.144345+0000", 2024-02-05T04:53:38.644 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.259e-05 2024-02-05T04:53:38.645 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.645 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.645 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:38.645 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.144354+0000", 2024-02-05T04:53:38.645 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.7110000000000001e-06 2024-02-05T04:53:38.645 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.645 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.645 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:38.645 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.144438+0000", 2024-02-05T04:53:38.645 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4466000000000001e-05 2024-02-05T04:53:38.645 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.645 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.645 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:38.645 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.144448+0000", 2024-02-05T04:53:38.645 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0419e-05 2024-02-05T04:53:38.646 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.646 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.646 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:38.646 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.148538+0000", 2024-02-05T04:53:38.646 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0040892769999999997 2024-02-05T04:53:38.646 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.646 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.646 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:38.646 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.148577+0000", 2024-02-05T04:53:38.646 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8952000000000002e-05 2024-02-05T04:53:38.646 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:38.646 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:38.646 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:38.646 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.646 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.647 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.2s2 38/31 ECSubWrite(tid=29, reqid=osd.12.0:28, at_version=0'0, trim_to=0'0, roll_forward_to=38'22))", 2024-02-05T04:53:38.647 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:30.154256+0000", 2024-02-05T04:53:38.647 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 8.4116592410000006, 2024-02-05T04:53:38.647 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002332673, 2024-02-05T04:53:38.647 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:38.647 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:38.647 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:38.647 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.647 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:38.647 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.154256+0000", 2024-02-05T04:53:38.647 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:38.647 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.647 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.647 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:38.647 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.154256+0000", 2024-02-05T04:53:38.648 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:38.648 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.648 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.648 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:38.648 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.154257+0000", 2024-02-05T04:53:38.648 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.062e-06 2024-02-05T04:53:38.648 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.648 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.648 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:38.648 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.154269+0000", 2024-02-05T04:53:38.648 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2147000000000001e-05 2024-02-05T04:53:38.648 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.648 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.648 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:38.649 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.154293+0000", 2024-02-05T04:53:38.649 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.4304999999999999e-05 2024-02-05T04:53:38.649 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.649 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.649 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:38.649 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.154303+0000", 2024-02-05T04:53:38.649 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5759999999999995e-06 2024-02-05T04:53:38.649 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.649 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.649 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:38.649 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.154372+0000", 2024-02-05T04:53:38.649 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9572000000000004e-05 2024-02-05T04:53:38.649 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.649 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.650 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:38.650 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.154382+0000", 2024-02-05T04:53:38.650 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.679e-06 2024-02-05T04:53:38.650 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.650 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.650 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:38.650 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.156518+0000", 2024-02-05T04:53:38.650 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002136097 2024-02-05T04:53:38.650 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.650 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.650 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:38.650 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.156588+0000", 2024-02-05T04:53:38.650 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0234999999999994e-05 2024-02-05T04:53:38.650 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:38.650 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:38.651 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:38.651 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.651 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.651 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.2s2 38/31 ECSubWrite(tid=30, reqid=osd.12.0:30, at_version=38'24, trim_to=0'0, roll_forward_to=38'22))", 2024-02-05T04:53:38.651 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:30.155268+0000", 2024-02-05T04:53:38.651 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 8.4106472189999995, 2024-02-05T04:53:38.651 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0043229849999999997, 2024-02-05T04:53:38.651 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:38.651 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:38.651 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:38.651 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.651 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:38.651 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.155268+0000", 2024-02-05T04:53:38.651 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:38.651 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.651 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.652 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:38.652 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.155268+0000", 2024-02-05T04:53:38.652 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:38.652 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.652 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.652 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:38.652 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.155269+0000", 2024-02-05T04:53:38.652 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.33e-07 2024-02-05T04:53:38.652 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.652 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.652 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:38.652 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.155291+0000", 2024-02-05T04:53:38.652 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2708e-05 2024-02-05T04:53:38.652 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.653 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.653 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:38.653 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.155308+0000", 2024-02-05T04:53:38.653 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6996000000000001e-05 2024-02-05T04:53:38.653 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.653 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.653 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:38.653 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.155318+0000", 2024-02-05T04:53:38.653 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8849999999999994e-06 2024-02-05T04:53:38.653 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.653 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.653 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:38.653 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.155388+0000", 2024-02-05T04:53:38.653 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9704999999999998e-05 2024-02-05T04:53:38.653 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.654 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.654 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:38.654 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.155399+0000", 2024-02-05T04:53:38.654 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0473e-05 2024-02-05T04:53:38.654 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.654 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.654 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:38.654 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.159532+0000", 2024-02-05T04:53:38.654 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0041339180000000003 2024-02-05T04:53:38.654 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.654 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.654 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:38.654 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.159591+0000", 2024-02-05T04:53:38.654 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8366999999999999e-05 2024-02-05T04:53:38.654 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:38.654 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:38.655 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:38.655 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.655 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.655 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.ds1 38/31 ECSubWrite(tid=28, reqid=client.4634.0:263, at_version=38'17, trim_to=0'0, roll_forward_to=38'16))", 2024-02-05T04:53:38.655 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:30.173452+0000", 2024-02-05T04:53:38.655 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 8.3924627320000003, 2024-02-05T04:53:38.655 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001734462, 2024-02-05T04:53:38.655 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:38.655 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:38.655 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:38.655 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.655 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:38.655 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.173452+0000", 2024-02-05T04:53:38.655 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:38.655 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.656 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.656 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:38.656 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.173452+0000", 2024-02-05T04:53:38.656 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:38.656 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.656 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.656 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:38.656 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.173453+0000", 2024-02-05T04:53:38.656 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.540000000000001e-07 2024-02-05T04:53:38.656 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.656 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.656 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:38.656 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.173738+0000", 2024-02-05T04:53:38.656 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00028479900000000002 2024-02-05T04:53:38.657 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.657 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.657 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:38.657 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.173761+0000", 2024-02-05T04:53:38.657 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2721000000000002e-05 2024-02-05T04:53:38.657 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.657 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.657 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:38.657 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.173781+0000", 2024-02-05T04:53:38.657 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.035e-05 2024-02-05T04:53:38.658 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.658 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.658 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:38.658 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.173883+0000", 2024-02-05T04:53:38.658 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000102098 2024-02-05T04:53:38.658 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.658 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.658 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:38.658 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.173904+0000", 2024-02-05T04:53:38.658 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1047999999999999e-05 2024-02-05T04:53:38.658 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.658 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.658 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:38.658 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.175143+0000", 2024-02-05T04:53:38.658 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001239108 2024-02-05T04:53:38.659 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.659 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.659 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:38.659 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.175187+0000", 2024-02-05T04:53:38.659 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3383999999999997e-05 2024-02-05T04:53:38.659 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:38.659 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:38.659 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:38.659 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.659 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.659 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.2s2 42/31 ECSubWrite(tid=46, reqid=client.4634.0:381, at_version=0'0, trim_to=0'0, roll_forward_to=42'35))", 2024-02-05T04:53:38.659 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:34.162004+0000", 2024-02-05T04:53:38.659 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 4.4039105899999997, 2024-02-05T04:53:38.659 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0025338520000000001, 2024-02-05T04:53:38.660 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:38.660 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:38.660 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:38.660 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.660 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:38.660 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.162004+0000", 2024-02-05T04:53:38.660 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:38.660 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.661 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.661 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:38.661 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.162004+0000", 2024-02-05T04:53:38.661 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:38.661 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.661 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.661 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:38.662 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.162005+0000", 2024-02-05T04:53:38.662 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.78e-07 2024-02-05T04:53:38.662 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.662 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.662 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:38.662 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.162025+0000", 2024-02-05T04:53:38.662 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9670999999999999e-05 2024-02-05T04:53:38.662 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.663 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.663 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:38.663 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.162051+0000", 2024-02-05T04:53:38.663 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5803e-05 2024-02-05T04:53:38.663 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.663 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.663 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:38.663 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.162062+0000", 2024-02-05T04:53:38.663 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0977e-05 2024-02-05T04:53:38.663 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.663 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.663 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:38.664 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.162151+0000", 2024-02-05T04:53:38.664 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.9441e-05 2024-02-05T04:53:38.664 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.664 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.664 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:38.664 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.162160+0000", 2024-02-05T04:53:38.664 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.6649999999999992e-06 2024-02-05T04:53:38.664 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.664 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.664 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:38.664 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.164512+0000", 2024-02-05T04:53:38.664 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0023523009999999998 2024-02-05T04:53:38.664 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.664 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.665 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:38.665 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.164538+0000", 2024-02-05T04:53:38.665 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.6115999999999999e-05 2024-02-05T04:53:38.665 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:38.665 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:38.665 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:38.665 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.665 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.665 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "pg_update_log_missing(3.2s2 epoch 43/31 rep_tid 47 entries 43'36 (0'0) error 3:4d05e2f1:::smithi19639362-34:head by client.4634.0:397 0.000000 -2 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 39'33)", 2024-02-05T04:53:38.665 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:35.163812+0000", 2024-02-05T04:53:38.665 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 3.4021034509999999, 2024-02-05T04:53:38.665 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0028187389999999998, 2024-02-05T04:53:38.665 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:38.665 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:38.666 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:38.666 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.666 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:38.666 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.163812+0000", 2024-02-05T04:53:38.666 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:38.666 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.666 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.666 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:38.666 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.163812+0000", 2024-02-05T04:53:38.666 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:38.666 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.666 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.666 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:38.666 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.163812+0000", 2024-02-05T04:53:38.667 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4100000000000001e-07 2024-02-05T04:53:38.667 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.667 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.667 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:38.667 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.163833+0000", 2024-02-05T04:53:38.667 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1659e-05 2024-02-05T04:53:38.667 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.667 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.667 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:38.667 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.163861+0000", 2024-02-05T04:53:38.667 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.7149999999999999e-05 2024-02-05T04:53:38.667 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.667 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.667 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:38.667 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.163878+0000", 2024-02-05T04:53:38.668 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7371999999999999e-05 2024-02-05T04:53:38.668 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.668 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.668 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:38.668 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.163982+0000", 2024-02-05T04:53:38.668 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010451299999999999 2024-02-05T04:53:38.668 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.668 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.668 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:38.668 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.166630+0000", 2024-02-05T04:53:38.668 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002647904 2024-02-05T04:53:38.668 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:38.668 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:38.668 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:38.669 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.669 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.669 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.2s2 43/31 ECSubWrite(tid=48, reqid=osd.12.0:48, at_version=43'38, trim_to=0'0, roll_forward_to=42'35))", 2024-02-05T04:53:38.669 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:35.164524+0000", 2024-02-05T04:53:38.669 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 3.4013909689999999, 2024-02-05T04:53:38.669 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0049387320000000004, 2024-02-05T04:53:38.669 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:38.669 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:38.669 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:38.669 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.669 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:38.669 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.164524+0000", 2024-02-05T04:53:38.669 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:38.669 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.670 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.670 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:38.670 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.164524+0000", 2024-02-05T04:53:38.670 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:38.670 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.670 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.670 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:38.670 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.164525+0000", 2024-02-05T04:53:38.670 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.2200000000000003e-07 2024-02-05T04:53:38.670 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.670 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.670 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:38.670 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.164556+0000", 2024-02-05T04:53:38.670 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.1050000000000003e-05 2024-02-05T04:53:38.670 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.671 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.671 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:38.671 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.164572+0000", 2024-02-05T04:53:38.671 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6387000000000001e-05 2024-02-05T04:53:38.671 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.671 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.671 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:38.671 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.164582+0000", 2024-02-05T04:53:38.671 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2050000000000008e-06 2024-02-05T04:53:38.671 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.671 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.671 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:38.671 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.164651+0000", 2024-02-05T04:53:38.671 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9219000000000003e-05 2024-02-05T04:53:38.671 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.672 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.672 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:38.672 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.164661+0000", 2024-02-05T04:53:38.672 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0210999999999999e-05 2024-02-05T04:53:38.672 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.672 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.672 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:38.672 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.169414+0000", 2024-02-05T04:53:38.672 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0047528470000000001 2024-02-05T04:53:38.672 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.672 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.672 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:38.672 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.169463+0000", 2024-02-05T04:53:38.672 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8990999999999999e-05 2024-02-05T04:53:38.673 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:38.673 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:38.673 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:38.673 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.673 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.673 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.2s2 43/31 ECSubWrite(tid=49, reqid=osd.12.0:48, at_version=0'0, trim_to=0'0, roll_forward_to=43'38))", 2024-02-05T04:53:38.673 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:35.170040+0000", 2024-02-05T04:53:38.673 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 3.3958754660000001, 2024-02-05T04:53:38.673 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.004631652, 2024-02-05T04:53:38.673 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:38.673 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:38.673 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:38.673 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.673 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:38.673 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.170040+0000", 2024-02-05T04:53:38.673 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:38.674 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.674 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.674 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:38.674 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.170040+0000", 2024-02-05T04:53:38.674 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:38.674 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.674 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.674 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:38.674 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.170040+0000", 2024-02-05T04:53:38.674 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.0599999999999999e-07 2024-02-05T04:53:38.674 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.674 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.674 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:38.674 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.170053+0000", 2024-02-05T04:53:38.674 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2179e-05 2024-02-05T04:53:38.675 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.675 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.675 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:38.675 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.170066+0000", 2024-02-05T04:53:38.675 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2875e-05 2024-02-05T04:53:38.675 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.675 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.675 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:38.675 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.170074+0000", 2024-02-05T04:53:38.675 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.7760000000000003e-06 2024-02-05T04:53:38.675 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.675 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.675 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:38.675 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.170139+0000", 2024-02-05T04:53:38.675 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5086000000000007e-05 2024-02-05T04:53:38.675 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.676 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.676 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:38.676 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.170149+0000", 2024-02-05T04:53:38.676 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0045e-05 2024-02-05T04:53:38.676 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.676 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.676 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:38.676 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.174632+0000", 2024-02-05T04:53:38.676 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0044826570000000001 2024-02-05T04:53:38.676 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.676 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.676 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:38.676 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.174671+0000", 2024-02-05T04:53:38.676 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9128000000000002e-05 2024-02-05T04:53:38.677 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:38.677 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:38.677 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:38.677 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.677 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.677 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.ds1 43/31 ECSubWrite(tid=44, reqid=client.4634.0:401, at_version=43'24, trim_to=0'0, roll_forward_to=38'22))", 2024-02-05T04:53:38.677 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:35.183587+0000", 2024-02-05T04:53:38.677 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 3.3823279670000002, 2024-02-05T04:53:38.677 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0048306419999999996, 2024-02-05T04:53:38.677 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:38.677 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:38.677 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:38.677 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.677 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:38.677 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.183587+0000", 2024-02-05T04:53:38.678 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:38.678 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.678 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.678 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:38.678 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.183587+0000", 2024-02-05T04:53:38.678 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:38.678 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.678 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.678 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:38.678 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.183588+0000", 2024-02-05T04:53:38.678 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5999999999999999e-07 2024-02-05T04:53:38.678 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.678 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.678 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:38.678 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.183610+0000", 2024-02-05T04:53:38.679 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2606e-05 2024-02-05T04:53:38.679 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.679 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.679 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:38.679 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.183623+0000", 2024-02-05T04:53:38.679 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2461e-05 2024-02-05T04:53:38.679 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.679 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.679 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:38.679 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.183631+0000", 2024-02-05T04:53:38.679 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8249999999999994e-06 2024-02-05T04:53:38.679 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.679 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.679 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:38.680 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.183703+0000", 2024-02-05T04:53:38.680 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1551999999999995e-05 2024-02-05T04:53:38.680 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.680 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.680 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:38.680 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.183713+0000", 2024-02-05T04:53:38.680 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0472000000000001e-05 2024-02-05T04:53:38.680 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.680 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.680 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:38.680 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.188379+0000", 2024-02-05T04:53:38.680 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0046652819999999998 2024-02-05T04:53:38.680 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:38.680 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:38.681 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:38.681 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.188418+0000", 2024-02-05T04:53:38.681 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8983999999999998e-05 2024-02-05T04:53:38.681 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:38.681 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:38.681 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:38.681 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:38.681 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:38.681 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T04:53:38.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-02-05T04:53:38.646+0000 7f852e0f1640 0 -- 172.21.15.62:0/1512866519 >> [v2:172.21.15.91:6809/3656255795,v1:172.21.15.91:6812/3656255795] conn(0x7f852821c100 msgr2=0x7f852821e4f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-02-05T04:53:38.683 INFO:tasks.ceph.osd.14.smithi150.stderr:2024-02-05T04:53:38.681+0000 7f5f81083640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 34020) UID: 0 2024-02-05T04:53:38.683 INFO:teuthology.orchestra.run.smithi062.stdout:osd.13: {} 2024-02-05T04:53:38.684 INFO:teuthology.orchestra.run.smithi062.stderr:osd.13: osd_enable_op_tracker = '' 2024-02-05T04:53:38.685 INFO:teuthology.orchestra.run.smithi062.stdout:osd.14: {} 2024-02-05T04:53:38.685 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-02-05T04:53:38.686 INFO:teuthology.orchestra.run.smithi062.stderr:osd.14: osd_enable_op_tracker = '' 2024-02-05T04:53:38.700 INFO:teuthology.orchestra.run.smithi062.stdout:osd.15: {} 2024-02-05T04:53:38.700 INFO:teuthology.orchestra.run.smithi062.stderr:osd.15: osd_enable_op_tracker = '' 2024-02-05T04:53:38.783 INFO:tasks.ceph.osd.1.smithi091.stderr:2024-02-05T04:53:38.782+0000 7ff9dc283640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34275) UID: 0 2024-02-05T04:53:38.792 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T04:53:38.792 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-02-05T04:53:38.793 INFO:teuthology.orchestra.run.smithi196.stdout: "num_ops": 0 2024-02-05T04:53:38.793 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T04:53:38.802 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-02-05T04:53:38.884 INFO:tasks.ceph.osd.15.smithi196.stderr:2024-02-05T04:53:38.883+0000 7f1facc45640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 34611) UID: 0 2024-02-05T04:53:38.908 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T04:53:38.908 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-02-05T04:53:38.908 INFO:teuthology.orchestra.run.smithi196.stdout: "complaint_time": 30, 2024-02-05T04:53:38.908 INFO:teuthology.orchestra.run.smithi196.stdout: "num_blocked_ops": 0 2024-02-05T04:53:38.908 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T04:53:38.916 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-02-05T04:53:38.938 INFO:teuthology.orchestra.run.smithi062.stderr:noscrub is unset 2024-02-05T04:53:38.941 INFO:tasks.rados.rados.0.smithi196.stdout:157: done (1 left) 2024-02-05T04:53:38.942 INFO:tasks.rados.rados.0.smithi196.stdout:158: done (0 left) 2024-02-05T04:53:38.942 INFO:tasks.rados.rados.0.smithi196.stdout:159: read oid 20 snap -1 2024-02-05T04:53:38.942 INFO:tasks.rados.rados.0.smithi196.stdout:159: expect deleted 2024-02-05T04:53:38.942 INFO:tasks.rados.rados.0.smithi196.stdout:160: setattr oid 16 current snap is 8 2024-02-05T04:53:38.944 INFO:tasks.rados.rados.0.smithi196.stdout:161: read oid 33 snap -1 2024-02-05T04:53:38.944 INFO:tasks.rados.rados.0.smithi196.stdout:161: expect (ObjNum 32 snap 0 seq_num 32) 2024-02-05T04:53:38.944 INFO:tasks.rados.rados.0.smithi196.stdout:162: snap_create 2024-02-05T04:53:38.949 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 16 v 33 (ObjNum 60 snap 2 seq_num 60) dirty exists 2024-02-05T04:53:38.951 INFO:tasks.rados.rados.0.smithi196.stdout:161: expect (ObjNum 32 snap 0 seq_num 32) 2024-02-05T04:53:38.985 INFO:tasks.ceph.osd.13.smithi091.stderr:2024-02-05T04:53:38.983+0000 7f526fd4d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 34281) UID: 0 2024-02-05T04:53:39.050 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T04:53:39.050 INFO:teuthology.orchestra.run.smithi196.stdout: "size": 20, 2024-02-05T04:53:39.050 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 600, 2024-02-05T04:53:39.050 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-02-05T04:53:39.051 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.051 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:225 3.ds0 3:b4e8e82c:::smithi19639362-25:head [copy-from ver 6] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:53:39.051 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:29.135146+0000", 2024-02-05T04:53:39.051 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 9.9132749409999992, 2024-02-05T04:53:39.051 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.016925534999999998, 2024-02-05T04:53:39.051 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:39.051 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:39.051 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:53:39.051 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:53:39.051 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:39.051 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 225 2024-02-05T04:53:39.051 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.051 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:39.051 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.052 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:39.052 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135146+0000", 2024-02-05T04:53:39.052 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:39.052 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.052 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.052 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:39.052 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135146+0000", 2024-02-05T04:53:39.052 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:39.052 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.052 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.052 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:39.052 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135148+0000", 2024-02-05T04:53:39.052 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5170000000000002e-06 2024-02-05T04:53:39.052 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.053 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.053 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:39.053 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135158+0000", 2024-02-05T04:53:39.053 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.3819999999999993e-06 2024-02-05T04:53:39.053 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.053 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.053 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:39.053 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135177+0000", 2024-02-05T04:53:39.053 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9513000000000001e-05 2024-02-05T04:53:39.053 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.053 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.053 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:39.053 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135187+0000", 2024-02-05T04:53:39.053 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0054000000000001e-05 2024-02-05T04:53:39.053 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.054 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.054 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:39.054 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135292+0000", 2024-02-05T04:53:39.054 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000104651 2024-02-05T04:53:39.054 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.054 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.054 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:53:39.054 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135405+0000", 2024-02-05T04:53:39.054 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011277099999999999 2024-02-05T04:53:39.054 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.054 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.054 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:39.054 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.146446+0000", 2024-02-05T04:53:39.054 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.011041535 2024-02-05T04:53:39.054 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.055 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.055 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:39.055 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.148233+0000", 2024-02-05T04:53:39.055 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001787089 2024-02-05T04:53:39.055 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.055 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.055 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T04:53:39.055 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.151960+0000", 2024-02-05T04:53:39.055 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0037264720000000002 2024-02-05T04:53:39.055 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.055 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.055 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:39.055 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.152071+0000", 2024-02-05T04:53:39.055 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000111551 2024-02-05T04:53:39.055 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.056 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:39.056 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.056 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.056 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.056 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:226 3.ds0 3:b4e8e82c:::smithi19639362-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:53:39.056 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:29.135222+0000", 2024-02-05T04:53:39.056 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 9.9131984210000006, 2024-02-05T04:53:39.056 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.012833986, 2024-02-05T04:53:39.056 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:39.056 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-02-05T04:53:39.056 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:53:39.056 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:53:39.056 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:39.056 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 226 2024-02-05T04:53:39.057 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.057 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:39.057 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.057 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:39.057 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135222+0000", 2024-02-05T04:53:39.057 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:39.057 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.057 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.057 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:39.057 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135222+0000", 2024-02-05T04:53:39.057 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:39.057 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.057 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.057 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:39.057 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135223+0000", 2024-02-05T04:53:39.058 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4099999999999998e-07 2024-02-05T04:53:39.058 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.058 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.058 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:39.058 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135227+0000", 2024-02-05T04:53:39.058 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8399999999999997e-06 2024-02-05T04:53:39.058 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.058 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.058 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:39.058 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135237+0000", 2024-02-05T04:53:39.058 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8290000000000003e-06 2024-02-05T04:53:39.058 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.058 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.058 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:39.058 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135244+0000", 2024-02-05T04:53:39.059 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4989999999999997e-06 2024-02-05T04:53:39.059 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.059 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.059 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:39.059 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135624+0000", 2024-02-05T04:53:39.059 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00037992100000000002 2024-02-05T04:53:39.059 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.059 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.059 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for blocked object", 2024-02-05T04:53:39.059 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135672+0000", 2024-02-05T04:53:39.059 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7861999999999997e-05 2024-02-05T04:53:39.059 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.059 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.059 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:39.059 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.147862+0000", 2024-02-05T04:53:39.059 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.012189847 2024-02-05T04:53:39.060 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.060 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.060 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:53:39.060 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.147947+0000", 2024-02-05T04:53:39.060 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4862000000000002e-05 2024-02-05T04:53:39.060 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.060 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.060 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:39.060 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.148056+0000", 2024-02-05T04:53:39.060 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000109585 2024-02-05T04:53:39.060 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.060 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:39.060 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.060 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.061 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.061 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.3s1 37/31 ECSubWrite(tid=33, reqid=client.4634.0:220, at_version=0'0, trim_to=0'0, roll_forward_to=37'14))", 2024-02-05T04:53:39.061 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:29.144152+0000", 2024-02-05T04:53:39.061 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 9.9042688929999994, 2024-02-05T04:53:39.061 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0036900819999999999, 2024-02-05T04:53:39.061 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:39.061 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:39.061 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:39.061 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.061 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:39.061 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.144152+0000", 2024-02-05T04:53:39.061 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:39.061 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.061 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.061 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:39.062 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.144152+0000", 2024-02-05T04:53:39.062 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:39.062 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.062 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.062 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:39.062 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.144153+0000", 2024-02-05T04:53:39.062 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0279999999999999e-06 2024-02-05T04:53:39.062 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.062 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.062 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:39.062 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.144165+0000", 2024-02-05T04:53:39.062 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2584e-05 2024-02-05T04:53:39.062 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.062 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.062 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:39.062 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.144184+0000", 2024-02-05T04:53:39.063 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8389e-05 2024-02-05T04:53:39.063 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.063 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.063 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:39.063 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.144197+0000", 2024-02-05T04:53:39.063 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3291e-05 2024-02-05T04:53:39.063 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.063 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.063 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:39.063 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.144268+0000", 2024-02-05T04:53:39.063 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0489999999999998e-05 2024-02-05T04:53:39.063 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.063 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.063 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:39.063 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.144276+0000", 2024-02-05T04:53:39.064 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.5359999999999992e-06 2024-02-05T04:53:39.064 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.064 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.064 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:39.064 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.147731+0000", 2024-02-05T04:53:39.064 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.003455175 2024-02-05T04:53:39.064 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.064 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.064 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:39.064 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.147842+0000", 2024-02-05T04:53:39.064 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011058899999999999 2024-02-05T04:53:39.064 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.064 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:39.064 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.065 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.065 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.065 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.3s1 38/31 ECSubWrite(tid=35, reqid=osd.4.0:35, at_version=38'16, trim_to=0'0, roll_forward_to=37'14))", 2024-02-05T04:53:39.065 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:30.131623+0000", 2024-02-05T04:53:39.065 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 8.9167974819999998, 2024-02-05T04:53:39.065 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0037957590000000001, 2024-02-05T04:53:39.065 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:39.065 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:39.065 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:39.065 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.065 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:39.065 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.131623+0000", 2024-02-05T04:53:39.065 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:39.065 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.066 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.066 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:39.066 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.131623+0000", 2024-02-05T04:53:39.066 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:39.066 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.066 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.066 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:39.066 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.131624+0000", 2024-02-05T04:53:39.066 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2999999999999999e-07 2024-02-05T04:53:39.066 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.066 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.066 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:39.066 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.131647+0000", 2024-02-05T04:53:39.067 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2982000000000001e-05 2024-02-05T04:53:39.067 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.067 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.067 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:39.067 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.131659+0000", 2024-02-05T04:53:39.067 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2371999999999999e-05 2024-02-05T04:53:39.067 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.067 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.067 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:39.067 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.131672+0000", 2024-02-05T04:53:39.067 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3108000000000001e-05 2024-02-05T04:53:39.067 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.067 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.067 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:39.068 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.131756+0000", 2024-02-05T04:53:39.068 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4287000000000001e-05 2024-02-05T04:53:39.068 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.068 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.068 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:39.068 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.131767+0000", 2024-02-05T04:53:39.068 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1113e-05 2024-02-05T04:53:39.068 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.068 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.068 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:39.068 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.135381+0000", 2024-02-05T04:53:39.068 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00361372 2024-02-05T04:53:39.068 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.068 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.069 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:39.069 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.135419+0000", 2024-02-05T04:53:39.069 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7947000000000001e-05 2024-02-05T04:53:39.069 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.069 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:39.069 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.069 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.069 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.069 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.6s5 38/31 ECSubWrite(tid=90, reqid=osd.13.0:90, at_version=38'11, trim_to=0'0, roll_forward_to=37'8))", 2024-02-05T04:53:39.069 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:30.142135+0000", 2024-02-05T04:53:39.069 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 8.9062853089999994, 2024-02-05T04:53:39.069 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0038445620000000002, 2024-02-05T04:53:39.070 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:39.070 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:39.070 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:39.070 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.070 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:39.070 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.142135+0000", 2024-02-05T04:53:39.070 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:39.070 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.070 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.070 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:39.070 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.142135+0000", 2024-02-05T04:53:39.070 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:39.070 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.070 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.070 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:39.071 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.142136+0000", 2024-02-05T04:53:39.071 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0400000000000005e-07 2024-02-05T04:53:39.071 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.071 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.071 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:39.071 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.142162+0000", 2024-02-05T04:53:39.071 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.6200999999999999e-05 2024-02-05T04:53:39.071 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.071 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.071 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:39.071 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.142176+0000", 2024-02-05T04:53:39.071 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3451e-05 2024-02-05T04:53:39.071 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.071 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.071 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:39.072 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.142190+0000", 2024-02-05T04:53:39.072 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4049000000000001e-05 2024-02-05T04:53:39.072 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.072 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.072 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:39.072 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.142273+0000", 2024-02-05T04:53:39.072 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3490000000000002e-05 2024-02-05T04:53:39.072 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.072 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.072 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:39.072 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.142285+0000", 2024-02-05T04:53:39.072 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1398e-05 2024-02-05T04:53:39.072 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.073 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.073 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:39.073 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.145940+0000", 2024-02-05T04:53:39.073 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.003655635 2024-02-05T04:53:39.073 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.073 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.073 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:39.073 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.145980+0000", 2024-02-05T04:53:39.073 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9533999999999998e-05 2024-02-05T04:53:39.073 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.073 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:39.073 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.073 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.073 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.073 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:263 3.ds0 3:b7ad4782:::smithi19639362-40:head [append 0~737280] snapc 3=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-02-05T04:53:39.074 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:30.172056+0000", 2024-02-05T04:53:39.074 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 8.8763648160000006, 2024-02-05T04:53:39.074 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0047829179999999997, 2024-02-05T04:53:39.074 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:39.074 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:39.074 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:53:39.074 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:53:39.074 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:39.074 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 263 2024-02-05T04:53:39.074 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.074 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:39.074 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.074 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:39.074 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.172056+0000", 2024-02-05T04:53:39.074 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:39.074 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.075 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.075 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:39.075 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.172056+0000", 2024-02-05T04:53:39.075 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:39.075 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.075 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.075 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:39.075 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.172058+0000", 2024-02-05T04:53:39.075 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0660000000000002e-06 2024-02-05T04:53:39.075 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.075 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.075 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:39.075 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.172410+0000", 2024-02-05T04:53:39.075 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00035195700000000002 2024-02-05T04:53:39.075 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.075 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.076 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:39.076 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.172441+0000", 2024-02-05T04:53:39.076 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.0676999999999998e-05 2024-02-05T04:53:39.076 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.076 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.076 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:39.076 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.172447+0000", 2024-02-05T04:53:39.076 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4250000000000003e-06 2024-02-05T04:53:39.076 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.076 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.076 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:39.076 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.172535+0000", 2024-02-05T04:53:39.076 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.7855000000000005e-05 2024-02-05T04:53:39.077 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.077 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.077 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:53:39.077 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.172613+0000", 2024-02-05T04:53:39.077 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8356999999999995e-05 2024-02-05T04:53:39.077 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.077 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.077 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:39.077 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.173406+0000", 2024-02-05T04:53:39.077 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00079290199999999995 2024-02-05T04:53:39.077 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.077 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.077 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:39.077 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.174300+0000", 2024-02-05T04:53:39.077 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00089424000000000003 2024-02-05T04:53:39.078 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.078 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.078 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T04:53:39.078 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.176774+0000", 2024-02-05T04:53:39.078 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0024735030000000002 2024-02-05T04:53:39.078 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.078 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.078 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:39.078 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.176839+0000", 2024-02-05T04:53:39.078 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4936000000000003e-05 2024-02-05T04:53:39.078 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.078 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:39.078 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.078 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.078 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.078 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:264 3.ds0 3:b7ad4782:::smithi19639362-40:head [append 0~376832] snapc 3=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-02-05T04:53:39.079 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:30.174572+0000", 2024-02-05T04:53:39.079 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 8.8738489900000008, 2024-02-05T04:53:39.079 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0036751700000000002, 2024-02-05T04:53:39.079 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:39.079 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:39.079 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:53:39.079 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:53:39.079 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:39.079 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 264 2024-02-05T04:53:39.079 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.079 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:39.079 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.079 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:39.079 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.174572+0000", 2024-02-05T04:53:39.079 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:39.080 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.080 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.080 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:39.080 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.174572+0000", 2024-02-05T04:53:39.080 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:39.080 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.080 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.080 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:39.080 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.174573+0000", 2024-02-05T04:53:39.080 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.192e-06 2024-02-05T04:53:39.080 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.080 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.080 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:39.080 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.174769+0000", 2024-02-05T04:53:39.080 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00019648300000000001 2024-02-05T04:53:39.081 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.081 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.081 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:39.081 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.174783+0000", 2024-02-05T04:53:39.081 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3903000000000001e-05 2024-02-05T04:53:39.081 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.081 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.081 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:39.081 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.174792+0000", 2024-02-05T04:53:39.081 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.2989999999999997e-06 2024-02-05T04:53:39.081 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.081 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.081 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:39.081 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.174840+0000", 2024-02-05T04:53:39.081 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8559e-05 2024-02-05T04:53:39.082 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.082 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.082 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:53:39.082 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.174920+0000", 2024-02-05T04:53:39.082 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9324000000000001e-05 2024-02-05T04:53:39.082 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.082 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.082 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:39.082 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.175494+0000", 2024-02-05T04:53:39.082 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000574202 2024-02-05T04:53:39.082 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.082 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.082 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:39.082 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.177068+0000", 2024-02-05T04:53:39.082 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001574724 2024-02-05T04:53:39.083 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.083 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.083 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T04:53:39.083 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.178205+0000", 2024-02-05T04:53:39.083 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0011367949999999999 2024-02-05T04:53:39.083 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.083 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.083 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:39.083 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.178247+0000", 2024-02-05T04:53:39.083 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1689e-05 2024-02-05T04:53:39.083 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.083 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:39.083 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.083 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.083 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.083 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:265 3.ds0 3:b7ad4782:::smithi19639362-40:head [setxattr _header (56) in=7b] snapc 3=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-02-05T04:53:39.084 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:30.174812+0000", 2024-02-05T04:53:39.084 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 8.8736085930000002, 2024-02-05T04:53:39.084 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0035712299999999999, 2024-02-05T04:53:39.084 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:39.084 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:39.084 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:53:39.084 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:53:39.084 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:39.084 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 265 2024-02-05T04:53:39.084 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.084 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:39.084 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.084 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:39.084 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.174812+0000", 2024-02-05T04:53:39.085 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:39.085 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.085 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.085 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:39.085 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.174812+0000", 2024-02-05T04:53:39.085 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:39.085 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.085 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.085 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:39.085 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.174813+0000", 2024-02-05T04:53:39.085 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1900000000000002e-07 2024-02-05T04:53:39.085 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.086 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.086 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:39.086 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.174817+0000", 2024-02-05T04:53:39.086 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9700000000000001e-06 2024-02-05T04:53:39.086 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.086 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.086 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:39.086 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.174825+0000", 2024-02-05T04:53:39.086 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9759999999999995e-06 2024-02-05T04:53:39.086 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.086 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.086 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:39.086 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.174831+0000", 2024-02-05T04:53:39.086 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.195e-06 2024-02-05T04:53:39.086 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.086 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.087 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:39.087 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.176055+0000", 2024-02-05T04:53:39.087 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0012240809999999999 2024-02-05T04:53:39.087 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.087 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.087 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:53:39.087 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.176114+0000", 2024-02-05T04:53:39.087 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8761000000000002e-05 2024-02-05T04:53:39.087 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.087 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.087 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:39.087 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.176385+0000", 2024-02-05T04:53:39.087 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00027111000000000003 2024-02-05T04:53:39.088 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.088 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.088 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:39.088 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.177182+0000", 2024-02-05T04:53:39.088 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00079685400000000005 2024-02-05T04:53:39.088 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.088 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.088 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T04:53:39.088 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.178340+0000", 2024-02-05T04:53:39.088 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00115788 2024-02-05T04:53:39.088 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.088 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.088 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:39.089 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.178383+0000", 2024-02-05T04:53:39.089 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3784e-05 2024-02-05T04:53:39.089 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.089 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:39.089 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.089 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.089 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.089 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:266 3.ds0 3:b7ad4782:::smithi19639362-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-02-05T04:53:39.089 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:30.174843+0000", 2024-02-05T04:53:39.089 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 8.8735772970000006, 2024-02-05T04:53:39.089 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.004642803, 2024-02-05T04:53:39.089 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:39.089 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-02-05T04:53:39.089 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:53:39.089 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:53:39.090 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:39.090 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 266 2024-02-05T04:53:39.090 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.090 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:39.090 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.090 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:39.090 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.174843+0000", 2024-02-05T04:53:39.090 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:39.090 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.090 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.090 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:39.090 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.174843+0000", 2024-02-05T04:53:39.090 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:39.090 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.090 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.091 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:39.091 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.174844+0000", 2024-02-05T04:53:39.091 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1900000000000003e-07 2024-02-05T04:53:39.091 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.091 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.091 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:39.091 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.174846+0000", 2024-02-05T04:53:39.091 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.486e-06 2024-02-05T04:53:39.091 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.091 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.091 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:39.091 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.174854+0000", 2024-02-05T04:53:39.091 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9230000000000002e-06 2024-02-05T04:53:39.092 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.092 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.092 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:39.092 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.174860+0000", 2024-02-05T04:53:39.092 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7559999999999996e-06 2024-02-05T04:53:39.092 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.092 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.092 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:39.092 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.176912+0000", 2024-02-05T04:53:39.092 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002051977 2024-02-05T04:53:39.092 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.092 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.092 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for rw locks", 2024-02-05T04:53:39.093 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.176993+0000", 2024-02-05T04:53:39.093 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.1355999999999998e-05 2024-02-05T04:53:39.093 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.093 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.093 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:39.093 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.178630+0000", 2024-02-05T04:53:39.093 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0016364210000000001 2024-02-05T04:53:39.093 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.093 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.093 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:53:39.093 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.178680+0000", 2024-02-05T04:53:39.093 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0340999999999998e-05 2024-02-05T04:53:39.093 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.093 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.093 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:39.093 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.179486+0000", 2024-02-05T04:53:39.094 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00080602400000000002 2024-02-05T04:53:39.094 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.094 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:39.094 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.094 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.094 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.094 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:274 3.ds0 3:be3a9027:::smithi19639362-44:head [append 0~770048] snapc 3=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-02-05T04:53:39.094 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:30.190943+0000", 2024-02-05T04:53:39.094 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 8.8574776400000008, 2024-02-05T04:53:39.094 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0053404719999999998, 2024-02-05T04:53:39.094 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:39.094 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:39.094 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:53:39.094 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:53:39.094 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:39.095 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 274 2024-02-05T04:53:39.095 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.095 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:39.095 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.095 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:39.095 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.190943+0000", 2024-02-05T04:53:39.095 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:39.095 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.095 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.095 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:39.095 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.190943+0000", 2024-02-05T04:53:39.095 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:39.095 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.095 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.095 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:39.096 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.190945+0000", 2024-02-05T04:53:39.096 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8530000000000001e-06 2024-02-05T04:53:39.096 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.096 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.096 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:39.096 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.191153+0000", 2024-02-05T04:53:39.096 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000207584 2024-02-05T04:53:39.096 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.096 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.096 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:39.096 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.191169+0000", 2024-02-05T04:53:39.096 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6568000000000001e-05 2024-02-05T04:53:39.096 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.096 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.096 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:39.097 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.191176+0000", 2024-02-05T04:53:39.097 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.3429999999999998e-06 2024-02-05T04:53:39.097 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.097 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.097 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:39.097 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.191233+0000", 2024-02-05T04:53:39.097 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6952e-05 2024-02-05T04:53:39.097 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.097 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.097 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:53:39.097 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.191301+0000", 2024-02-05T04:53:39.097 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7972999999999994e-05 2024-02-05T04:53:39.097 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.097 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.097 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:39.098 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.192078+0000", 2024-02-05T04:53:39.098 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00077698899999999996 2024-02-05T04:53:39.098 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.098 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.098 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:39.098 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.192971+0000", 2024-02-05T04:53:39.098 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000892971 2024-02-05T04:53:39.098 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.098 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.098 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T04:53:39.098 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.196200+0000", 2024-02-05T04:53:39.099 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0032284810000000001 2024-02-05T04:53:39.099 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.099 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.099 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:39.099 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.196284+0000", 2024-02-05T04:53:39.099 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3757999999999993e-05 2024-02-05T04:53:39.099 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.099 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:39.099 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.099 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.099 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.099 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:277 3.ds0 3:be3a9027:::smithi19639362-44:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-02-05T04:53:39.099 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:30.195060+0000", 2024-02-05T04:53:39.099 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 8.8533602850000008, 2024-02-05T04:53:39.099 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0044373099999999999, 2024-02-05T04:53:39.100 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:39.100 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-02-05T04:53:39.100 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:53:39.100 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:53:39.100 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:39.100 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 277 2024-02-05T04:53:39.100 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.100 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:39.100 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.100 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:39.100 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.195060+0000", 2024-02-05T04:53:39.100 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:39.100 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.100 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.100 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:39.101 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.195060+0000", 2024-02-05T04:53:39.101 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:39.101 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.101 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.101 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:39.101 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.195061+0000", 2024-02-05T04:53:39.101 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9400000000000005e-07 2024-02-05T04:53:39.101 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.101 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.101 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:39.101 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.195063+0000", 2024-02-05T04:53:39.101 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2639999999999998e-06 2024-02-05T04:53:39.101 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.101 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.101 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:39.102 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.195070+0000", 2024-02-05T04:53:39.102 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0910000000000004e-06 2024-02-05T04:53:39.102 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.102 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.102 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:39.102 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.195075+0000", 2024-02-05T04:53:39.102 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3000000000000003e-06 2024-02-05T04:53:39.102 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.102 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.102 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:39.102 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.196855+0000", 2024-02-05T04:53:39.102 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001780694 2024-02-05T04:53:39.102 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.102 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.102 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for rw locks", 2024-02-05T04:53:39.102 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.196953+0000", 2024-02-05T04:53:39.103 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.7237000000000004e-05 2024-02-05T04:53:39.103 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.103 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.103 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:39.103 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.198325+0000", 2024-02-05T04:53:39.103 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0013727240000000001 2024-02-05T04:53:39.103 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.103 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.103 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:53:39.103 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.198422+0000", 2024-02-05T04:53:39.103 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6675999999999994e-05 2024-02-05T04:53:39.103 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.103 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.103 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:39.103 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.199498+0000", 2024-02-05T04:53:39.104 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00107573 2024-02-05T04:53:39.104 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.104 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:39.104 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.104 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.104 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.104 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:387 3.ds0 3:b4e8e82c:::smithi19639362-25:head [read 0~638976,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-02-05T04:53:39.104 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:34.171793+0000", 2024-02-05T04:53:39.104 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 4.8766274420000002, 2024-02-05T04:53:39.104 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0052279479999999996, 2024-02-05T04:53:39.104 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:39.104 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-02-05T04:53:39.104 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:53:39.104 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:53:39.104 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:39.105 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 387 2024-02-05T04:53:39.105 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.105 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:39.105 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.105 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:39.105 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.171793+0000", 2024-02-05T04:53:39.105 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:39.105 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.105 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.105 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:39.105 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.171793+0000", 2024-02-05T04:53:39.105 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:39.105 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.105 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.105 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:39.106 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.171795+0000", 2024-02-05T04:53:39.106 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0729999999999999e-06 2024-02-05T04:53:39.106 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.106 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.106 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:39.106 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.171805+0000", 2024-02-05T04:53:39.106 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1339999999999993e-06 2024-02-05T04:53:39.106 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.106 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.106 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:39.106 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.171827+0000", 2024-02-05T04:53:39.106 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2954000000000001e-05 2024-02-05T04:53:39.106 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.106 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.106 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:39.106 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.171839+0000", 2024-02-05T04:53:39.107 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1646e-05 2024-02-05T04:53:39.107 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.107 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.107 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:39.107 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.171919+0000", 2024-02-05T04:53:39.107 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0232e-05 2024-02-05T04:53:39.107 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.107 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.107 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op must wait for map", 2024-02-05T04:53:39.107 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.171927+0000", 2024-02-05T04:53:39.107 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8369999999999997e-06 2024-02-05T04:53:39.107 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.107 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.107 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:39.107 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.174842+0000", 2024-02-05T04:53:39.107 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0029150309999999998 2024-02-05T04:53:39.108 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.108 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.108 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:53:39.108 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.174940+0000", 2024-02-05T04:53:39.108 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8147999999999996e-05 2024-02-05T04:53:39.108 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.108 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.108 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:39.108 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.177021+0000", 2024-02-05T04:53:39.108 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0020808929999999999 2024-02-05T04:53:39.108 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.108 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:39.108 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.108 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.109 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.109 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:388 3.ds0 3:b4e8e82c:::smithi19639362-25:head [read 0~638976,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-02-05T04:53:39.109 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:34.171871+0000", 2024-02-05T04:53:39.109 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 4.8765499959999996, 2024-02-05T04:53:39.109 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0058757660000000001, 2024-02-05T04:53:39.109 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:39.109 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-02-05T04:53:39.109 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:53:39.109 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:53:39.109 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:39.110 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 388 2024-02-05T04:53:39.110 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.110 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:39.110 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.110 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:39.110 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.171871+0000", 2024-02-05T04:53:39.110 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:39.110 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.110 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.110 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:39.110 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.171871+0000", 2024-02-05T04:53:39.110 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:39.110 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.111 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.111 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:39.111 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.171871+0000", 2024-02-05T04:53:39.111 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8899999999999999e-07 2024-02-05T04:53:39.111 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.111 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.111 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:39.111 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.171875+0000", 2024-02-05T04:53:39.111 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.777e-06 2024-02-05T04:53:39.111 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.111 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.111 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:39.111 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.171884+0000", 2024-02-05T04:53:39.111 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8559999999999996e-06 2024-02-05T04:53:39.111 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.112 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.112 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:39.112 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.171892+0000", 2024-02-05T04:53:39.112 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6000000000000001e-06 2024-02-05T04:53:39.112 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.112 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.112 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:39.112 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.171986+0000", 2024-02-05T04:53:39.112 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.4010000000000003e-05 2024-02-05T04:53:39.112 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.112 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.112 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting_for_map not empty", 2024-02-05T04:53:39.112 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.171990+0000", 2024-02-05T04:53:39.112 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2719999999999999e-06 2024-02-05T04:53:39.112 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.113 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.113 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:39.113 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.175603+0000", 2024-02-05T04:53:39.113 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0036129080000000002 2024-02-05T04:53:39.113 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.113 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.113 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:53:39.113 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.175679+0000", 2024-02-05T04:53:39.113 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6552999999999997e-05 2024-02-05T04:53:39.113 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.113 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.113 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:39.113 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.177747+0000", 2024-02-05T04:53:39.113 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0020672009999999998 2024-02-05T04:53:39.113 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.114 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:39.114 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.114 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.114 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.114 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:389 3.ds0 3:b4e8e82c:::smithi19639362-25:head [read 0~638976,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-02-05T04:53:39.114 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:34.171905+0000", 2024-02-05T04:53:39.114 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 4.876515521, 2024-02-05T04:53:39.114 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0072757830000000001, 2024-02-05T04:53:39.114 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:39.114 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-02-05T04:53:39.114 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:53:39.114 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:53:39.114 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:39.114 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 389 2024-02-05T04:53:39.114 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.114 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:39.115 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.115 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:39.115 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.171905+0000", 2024-02-05T04:53:39.115 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:39.115 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.115 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.115 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:39.115 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.171905+0000", 2024-02-05T04:53:39.115 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:39.115 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.115 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.115 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:39.115 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.171906+0000", 2024-02-05T04:53:39.115 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2499999999999995e-07 2024-02-05T04:53:39.115 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.115 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.116 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:39.116 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.171910+0000", 2024-02-05T04:53:39.116 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4490000000000001e-06 2024-02-05T04:53:39.116 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.116 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.116 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:39.116 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.171918+0000", 2024-02-05T04:53:39.116 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.2449999999999998e-06 2024-02-05T04:53:39.116 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.116 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.116 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:39.116 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.171924+0000", 2024-02-05T04:53:39.116 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6350000000000003e-06 2024-02-05T04:53:39.116 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.117 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.117 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:39.117 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.172022+0000", 2024-02-05T04:53:39.117 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8004000000000006e-05 2024-02-05T04:53:39.117 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.117 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.117 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting_for_map not empty", 2024-02-05T04:53:39.117 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.172026+0000", 2024-02-05T04:53:39.117 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8500000000000004e-06 2024-02-05T04:53:39.117 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.117 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.117 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:39.117 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.177805+0000", 2024-02-05T04:53:39.117 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0057789419999999996 2024-02-05T04:53:39.117 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.118 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.118 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:53:39.118 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.177900+0000", 2024-02-05T04:53:39.118 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5529e-05 2024-02-05T04:53:39.118 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.118 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.118 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:39.118 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.179181+0000", 2024-02-05T04:53:39.118 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0012806040000000001 2024-02-05T04:53:39.118 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.118 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:39.118 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.118 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.118 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.118 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.bs5 43/31 ECSubWrite(tid=137, reqid=osd.13.0:137, at_version=43'28, trim_to=0'0, roll_forward_to=41'26))", 2024-02-05T04:53:39.119 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:35.171816+0000", 2024-02-05T04:53:39.119 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 3.8766045220000001, 2024-02-05T04:53:39.119 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0057279920000000003, 2024-02-05T04:53:39.119 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:39.119 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:39.119 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:39.119 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.119 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:39.119 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.171816+0000", 2024-02-05T04:53:39.119 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:39.119 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.119 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.119 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:39.119 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.171816+0000", 2024-02-05T04:53:39.119 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:39.119 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.120 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.120 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:39.120 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.171817+0000", 2024-02-05T04:53:39.120 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.89e-07 2024-02-05T04:53:39.120 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.120 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.120 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:39.120 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.171845+0000", 2024-02-05T04:53:39.120 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.7804e-05 2024-02-05T04:53:39.120 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.120 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.120 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:39.120 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.171855+0000", 2024-02-05T04:53:39.120 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0227999999999999e-05 2024-02-05T04:53:39.120 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.121 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.121 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:39.121 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.171865+0000", 2024-02-05T04:53:39.121 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.026e-05 2024-02-05T04:53:39.121 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.121 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.121 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:39.121 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.171958+0000", 2024-02-05T04:53:39.121 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2785e-05 2024-02-05T04:53:39.121 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.121 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.121 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:39.121 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.171969+0000", 2024-02-05T04:53:39.121 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0998e-05 2024-02-05T04:53:39.121 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.122 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.122 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:39.122 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.177502+0000", 2024-02-05T04:53:39.122 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0055336270000000002 2024-02-05T04:53:39.122 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.122 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.122 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:39.122 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.177544+0000", 2024-02-05T04:53:39.122 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1801000000000002e-05 2024-02-05T04:53:39.122 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.122 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:39.122 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.122 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.122 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.123 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.6s5 43/31 ECSubWrite(tid=138, reqid=osd.13.0:138, at_version=43'24, trim_to=0'0, roll_forward_to=40'22))", 2024-02-05T04:53:39.123 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:35.172477+0000", 2024-02-05T04:53:39.123 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 3.8759438180000001, 2024-02-05T04:53:39.123 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0079411910000000002, 2024-02-05T04:53:39.123 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:39.123 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:39.123 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:39.123 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.123 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:39.123 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.172477+0000", 2024-02-05T04:53:39.123 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:39.123 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.123 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.123 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:39.123 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.172477+0000", 2024-02-05T04:53:39.124 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:39.124 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.124 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.124 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:39.124 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.172478+0000", 2024-02-05T04:53:39.124 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9299999999999997e-07 2024-02-05T04:53:39.124 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.124 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.124 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:39.124 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.172502+0000", 2024-02-05T04:53:39.124 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.4071e-05 2024-02-05T04:53:39.124 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.124 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.124 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:39.124 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.172525+0000", 2024-02-05T04:53:39.125 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3666e-05 2024-02-05T04:53:39.125 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.125 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.125 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:39.125 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.172534+0000", 2024-02-05T04:53:39.125 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.7749999999999998e-06 2024-02-05T04:53:39.128 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.128 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.128 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:39.128 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.172612+0000", 2024-02-05T04:53:39.128 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7486999999999999e-05 2024-02-05T04:53:39.128 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.128 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.128 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:39.128 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.172622+0000", 2024-02-05T04:53:39.128 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0040999999999999e-05 2024-02-05T04:53:39.128 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.128 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.128 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:39.129 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.180380+0000", 2024-02-05T04:53:39.129 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0077582739999999999 2024-02-05T04:53:39.129 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.129 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.129 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:39.129 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.180418+0000", 2024-02-05T04:53:39.129 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8183999999999999e-05 2024-02-05T04:53:39.129 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.129 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:39.129 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.129 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.129 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.129 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.3s1 43/31 ECSubWrite(tid=63, reqid=osd.4.0:63, at_version=43'27, trim_to=0'0, roll_forward_to=40'25))", 2024-02-05T04:53:39.129 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:35.173843+0000", 2024-02-05T04:53:39.129 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 3.8745778749999999, 2024-02-05T04:53:39.130 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.006542473, 2024-02-05T04:53:39.130 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:39.130 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:39.130 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:39.130 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.130 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:39.130 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.173843+0000", 2024-02-05T04:53:39.130 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:39.130 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.130 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.130 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:39.130 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.173843+0000", 2024-02-05T04:53:39.130 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:39.130 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.130 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.130 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:39.131 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.173843+0000", 2024-02-05T04:53:39.131 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.18e-07 2024-02-05T04:53:39.131 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.131 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.131 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:39.131 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.173862+0000", 2024-02-05T04:53:39.131 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8770000000000002e-05 2024-02-05T04:53:39.131 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.131 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.131 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:39.131 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.173883+0000", 2024-02-05T04:53:39.131 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1072000000000001e-05 2024-02-05T04:53:39.131 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.132 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.132 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:39.132 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.173890+0000", 2024-02-05T04:53:39.132 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9569999999999996e-06 2024-02-05T04:53:39.132 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.132 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.132 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:39.133 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.173952+0000", 2024-02-05T04:53:39.133 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.2553999999999998e-05 2024-02-05T04:53:39.133 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.133 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.133 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:39.133 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.173960+0000", 2024-02-05T04:53:39.133 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7600000000000002e-06 2024-02-05T04:53:39.133 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.134 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.134 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:39.134 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.180350+0000", 2024-02-05T04:53:39.134 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0063900290000000002 2024-02-05T04:53:39.134 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.134 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.134 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:39.135 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.180385+0000", 2024-02-05T04:53:39.135 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5213000000000003e-05 2024-02-05T04:53:39.135 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.135 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:39.135 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.135 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.136 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.136 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.3s1 43/31 ECSubWrite(tid=65, reqid=osd.4.0:63, at_version=0'0, trim_to=0'0, roll_forward_to=43'27))", 2024-02-05T04:53:39.136 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:35.182145+0000", 2024-02-05T04:53:39.136 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 3.8662759690000001, 2024-02-05T04:53:39.136 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0036892840000000001, 2024-02-05T04:53:39.136 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:39.136 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:39.137 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:39.137 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.137 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:39.137 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.182145+0000", 2024-02-05T04:53:39.137 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:39.137 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.137 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.137 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:39.138 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.182145+0000", 2024-02-05T04:53:39.138 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:39.138 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.138 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.138 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:39.138 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.182146+0000", 2024-02-05T04:53:39.138 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.136e-06 2024-02-05T04:53:39.138 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.139 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.139 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:39.139 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.182159+0000", 2024-02-05T04:53:39.139 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.277e-05 2024-02-05T04:53:39.139 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.139 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.139 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:39.140 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.182177+0000", 2024-02-05T04:53:39.140 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8102e-05 2024-02-05T04:53:39.140 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.140 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.140 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:39.140 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.182190+0000", 2024-02-05T04:53:39.141 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3543000000000001e-05 2024-02-05T04:53:39.141 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.141 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.141 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:39.141 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.182257+0000", 2024-02-05T04:53:39.141 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7032000000000005e-05 2024-02-05T04:53:39.141 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.141 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.141 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:39.142 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.182267+0000", 2024-02-05T04:53:39.142 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2089999999999994e-06 2024-02-05T04:53:39.142 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.142 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.142 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:39.142 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.185799+0000", 2024-02-05T04:53:39.142 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.003532332 2024-02-05T04:53:39.142 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.142 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.142 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:39.142 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.185834+0000", 2024-02-05T04:53:39.142 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5160000000000002e-05 2024-02-05T04:53:39.142 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.142 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:39.143 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.143 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.143 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.143 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:401 3.ds0 3:b9c88094:::smithi19639362-50:head [delete] snapc 7=[7,6,4] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e43)", 2024-02-05T04:53:39.143 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:35.182518+0000", 2024-02-05T04:53:39.143 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 3.8659026779999999, 2024-02-05T04:53:39.143 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0087143140000000008, 2024-02-05T04:53:39.143 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:39.143 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:39.143 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:53:39.143 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:53:39.143 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:39.143 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 401 2024-02-05T04:53:39.143 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.143 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:39.144 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.144 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:39.144 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.182518+0000", 2024-02-05T04:53:39.144 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:39.144 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.144 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.144 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:39.144 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.182518+0000", 2024-02-05T04:53:39.144 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:39.144 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.144 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.144 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:39.144 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.182519+0000", 2024-02-05T04:53:39.144 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.313e-06 2024-02-05T04:53:39.145 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.145 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.145 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:39.145 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.182524+0000", 2024-02-05T04:53:39.145 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9150000000000004e-06 2024-02-05T04:53:39.145 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.145 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.145 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:39.145 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.182544+0000", 2024-02-05T04:53:39.145 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9327999999999999e-05 2024-02-05T04:53:39.145 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.145 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.145 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:39.145 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.182572+0000", 2024-02-05T04:53:39.145 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8180000000000001e-05 2024-02-05T04:53:39.146 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.146 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.146 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:39.146 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.182643+0000", 2024-02-05T04:53:39.146 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1598000000000001e-05 2024-02-05T04:53:39.146 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.146 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.146 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:53:39.146 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.182744+0000", 2024-02-05T04:53:39.146 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000100881 2024-02-05T04:53:39.146 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.146 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.146 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:39.146 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.183600+0000", 2024-02-05T04:53:39.146 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00085556200000000003 2024-02-05T04:53:39.146 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.147 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.147 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:39.147 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.188714+0000", 2024-02-05T04:53:39.147 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.005113828 2024-02-05T04:53:39.147 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.147 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.147 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T04:53:39.147 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.191180+0000", 2024-02-05T04:53:39.147 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0024662130000000001 2024-02-05T04:53:39.147 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.147 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.147 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:39.147 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.191232+0000", 2024-02-05T04:53:39.147 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2496e-05 2024-02-05T04:53:39.147 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.148 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:39.148 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.148 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.148 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.148 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.6s5 43/31 ECSubWrite(tid=141, reqid=osd.13.0:138, at_version=0'0, trim_to=0'0, roll_forward_to=43'24))", 2024-02-05T04:53:39.148 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:35.183142+0000", 2024-02-05T04:53:39.148 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 3.8652790000000001, 2024-02-05T04:53:39.148 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0055687339999999997, 2024-02-05T04:53:39.148 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:39.148 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:39.148 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:39.148 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.148 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:39.148 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.183142+0000", 2024-02-05T04:53:39.149 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:39.149 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.149 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.149 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:39.149 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.183142+0000", 2024-02-05T04:53:39.149 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:39.149 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.149 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.149 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:39.149 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.183142+0000", 2024-02-05T04:53:39.149 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3799999999999998e-07 2024-02-05T04:53:39.149 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.149 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.149 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:39.149 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.183153+0000", 2024-02-05T04:53:39.150 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0924000000000001e-05 2024-02-05T04:53:39.150 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.150 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.150 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:39.150 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.183171+0000", 2024-02-05T04:53:39.150 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8199e-05 2024-02-05T04:53:39.150 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.150 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.150 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:39.150 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.183185+0000", 2024-02-05T04:53:39.150 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3594999999999999e-05 2024-02-05T04:53:39.150 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.150 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.151 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:39.151 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.183244+0000", 2024-02-05T04:53:39.151 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8714000000000001e-05 2024-02-05T04:53:39.151 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.151 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.151 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:39.151 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.183253+0000", 2024-02-05T04:53:39.151 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5340000000000005e-06 2024-02-05T04:53:39.151 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.151 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.151 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:39.151 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.188665+0000", 2024-02-05T04:53:39.151 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0054115350000000003 2024-02-05T04:53:39.151 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.151 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.152 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:39.152 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.188711+0000", 2024-02-05T04:53:39.152 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5794999999999998e-05 2024-02-05T04:53:39.152 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.152 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:39.152 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.152 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.152 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:39.152 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T04:53:39.154 INFO:tasks.ceph.osd.1.smithi091.stderr:2024-02-05T04:53:39.152+0000 7ff9dc283640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34275) UID: 0 2024-02-05T04:53:39.154 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-02-05T04:53:39.189 INFO:tasks.thrashosds.thrasher:Setting full ratio back to .95 2024-02-05T04:53:39.189 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set-full-ratio .95 2024-02-05T04:53:39.254 INFO:tasks.ceph.osd.7.smithi196.stderr:2024-02-05T04:53:39.253+0000 7fb281419640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34606) UID: 0 2024-02-05T04:53:39.301 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T04:53:39.301 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-02-05T04:53:39.301 INFO:teuthology.orchestra.run.smithi196.stdout: "num_ops": 0 2024-02-05T04:53:39.301 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T04:53:39.309 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-02-05T04:53:39.355 INFO:tasks.ceph.osd.2.smithi150.stderr:2024-02-05T04:53:39.354+0000 7f7337bf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 34014) UID: 0 2024-02-05T04:53:39.455 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T04:53:39.455 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-02-05T04:53:39.455 INFO:teuthology.orchestra.run.smithi196.stdout: "complaint_time": 30, 2024-02-05T04:53:39.456 INFO:teuthology.orchestra.run.smithi196.stdout: "num_blocked_ops": 0 2024-02-05T04:53:39.456 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T04:53:39.456 INFO:tasks.ceph.osd.2.smithi150.stderr:2024-02-05T04:53:39.454+0000 7f7337bf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 34014) UID: 0 2024-02-05T04:53:39.463 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-02-05T04:53:39.556 INFO:tasks.ceph.osd.1.smithi091.stderr:2024-02-05T04:53:39.555+0000 7ff9dc283640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34275) UID: 0 2024-02-05T04:53:39.572 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T04:53:39.572 INFO:teuthology.orchestra.run.smithi196.stdout: "size": 20, 2024-02-05T04:53:39.572 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 600, 2024-02-05T04:53:39.572 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-02-05T04:53:39.572 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.572 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:222 3.cs0 3:30950ab7:::smithi19639362-4:3 [read 0~65536,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:53:39.572 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:29.134965+0000", 2024-02-05T04:53:39.572 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 10.434632047999999, 2024-02-05T04:53:39.573 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0084490469999999995, 2024-02-05T04:53:39.573 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:39.573 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-02-05T04:53:39.573 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:53:39.573 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:53:39.573 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:39.573 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 222 2024-02-05T04:53:39.573 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.574 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:39.574 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.574 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:39.574 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.134965+0000", 2024-02-05T04:53:39.574 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:39.574 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.574 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.574 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:39.574 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.134965+0000", 2024-02-05T04:53:39.574 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:39.575 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.575 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.575 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:39.575 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.134991+0000", 2024-02-05T04:53:39.575 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.6359999999999998e-05 2024-02-05T04:53:39.575 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.575 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.575 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:39.575 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135011+0000", 2024-02-05T04:53:39.575 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9874e-05 2024-02-05T04:53:39.576 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.576 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.576 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:39.576 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135044+0000", 2024-02-05T04:53:39.576 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2778999999999999e-05 2024-02-05T04:53:39.576 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.576 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.576 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:39.576 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135057+0000", 2024-02-05T04:53:39.577 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2582000000000001e-05 2024-02-05T04:53:39.577 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.577 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.577 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:39.577 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135163+0000", 2024-02-05T04:53:39.577 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010616900000000001 2024-02-05T04:53:39.577 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.577 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.577 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op must wait for map", 2024-02-05T04:53:39.577 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135171+0000", 2024-02-05T04:53:39.578 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.7709999999999995e-06 2024-02-05T04:53:39.578 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.578 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.578 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:39.578 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.139041+0000", 2024-02-05T04:53:39.578 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0038695769999999999 2024-02-05T04:53:39.578 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.579 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.579 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:53:39.579 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.139163+0000", 2024-02-05T04:53:39.579 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012187499999999999 2024-02-05T04:53:39.579 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.579 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.580 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:39.580 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.143414+0000", 2024-02-05T04:53:39.580 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0042510600000000001 2024-02-05T04:53:39.580 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.580 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:39.580 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.581 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.581 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.581 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:223 3.cs0 3:30950ab7:::smithi19639362-4:3 [read 0~65536,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:53:39.581 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:29.135093+0000", 2024-02-05T04:53:39.581 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 10.434504111000001, 2024-02-05T04:53:39.581 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0089544459999999996, 2024-02-05T04:53:39.582 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:39.582 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-02-05T04:53:39.582 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:53:39.582 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:53:39.582 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:39.582 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 223 2024-02-05T04:53:39.583 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.583 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:39.583 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.583 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:39.583 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135093+0000", 2024-02-05T04:53:39.583 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:39.584 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.584 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.584 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:39.584 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135093+0000", 2024-02-05T04:53:39.584 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:39.584 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.585 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.585 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:39.585 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135094+0000", 2024-02-05T04:53:39.585 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0100000000000004e-07 2024-02-05T04:53:39.585 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.585 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.585 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:39.586 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135100+0000", 2024-02-05T04:53:39.586 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9950000000000002e-06 2024-02-05T04:53:39.586 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.586 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.586 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:39.586 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135114+0000", 2024-02-05T04:53:39.587 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4831000000000001e-05 2024-02-05T04:53:39.587 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.587 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.587 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:39.587 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135123+0000", 2024-02-05T04:53:39.587 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0730000000000005e-06 2024-02-05T04:53:39.587 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.587 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.588 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:39.588 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135265+0000", 2024-02-05T04:53:39.588 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000142423 2024-02-05T04:53:39.588 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.588 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.588 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting_for_map not empty", 2024-02-05T04:53:39.588 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135271+0000", 2024-02-05T04:53:39.588 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.2689999999999996e-06 2024-02-05T04:53:39.588 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.588 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.588 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:39.588 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.140216+0000", 2024-02-05T04:53:39.588 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0049449699999999999 2024-02-05T04:53:39.588 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.588 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.589 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:53:39.589 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.140326+0000", 2024-02-05T04:53:39.589 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000109412 2024-02-05T04:53:39.589 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.589 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.589 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:39.589 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.144047+0000", 2024-02-05T04:53:39.589 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0037216720000000001 2024-02-05T04:53:39.589 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.589 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:39.589 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.589 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.589 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.589 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:224 3.cs0 3:30950ab7:::smithi19639362-4:3 [read 0~65536,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:53:39.589 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:29.135196+0000", 2024-02-05T04:53:39.590 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 10.434400875, 2024-02-05T04:53:39.590 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0091200659999999996, 2024-02-05T04:53:39.590 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:39.590 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-02-05T04:53:39.590 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:53:39.590 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:53:39.590 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:39.590 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 224 2024-02-05T04:53:39.590 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.590 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:39.590 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.590 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:39.590 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135196+0000", 2024-02-05T04:53:39.590 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:39.590 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.590 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.591 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:39.591 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135196+0000", 2024-02-05T04:53:39.591 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:39.591 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.591 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.591 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:39.591 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135197+0000", 2024-02-05T04:53:39.591 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1790000000000001e-06 2024-02-05T04:53:39.591 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.591 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.591 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:39.591 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135204+0000", 2024-02-05T04:53:39.591 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5309999999999998e-06 2024-02-05T04:53:39.591 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.591 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.591 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:39.592 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135216+0000", 2024-02-05T04:53:39.592 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2573e-05 2024-02-05T04:53:39.592 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.592 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.592 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:39.592 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135227+0000", 2024-02-05T04:53:39.592 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1029000000000001e-05 2024-02-05T04:53:39.592 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.592 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.592 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:39.592 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135925+0000", 2024-02-05T04:53:39.592 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00069716299999999999 2024-02-05T04:53:39.592 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.593 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.593 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting_for_map not empty", 2024-02-05T04:53:39.593 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135930+0000", 2024-02-05T04:53:39.593 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5799999999999999e-06 2024-02-05T04:53:39.593 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.593 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.593 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:39.593 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.142325+0000", 2024-02-05T04:53:39.593 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0063945950000000003 2024-02-05T04:53:39.593 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.593 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.593 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:53:39.593 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.142420+0000", 2024-02-05T04:53:39.593 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5747000000000003e-05 2024-02-05T04:53:39.593 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.594 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.594 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:39.594 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.144316+0000", 2024-02-05T04:53:39.594 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001895669 2024-02-05T04:53:39.594 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.594 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:39.594 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.594 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.594 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.594 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:230 3.fs0 3:fdb87099:::smithi19639362-41:head [read 0~688128,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:53:39.594 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:29.149581+0000", 2024-02-05T04:53:39.594 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 10.420015652, 2024-02-05T04:53:39.594 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0047953409999999998, 2024-02-05T04:53:39.594 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:39.594 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-02-05T04:53:39.594 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:53:39.595 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:53:39.595 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:39.595 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 230 2024-02-05T04:53:39.595 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.595 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:39.595 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.595 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:39.595 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.149581+0000", 2024-02-05T04:53:39.595 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:39.595 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.595 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.595 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:39.595 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.149581+0000", 2024-02-05T04:53:39.595 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:39.595 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.595 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.596 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:39.596 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.149584+0000", 2024-02-05T04:53:39.596 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3800000000000001e-06 2024-02-05T04:53:39.596 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.596 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.596 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:39.596 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.149593+0000", 2024-02-05T04:53:39.596 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.7110000000000007e-06 2024-02-05T04:53:39.596 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.596 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.596 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:39.596 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.149622+0000", 2024-02-05T04:53:39.596 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8856999999999998e-05 2024-02-05T04:53:39.596 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.597 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.597 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:39.597 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.149635+0000", 2024-02-05T04:53:39.597 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2425999999999999e-05 2024-02-05T04:53:39.597 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.597 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.597 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:39.597 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.149717+0000", 2024-02-05T04:53:39.597 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.1804000000000005e-05 2024-02-05T04:53:39.597 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.597 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.597 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:53:39.597 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.149793+0000", 2024-02-05T04:53:39.597 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6687000000000006e-05 2024-02-05T04:53:39.597 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.597 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.598 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:39.598 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.154377+0000", 2024-02-05T04:53:39.598 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0045834760000000004 2024-02-05T04:53:39.598 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.598 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:39.598 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.598 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.598 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.598 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:231 3.fs0 3:fdb87099:::smithi19639362-41:head [read 0~688128,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:53:39.598 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:29.149664+0000", 2024-02-05T04:53:39.598 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 10.419932555000001, 2024-02-05T04:53:39.598 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.005165673, 2024-02-05T04:53:39.598 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:39.598 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-02-05T04:53:39.598 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:53:39.599 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:53:39.599 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:39.599 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 231 2024-02-05T04:53:39.599 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.599 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:39.599 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.599 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:39.599 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.149664+0000", 2024-02-05T04:53:39.599 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:39.599 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.599 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.599 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:39.599 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.149664+0000", 2024-02-05T04:53:39.599 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:39.599 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.599 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.600 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:39.600 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.149665+0000", 2024-02-05T04:53:39.600 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8999999999999996e-07 2024-02-05T04:53:39.600 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.600 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.600 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:39.600 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.149671+0000", 2024-02-05T04:53:39.600 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1029999999999998e-06 2024-02-05T04:53:39.600 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.600 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.600 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:39.600 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.149681+0000", 2024-02-05T04:53:39.600 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.9019999999999994e-06 2024-02-05T04:53:39.600 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.601 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.601 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:39.601 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.149687+0000", 2024-02-05T04:53:39.601 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3690000000000003e-06 2024-02-05T04:53:39.601 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.601 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.601 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:39.601 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.150203+0000", 2024-02-05T04:53:39.601 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00051509099999999996 2024-02-05T04:53:39.601 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.601 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.601 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:53:39.601 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.150289+0000", 2024-02-05T04:53:39.601 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.6275999999999999e-05 2024-02-05T04:53:39.601 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.601 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.602 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:39.602 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.154830+0000", 2024-02-05T04:53:39.602 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.004541242 2024-02-05T04:53:39.602 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.602 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:39.602 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.602 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.602 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.602 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:232 3.fs0 3:fdb87099:::smithi19639362-41:head [read 0~688128,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:53:39.602 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:29.149702+0000", 2024-02-05T04:53:39.602 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 10.419894874000001, 2024-02-05T04:53:39.602 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0055476930000000002, 2024-02-05T04:53:39.602 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:39.602 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-02-05T04:53:39.602 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:53:39.602 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:53:39.603 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:39.603 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 232 2024-02-05T04:53:39.603 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.603 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:39.603 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.603 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:39.603 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.149702+0000", 2024-02-05T04:53:39.603 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:39.603 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.603 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.603 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:39.603 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.149702+0000", 2024-02-05T04:53:39.603 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:39.603 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.603 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.603 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:39.604 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.149703+0000", 2024-02-05T04:53:39.604 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1999999999999999e-07 2024-02-05T04:53:39.604 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.604 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.604 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:39.604 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.149707+0000", 2024-02-05T04:53:39.604 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8e-06 2024-02-05T04:53:39.604 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.604 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.604 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:39.604 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.149728+0000", 2024-02-05T04:53:39.604 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1520999999999999e-05 2024-02-05T04:53:39.604 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.604 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.604 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:39.605 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.149736+0000", 2024-02-05T04:53:39.605 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.1270000000000003e-06 2024-02-05T04:53:39.605 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.605 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.605 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:39.605 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.151494+0000", 2024-02-05T04:53:39.605 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001757322 2024-02-05T04:53:39.605 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.605 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.605 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:53:39.605 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.151569+0000", 2024-02-05T04:53:39.605 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5158000000000001e-05 2024-02-05T04:53:39.605 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.605 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.605 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:39.605 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.155250+0000", 2024-02-05T04:53:39.606 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.003681045 2024-02-05T04:53:39.606 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.606 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:39.606 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.606 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.606 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.606 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.2s6 38/31 ECSubWrite(tid=29, reqid=osd.12.0:28, at_version=0'0, trim_to=0'0, roll_forward_to=38'22))", 2024-02-05T04:53:39.606 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:30.154278+0000", 2024-02-05T04:53:39.606 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 9.4153185019999999, 2024-02-05T04:53:39.606 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0052795960000000001, 2024-02-05T04:53:39.606 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:39.606 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:39.606 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:39.606 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.607 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:39.607 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.154278+0000", 2024-02-05T04:53:39.607 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:39.607 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.607 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.607 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:39.607 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.154278+0000", 2024-02-05T04:53:39.607 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:39.607 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.607 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.607 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:39.607 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.154279+0000", 2024-02-05T04:53:39.607 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.2600000000000001e-07 2024-02-05T04:53:39.607 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.608 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.608 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:39.608 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.154291+0000", 2024-02-05T04:53:39.608 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1982e-05 2024-02-05T04:53:39.608 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.608 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.608 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:39.608 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.154304+0000", 2024-02-05T04:53:39.608 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2850999999999999e-05 2024-02-05T04:53:39.608 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.608 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.608 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:39.608 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.154310+0000", 2024-02-05T04:53:39.608 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8590000000000001e-06 2024-02-05T04:53:39.608 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.609 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.609 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:39.609 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.154393+0000", 2024-02-05T04:53:39.609 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3461999999999995e-05 2024-02-05T04:53:39.609 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.609 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.609 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:39.609 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.154403+0000", 2024-02-05T04:53:39.609 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.3300000000000005e-06 2024-02-05T04:53:39.609 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.609 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.609 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:39.609 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.159511+0000", 2024-02-05T04:53:39.609 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0051083999999999999 2024-02-05T04:53:39.609 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.610 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.610 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:39.610 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.159558+0000", 2024-02-05T04:53:39.610 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6885999999999998e-05 2024-02-05T04:53:39.610 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.610 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:39.610 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.610 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.610 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.610 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:267 3.cs0 3:30950ab7:::smithi19639362-4:head [append 0~753664] snapc 3=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-02-05T04:53:39.610 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:30.179941+0000", 2024-02-05T04:53:39.610 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 9.3896562780000004, 2024-02-05T04:53:39.610 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0094095010000000007, 2024-02-05T04:53:39.610 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:39.610 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:39.611 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:53:39.611 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:53:39.611 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:39.611 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 267 2024-02-05T04:53:39.611 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.611 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:39.611 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.611 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:39.611 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.179941+0000", 2024-02-05T04:53:39.611 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:39.611 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.611 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.611 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:39.611 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.179941+0000", 2024-02-05T04:53:39.611 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:39.612 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.612 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.612 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:39.612 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.179943+0000", 2024-02-05T04:53:39.612 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3939999999999999e-06 2024-02-05T04:53:39.612 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.612 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.612 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:39.612 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.180152+0000", 2024-02-05T04:53:39.612 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000208791 2024-02-05T04:53:39.612 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.612 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.612 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:39.612 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.180171+0000", 2024-02-05T04:53:39.612 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9001e-05 2024-02-05T04:53:39.612 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.613 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.613 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:39.613 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.180178+0000", 2024-02-05T04:53:39.613 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4309999999999997e-06 2024-02-05T04:53:39.613 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.613 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.613 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:39.613 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.180243+0000", 2024-02-05T04:53:39.613 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4887000000000004e-05 2024-02-05T04:53:39.613 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.613 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.613 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:53:39.613 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.180307+0000", 2024-02-05T04:53:39.613 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3909000000000001e-05 2024-02-05T04:53:39.613 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.613 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.614 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:39.614 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.181056+0000", 2024-02-05T04:53:39.614 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00074879799999999995 2024-02-05T04:53:39.614 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.614 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.614 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:39.614 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.182113+0000", 2024-02-05T04:53:39.614 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001057555 2024-02-05T04:53:39.614 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.614 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.614 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T04:53:39.614 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.189310+0000", 2024-02-05T04:53:39.614 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0071969490000000002 2024-02-05T04:53:39.614 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.614 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.615 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:39.615 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.189350+0000", 2024-02-05T04:53:39.615 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9786000000000002e-05 2024-02-05T04:53:39.615 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.615 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:39.615 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.615 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.615 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.615 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:268 3.cs0 3:30950ab7:::smithi19639362-4:head [append 0~376832] snapc 3=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-02-05T04:53:39.615 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:30.182324+0000", 2024-02-05T04:53:39.615 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 9.3872725799999994, 2024-02-05T04:53:39.615 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0072136470000000001, 2024-02-05T04:53:39.615 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:39.615 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:39.615 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:53:39.615 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:53:39.615 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:39.616 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 268 2024-02-05T04:53:39.616 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.616 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:39.616 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.616 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:39.616 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.182324+0000", 2024-02-05T04:53:39.616 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:39.616 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.616 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.616 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:39.616 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.182324+0000", 2024-02-05T04:53:39.616 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:39.616 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.616 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.616 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:39.617 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.182326+0000", 2024-02-05T04:53:39.617 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.116e-06 2024-02-05T04:53:39.617 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.617 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.617 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:39.617 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.182515+0000", 2024-02-05T04:53:39.617 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00018944200000000001 2024-02-05T04:53:39.617 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.617 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.617 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:39.617 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.182527+0000", 2024-02-05T04:53:39.617 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2228e-05 2024-02-05T04:53:39.617 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.617 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.617 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:39.617 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.182536+0000", 2024-02-05T04:53:39.618 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3189999999999995e-06 2024-02-05T04:53:39.618 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.618 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.618 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:39.618 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.182587+0000", 2024-02-05T04:53:39.618 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1752000000000002e-05 2024-02-05T04:53:39.618 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.618 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.618 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:53:39.618 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.182649+0000", 2024-02-05T04:53:39.618 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1457999999999993e-05 2024-02-05T04:53:39.618 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.618 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.618 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:39.618 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.183319+0000", 2024-02-05T04:53:39.619 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00067060399999999999 2024-02-05T04:53:39.619 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.619 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.619 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:39.619 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.184039+0000", 2024-02-05T04:53:39.619 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00071980500000000001 2024-02-05T04:53:39.619 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.619 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.619 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T04:53:39.619 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.189492+0000", 2024-02-05T04:53:39.619 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0054526000000000002 2024-02-05T04:53:39.619 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.619 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.619 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:39.619 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.189538+0000", 2024-02-05T04:53:39.619 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6322999999999997e-05 2024-02-05T04:53:39.620 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.620 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:39.620 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.620 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.620 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.620 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:290 3.fs0 3:f8d01528:::smithi19639362-17:head [append 0~425984] snapc 3=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-02-05T04:53:39.620 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:30.384720+0000", 2024-02-05T04:53:39.620 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 9.1848773250000004, 2024-02-05T04:53:39.620 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.007038997, 2024-02-05T04:53:39.620 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:39.620 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:39.620 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:53:39.620 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:53:39.620 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:39.621 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 290 2024-02-05T04:53:39.621 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.621 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:39.621 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.621 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:39.621 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.384720+0000", 2024-02-05T04:53:39.621 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:39.621 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.621 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.621 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:39.621 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.384720+0000", 2024-02-05T04:53:39.621 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:39.621 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.621 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.621 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:39.621 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.384720+0000", 2024-02-05T04:53:39.622 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4499999999999997e-07 2024-02-05T04:53:39.622 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.622 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.622 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:39.622 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.384790+0000", 2024-02-05T04:53:39.622 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9492000000000005e-05 2024-02-05T04:53:39.622 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.622 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.622 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:39.622 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.384795+0000", 2024-02-05T04:53:39.622 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4e-06 2024-02-05T04:53:39.622 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.622 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.622 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:39.622 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.384798+0000", 2024-02-05T04:53:39.623 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4319999999999999e-06 2024-02-05T04:53:39.623 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.623 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.623 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:39.623 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.387164+0000", 2024-02-05T04:53:39.623 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0023656839999999998 2024-02-05T04:53:39.623 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.623 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.623 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:53:39.623 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.387248+0000", 2024-02-05T04:53:39.623 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3969999999999997e-05 2024-02-05T04:53:39.623 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.623 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.623 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:39.623 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.387871+0000", 2024-02-05T04:53:39.623 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00062281399999999996 2024-02-05T04:53:39.624 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.624 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.624 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:39.624 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.390152+0000", 2024-02-05T04:53:39.624 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0022811929999999999 2024-02-05T04:53:39.624 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.624 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.624 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T04:53:39.624 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.391712+0000", 2024-02-05T04:53:39.625 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00156028 2024-02-05T04:53:39.625 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.625 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.625 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:39.625 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.391759+0000", 2024-02-05T04:53:39.625 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6286999999999999e-05 2024-02-05T04:53:39.625 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.625 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:39.625 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.625 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.625 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.625 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:291 3.fs0 3:f8d01528:::smithi19639362-17:head [append 0~573440] snapc 3=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-02-05T04:53:39.626 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:30.384804+0000", 2024-02-05T04:53:39.626 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 9.1847928299999992, 2024-02-05T04:53:39.626 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.009396738, 2024-02-05T04:53:39.626 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:39.626 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:39.626 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:53:39.626 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:53:39.627 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:39.627 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 291 2024-02-05T04:53:39.627 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.627 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:39.627 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.627 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:39.627 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.384804+0000", 2024-02-05T04:53:39.627 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:39.627 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.627 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.627 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:39.627 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.384804+0000", 2024-02-05T04:53:39.627 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:39.627 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.627 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.628 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:39.628 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.384805+0000", 2024-02-05T04:53:39.628 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5400000000000002e-07 2024-02-05T04:53:39.628 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.628 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.628 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:39.628 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.385086+0000", 2024-02-05T04:53:39.628 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00028146300000000003 2024-02-05T04:53:39.628 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.628 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.628 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:39.628 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.385095+0000", 2024-02-05T04:53:39.628 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.6449999999999994e-06 2024-02-05T04:53:39.628 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.629 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.629 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:39.629 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.385101+0000", 2024-02-05T04:53:39.629 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7209999999999997e-06 2024-02-05T04:53:39.629 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.629 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.629 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:39.629 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.388360+0000", 2024-02-05T04:53:39.629 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0032590010000000001 2024-02-05T04:53:39.629 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.629 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.629 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:53:39.629 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.388427+0000", 2024-02-05T04:53:39.629 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6960999999999998e-05 2024-02-05T04:53:39.629 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.629 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.630 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:39.630 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.388961+0000", 2024-02-05T04:53:39.630 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00053408100000000005 2024-02-05T04:53:39.630 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.630 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.630 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:39.630 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.390213+0000", 2024-02-05T04:53:39.630 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001251807 2024-02-05T04:53:39.630 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.630 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.630 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T04:53:39.630 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.394146+0000", 2024-02-05T04:53:39.630 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.003933219 2024-02-05T04:53:39.630 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.630 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.631 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:39.631 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.394201+0000", 2024-02-05T04:53:39.631 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4486e-05 2024-02-05T04:53:39.631 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.631 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:39.631 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.631 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.631 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.631 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:292 3.fs0 3:f8d01528:::smithi19639362-17:head [append 0~196608] snapc 3=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-02-05T04:53:39.631 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:30.385112+0000", 2024-02-05T04:53:39.631 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 9.1844846069999999, 2024-02-05T04:53:39.631 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0092752849999999994, 2024-02-05T04:53:39.631 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:39.631 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:39.631 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:53:39.631 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:53:39.632 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:39.632 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 292 2024-02-05T04:53:39.632 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.632 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:39.632 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.632 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:39.632 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.385112+0000", 2024-02-05T04:53:39.632 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:39.632 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.632 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.632 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:39.632 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.385112+0000", 2024-02-05T04:53:39.632 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:39.632 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.633 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.633 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:39.633 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.385113+0000", 2024-02-05T04:53:39.633 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7199999999999999e-07 2024-02-05T04:53:39.633 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.633 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.633 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:39.633 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.385165+0000", 2024-02-05T04:53:39.633 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1607000000000002e-05 2024-02-05T04:53:39.633 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.633 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.633 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:39.633 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.385172+0000", 2024-02-05T04:53:39.633 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5939999999999996e-06 2024-02-05T04:53:39.633 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.633 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.634 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:39.634 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.385177+0000", 2024-02-05T04:53:39.634 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1760000000000001e-06 2024-02-05T04:53:39.634 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.634 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.634 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:39.634 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.389345+0000", 2024-02-05T04:53:39.634 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0041681640000000002 2024-02-05T04:53:39.634 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.634 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.634 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:53:39.634 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.389413+0000", 2024-02-05T04:53:39.634 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7166e-05 2024-02-05T04:53:39.634 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.634 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.635 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:39.635 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.389814+0000", 2024-02-05T04:53:39.635 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00040106400000000002 2024-02-05T04:53:39.635 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.635 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.635 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:39.635 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.390855+0000", 2024-02-05T04:53:39.635 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001041518 2024-02-05T04:53:39.635 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.635 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.635 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T04:53:39.635 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.394348+0000", 2024-02-05T04:53:39.635 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0034927159999999999 2024-02-05T04:53:39.635 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.635 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.635 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:39.636 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.394388+0000", 2024-02-05T04:53:39.636 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9708e-05 2024-02-05T04:53:39.636 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.636 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:39.636 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.636 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.636 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.636 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:293 3.fs0 3:f8d01528:::smithi19639362-17:head [setxattr _header (56) in=7b] snapc 3=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-02-05T04:53:39.636 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:30.385185+0000", 2024-02-05T04:53:39.636 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 9.1844115689999999, 2024-02-05T04:53:39.636 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.010325663000000001, 2024-02-05T04:53:39.636 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:39.636 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:39.636 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:53:39.636 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:53:39.637 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:39.637 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 293 2024-02-05T04:53:39.637 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.637 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:39.637 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.637 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:39.637 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.385185+0000", 2024-02-05T04:53:39.637 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:39.637 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.638 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.638 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:39.638 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.385185+0000", 2024-02-05T04:53:39.638 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:39.638 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.638 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.638 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:39.638 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.385186+0000", 2024-02-05T04:53:39.639 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9400000000000005e-07 2024-02-05T04:53:39.639 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.639 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.639 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:39.639 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.385189+0000", 2024-02-05T04:53:39.639 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.6809999999999998e-06 2024-02-05T04:53:39.639 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.639 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.639 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:39.639 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.385195+0000", 2024-02-05T04:53:39.639 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0630000000000002e-06 2024-02-05T04:53:39.639 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.639 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.639 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:39.639 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.385200+0000", 2024-02-05T04:53:39.639 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0560000000000004e-06 2024-02-05T04:53:39.640 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.640 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.640 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:39.640 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.390293+0000", 2024-02-05T04:53:39.640 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0050930180000000004 2024-02-05T04:53:39.640 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.640 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.640 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:53:39.640 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.390337+0000", 2024-02-05T04:53:39.640 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3721999999999999e-05 2024-02-05T04:53:39.640 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.640 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.640 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:39.640 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.390566+0000", 2024-02-05T04:53:39.640 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00022925199999999999 2024-02-05T04:53:39.641 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.641 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.641 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:39.641 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.395220+0000", 2024-02-05T04:53:39.641 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0046545919999999999 2024-02-05T04:53:39.641 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.641 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.641 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T04:53:39.641 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.395254+0000", 2024-02-05T04:53:39.641 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3522000000000002e-05 2024-02-05T04:53:39.641 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.641 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.642 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:39.642 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.395511+0000", 2024-02-05T04:53:39.642 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00025716299999999998 2024-02-05T04:53:39.642 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.642 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:39.642 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.642 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.642 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.642 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:294 3.fs0 3:f8d01528:::smithi19639362-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-02-05T04:53:39.642 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:30.385209+0000", 2024-02-05T04:53:39.642 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 9.1843879669999993, 2024-02-05T04:53:39.642 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.011163398999999999, 2024-02-05T04:53:39.642 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:39.642 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-02-05T04:53:39.642 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:53:39.642 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:53:39.643 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:39.643 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 294 2024-02-05T04:53:39.643 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.643 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:39.643 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.643 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:39.643 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.385209+0000", 2024-02-05T04:53:39.643 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:39.643 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.643 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.643 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:39.643 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.385209+0000", 2024-02-05T04:53:39.643 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:39.643 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.643 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.644 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:39.644 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.385209+0000", 2024-02-05T04:53:39.644 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.32e-07 2024-02-05T04:53:39.644 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.644 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.644 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:39.644 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.385212+0000", 2024-02-05T04:53:39.644 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1900000000000002e-06 2024-02-05T04:53:39.644 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.644 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.644 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:39.644 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.385218+0000", 2024-02-05T04:53:39.644 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4169999999999997e-06 2024-02-05T04:53:39.644 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.644 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.645 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:39.645 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.385229+0000", 2024-02-05T04:53:39.645 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0616e-05 2024-02-05T04:53:39.645 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.645 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.645 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:39.645 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.390742+0000", 2024-02-05T04:53:39.645 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0055133980000000001 2024-02-05T04:53:39.645 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.645 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.645 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for rw locks", 2024-02-05T04:53:39.645 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.390788+0000", 2024-02-05T04:53:39.645 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5627000000000002e-05 2024-02-05T04:53:39.645 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.645 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.646 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:39.646 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.395549+0000", 2024-02-05T04:53:39.646 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0047614520000000002 2024-02-05T04:53:39.646 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.646 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.646 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:53:39.646 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.395622+0000", 2024-02-05T04:53:39.646 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2916000000000005e-05 2024-02-05T04:53:39.646 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.646 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.646 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:39.646 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.396372+0000", 2024-02-05T04:53:39.646 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00075035100000000001 2024-02-05T04:53:39.646 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.646 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:39.646 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.647 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.647 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.647 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.2s6 43/31 ECSubWrite(tid=48, reqid=osd.12.0:48, at_version=43'38, trim_to=0'0, roll_forward_to=42'35))", 2024-02-05T04:53:39.647 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:35.164545+0000", 2024-02-05T04:53:39.647 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 4.4050518429999999, 2024-02-05T04:53:39.647 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0048901379999999996, 2024-02-05T04:53:39.647 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:39.647 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:39.647 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:39.647 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.647 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:39.647 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.164545+0000", 2024-02-05T04:53:39.647 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:39.647 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.647 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.648 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:39.648 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.164545+0000", 2024-02-05T04:53:39.648 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:39.648 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.648 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.648 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:39.648 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.164546+0000", 2024-02-05T04:53:39.648 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6000000000000003e-07 2024-02-05T04:53:39.648 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.648 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.648 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:39.648 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.164569+0000", 2024-02-05T04:53:39.648 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2900000000000001e-05 2024-02-05T04:53:39.648 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.648 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.649 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:39.649 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.164588+0000", 2024-02-05T04:53:39.649 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9519999999999999e-05 2024-02-05T04:53:39.649 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.649 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.649 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:39.649 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.164598+0000", 2024-02-05T04:53:39.649 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5740000000000002e-06 2024-02-05T04:53:39.649 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.649 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.649 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:39.649 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.164677+0000", 2024-02-05T04:53:39.649 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8951999999999999e-05 2024-02-05T04:53:39.649 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.649 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.650 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:39.650 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.164687+0000", 2024-02-05T04:53:39.650 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.026e-05 2024-02-05T04:53:39.650 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.650 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.650 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:39.650 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.169392+0000", 2024-02-05T04:53:39.650 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0047053850000000003 2024-02-05T04:53:39.650 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.650 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.650 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:39.650 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.169435+0000", 2024-02-05T04:53:39.650 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2787000000000002e-05 2024-02-05T04:53:39.650 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.650 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:39.650 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.651 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.651 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.651 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.es3 43/31 ECSubWrite(tid=39, reqid=osd.2.0:39, at_version=43'32, trim_to=0'0, roll_forward_to=42'30))", 2024-02-05T04:53:39.651 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:35.168861+0000", 2024-02-05T04:53:39.651 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 4.4007357320000002, 2024-02-05T04:53:39.651 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.005693867, 2024-02-05T04:53:39.651 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:39.651 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:39.651 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:39.651 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.651 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:39.651 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.168861+0000", 2024-02-05T04:53:39.651 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:39.651 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.651 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.652 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:39.652 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.168861+0000", 2024-02-05T04:53:39.652 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:39.652 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.652 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.652 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:39.652 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.168861+0000", 2024-02-05T04:53:39.652 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.01e-07 2024-02-05T04:53:39.652 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.652 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.652 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:39.652 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.168880+0000", 2024-02-05T04:53:39.652 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8878000000000001e-05 2024-02-05T04:53:39.652 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.652 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.653 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:39.653 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.168891+0000", 2024-02-05T04:53:39.653 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0502e-05 2024-02-05T04:53:39.653 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.653 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.653 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:39.653 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.168897+0000", 2024-02-05T04:53:39.653 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4470000000000003e-06 2024-02-05T04:53:39.653 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.653 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.653 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:39.653 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.168955+0000", 2024-02-05T04:53:39.653 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7788000000000003e-05 2024-02-05T04:53:39.653 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.653 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.653 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:39.654 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.168963+0000", 2024-02-05T04:53:39.654 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9340000000000006e-06 2024-02-05T04:53:39.654 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.654 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.654 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:39.654 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.174510+0000", 2024-02-05T04:53:39.654 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0055475890000000003 2024-02-05T04:53:39.654 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.654 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.654 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:39.654 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.174555+0000", 2024-02-05T04:53:39.654 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4628e-05 2024-02-05T04:53:39.654 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.654 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:39.654 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.655 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.655 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.655 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.es3 43/31 ECSubWrite(tid=40, reqid=osd.2.0:40, at_version=43'34, trim_to=0'0, roll_forward_to=42'30))", 2024-02-05T04:53:39.655 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:35.170180+0000", 2024-02-05T04:53:39.655 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 4.399417186, 2024-02-05T04:53:39.655 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0050317419999999996, 2024-02-05T04:53:39.655 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:39.655 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:39.655 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:39.655 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.655 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:39.655 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.170180+0000", 2024-02-05T04:53:39.655 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:39.655 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.655 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.655 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:39.656 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.170180+0000", 2024-02-05T04:53:39.656 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:39.656 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.656 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.656 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:39.656 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.170181+0000", 2024-02-05T04:53:39.656 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3600000000000002e-07 2024-02-05T04:53:39.656 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.656 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.656 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:39.656 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.170206+0000", 2024-02-05T04:53:39.656 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5109e-05 2024-02-05T04:53:39.656 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.656 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.656 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:39.656 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.170224+0000", 2024-02-05T04:53:39.657 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8610999999999999e-05 2024-02-05T04:53:39.657 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.657 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.657 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:39.657 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.170251+0000", 2024-02-05T04:53:39.657 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.6536000000000001e-05 2024-02-05T04:53:39.657 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.657 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.657 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:39.657 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.170368+0000", 2024-02-05T04:53:39.657 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000117104 2024-02-05T04:53:39.657 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.657 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.657 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:39.657 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.170378+0000", 2024-02-05T04:53:39.658 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0036e-05 2024-02-05T04:53:39.658 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.658 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.658 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:39.658 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.175182+0000", 2024-02-05T04:53:39.658 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.004804366 2024-02-05T04:53:39.658 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.658 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.658 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:39.658 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.175212+0000", 2024-02-05T04:53:39.658 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.9144000000000001e-05 2024-02-05T04:53:39.658 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.658 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:39.658 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.658 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.658 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.659 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.2s6 43/31 ECSubWrite(tid=49, reqid=osd.12.0:48, at_version=0'0, trim_to=0'0, roll_forward_to=43'38))", 2024-02-05T04:53:39.659 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:35.170266+0000", 2024-02-05T04:53:39.659 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 4.3993309150000002, 2024-02-05T04:53:39.659 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0049055610000000001, 2024-02-05T04:53:39.659 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:39.659 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:39.659 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:39.659 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.659 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:39.659 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.170266+0000", 2024-02-05T04:53:39.659 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:39.659 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.659 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.659 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:39.659 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.170266+0000", 2024-02-05T04:53:39.660 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:39.660 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.660 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.660 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:39.660 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.170267+0000", 2024-02-05T04:53:39.660 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.9199999999999999e-07 2024-02-05T04:53:39.660 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.660 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.660 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:39.660 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.170279+0000", 2024-02-05T04:53:39.660 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1559999999999999e-05 2024-02-05T04:53:39.660 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.660 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.660 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:39.660 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.170292+0000", 2024-02-05T04:53:39.661 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3251000000000001e-05 2024-02-05T04:53:39.661 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.661 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.661 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:39.661 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.170305+0000", 2024-02-05T04:53:39.661 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3378e-05 2024-02-05T04:53:39.661 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.661 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.661 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:39.661 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.170369+0000", 2024-02-05T04:53:39.661 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3670999999999994e-05 2024-02-05T04:53:39.661 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.661 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.661 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:39.661 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.170378+0000", 2024-02-05T04:53:39.661 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8060000000000001e-06 2024-02-05T04:53:39.662 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.662 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.662 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:39.662 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.175131+0000", 2024-02-05T04:53:39.662 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0047528830000000003 2024-02-05T04:53:39.662 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.662 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.662 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:39.662 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.175172+0000", 2024-02-05T04:53:39.662 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1019999999999997e-05 2024-02-05T04:53:39.662 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.662 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:39.662 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.662 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.662 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.662 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.1s1 43/31 ECSubWrite(tid=140, reqid=osd.13.0:140, at_version=43'29, trim_to=0'0, roll_forward_to=42'27))", 2024-02-05T04:53:39.663 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:35.183203+0000", 2024-02-05T04:53:39.663 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 4.3863943430000001, 2024-02-05T04:53:39.663 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0053541140000000001, 2024-02-05T04:53:39.663 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:39.663 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:39.663 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:39.663 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.663 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:39.663 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.183203+0000", 2024-02-05T04:53:39.663 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:39.663 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.663 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.663 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:39.663 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.183203+0000", 2024-02-05T04:53:39.664 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:39.664 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.664 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.664 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:39.664 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.183203+0000", 2024-02-05T04:53:39.664 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2800000000000002e-07 2024-02-05T04:53:39.664 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.664 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.664 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:39.664 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.183229+0000", 2024-02-05T04:53:39.664 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.6242e-05 2024-02-05T04:53:39.664 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.665 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.665 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:39.665 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.183245+0000", 2024-02-05T04:53:39.665 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5644000000000001e-05 2024-02-05T04:53:39.665 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.665 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.665 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:39.665 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.183253+0000", 2024-02-05T04:53:39.665 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.524e-06 2024-02-05T04:53:39.665 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.665 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.665 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:39.665 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.183349+0000", 2024-02-05T04:53:39.665 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5149000000000007e-05 2024-02-05T04:53:39.665 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.665 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.666 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:39.666 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.183358+0000", 2024-02-05T04:53:39.666 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.363e-06 2024-02-05T04:53:39.666 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.666 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.666 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:39.666 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.188506+0000", 2024-02-05T04:53:39.666 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0051482150000000003 2024-02-05T04:53:39.666 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.666 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.666 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:39.666 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.188557+0000", 2024-02-05T04:53:39.666 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0549e-05 2024-02-05T04:53:39.666 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.666 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:39.667 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.667 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.667 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.667 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.1s1 43/31 ECSubWrite(tid=142, reqid=osd.13.0:142, at_version=43'31, trim_to=0'0, roll_forward_to=42'27))", 2024-02-05T04:53:39.667 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:35.184245+0000", 2024-02-05T04:53:39.667 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 4.3853514860000002, 2024-02-05T04:53:39.667 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0048051450000000002, 2024-02-05T04:53:39.667 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:39.667 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:39.667 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:39.668 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.668 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:39.668 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.184245+0000", 2024-02-05T04:53:39.668 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:39.668 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.668 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.668 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:39.668 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.184245+0000", 2024-02-05T04:53:39.668 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:39.668 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.668 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.668 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:39.668 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.184246+0000", 2024-02-05T04:53:39.668 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1699999999999997e-07 2024-02-05T04:53:39.668 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.669 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.669 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:39.669 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.184268+0000", 2024-02-05T04:53:39.669 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1552e-05 2024-02-05T04:53:39.669 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.669 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.669 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:39.669 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.184282+0000", 2024-02-05T04:53:39.669 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4201e-05 2024-02-05T04:53:39.669 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.669 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.670 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:39.670 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.184291+0000", 2024-02-05T04:53:39.670 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8570000000000001e-06 2024-02-05T04:53:39.670 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.670 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.670 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:39.670 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.184351+0000", 2024-02-05T04:53:39.670 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0069999999999999e-05 2024-02-05T04:53:39.670 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.670 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.670 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:39.670 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.184358+0000", 2024-02-05T04:53:39.670 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7689999999999999e-06 2024-02-05T04:53:39.670 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.670 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.671 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:39.671 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.189012+0000", 2024-02-05T04:53:39.671 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0046540970000000003 2024-02-05T04:53:39.671 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:39.671 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:39.671 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:39.671 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.189051+0000", 2024-02-05T04:53:39.671 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8881999999999998e-05 2024-02-05T04:53:39.671 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.671 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:39.671 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.671 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:39.671 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:39.671 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T04:53:39.673 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-02-05T04:53:39.673 INFO:tasks.ceph.osd.15.smithi196.stderr:2024-02-05T04:53:39.671+0000 7f1facc45640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 34611) UID: 0 2024-02-05T04:53:39.774 INFO:tasks.ceph.osd.1.smithi091.stderr:2024-02-05T04:53:39.772+0000 7ff9dc283640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34275) UID: 0 2024-02-05T04:53:39.819 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T04:53:39.820 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-02-05T04:53:39.820 INFO:teuthology.orchestra.run.smithi196.stdout: "num_ops": 0 2024-02-05T04:53:39.820 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T04:53:39.828 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-02-05T04:53:39.874 INFO:tasks.ceph.osd.9.smithi091.stderr:2024-02-05T04:53:39.873+0000 7efc18827640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 34279) UID: 0 2024-02-05T04:53:39.933 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T04:53:39.934 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-02-05T04:53:39.934 INFO:teuthology.orchestra.run.smithi196.stdout: "complaint_time": 30, 2024-02-05T04:53:39.934 INFO:teuthology.orchestra.run.smithi196.stdout: "num_blocked_ops": 0 2024-02-05T04:53:39.934 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T04:53:39.943 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-02-05T04:53:39.944 INFO:teuthology.orchestra.run.smithi062.stderr:osd set-full-ratio 0.95 2024-02-05T04:53:39.947 INFO:tasks.rados.rados.0.smithi196.stdout:159: done (3 left) 2024-02-05T04:53:39.947 INFO:tasks.rados.rados.0.smithi196.stdout:160: done (2 left) 2024-02-05T04:53:39.947 INFO:tasks.rados.rados.0.smithi196.stdout:161: done (1 left) 2024-02-05T04:53:39.947 INFO:tasks.rados.rados.0.smithi196.stdout:162: done (0 left) 2024-02-05T04:53:39.947 INFO:tasks.rados.rados.0.smithi196.stdout:163: snap_create 2024-02-05T04:53:39.976 INFO:tasks.ceph.osd.8.smithi062.stderr:2024-02-05T04:53:39.975+0000 7f9385bf2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 34693) UID: 0 2024-02-05T04:53:40.051 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T04:53:40.051 INFO:teuthology.orchestra.run.smithi196.stdout: "size": 20, 2024-02-05T04:53:40.051 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 600, 2024-02-05T04:53:40.051 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-02-05T04:53:40.051 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.052 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.3s3 37/31 ECSubWrite(tid=27, reqid=osd.4.0:26, at_version=0'0, trim_to=0'0, roll_forward_to=37'10))", 2024-02-05T04:53:40.052 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:29.131465+0000", 2024-02-05T04:53:40.052 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 10.917870820999999, 2024-02-05T04:53:40.052 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0023720080000000001, 2024-02-05T04:53:40.052 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:40.052 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:40.052 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:40.052 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.052 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:40.052 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.131465+0000", 2024-02-05T04:53:40.052 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:40.052 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.052 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.052 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:40.053 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.131465+0000", 2024-02-05T04:53:40.053 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:40.053 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.053 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.053 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:40.053 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.131466+0000", 2024-02-05T04:53:40.053 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1999999999999998e-07 2024-02-05T04:53:40.053 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.053 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.054 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:40.054 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.131478+0000", 2024-02-05T04:53:40.054 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2438e-05 2024-02-05T04:53:40.054 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.054 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.054 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:40.054 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.131497+0000", 2024-02-05T04:53:40.054 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8386999999999999e-05 2024-02-05T04:53:40.054 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.054 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.054 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:40.054 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.131506+0000", 2024-02-05T04:53:40.054 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.7219999999999994e-06 2024-02-05T04:53:40.054 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.055 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.055 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:40.055 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.131568+0000", 2024-02-05T04:53:40.055 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1722000000000003e-05 2024-02-05T04:53:40.055 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.055 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.056 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:40.056 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.131578+0000", 2024-02-05T04:53:40.056 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.9699999999999994e-06 2024-02-05T04:53:40.056 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.056 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.056 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:40.056 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.133798+0000", 2024-02-05T04:53:40.056 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0022196120000000001 2024-02-05T04:53:40.056 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.056 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.056 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:40.056 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.133837+0000", 2024-02-05T04:53:40.057 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9236999999999997e-05 2024-02-05T04:53:40.057 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:40.057 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:40.057 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:40.057 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.057 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.057 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.3s3 37/31 ECSubWrite(tid=29, reqid=client.4634.0:218, at_version=37'12, trim_to=0'0, roll_forward_to=37'10))", 2024-02-05T04:53:40.057 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:29.134656+0000", 2024-02-05T04:53:40.057 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 10.914679454, 2024-02-05T04:53:40.057 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0044628000000000003, 2024-02-05T04:53:40.057 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:40.057 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:40.057 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:40.057 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.057 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:40.058 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.134656+0000", 2024-02-05T04:53:40.058 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:40.058 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.058 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.058 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:40.058 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.134656+0000", 2024-02-05T04:53:40.058 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:40.058 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.058 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.058 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:40.058 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.134657+0000", 2024-02-05T04:53:40.058 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.4099999999999997e-07 2024-02-05T04:53:40.058 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.058 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.058 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:40.059 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135965+0000", 2024-02-05T04:53:40.059 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001307674 2024-02-05T04:53:40.059 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.059 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.059 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:40.059 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135980+0000", 2024-02-05T04:53:40.059 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5400000000000002e-05 2024-02-05T04:53:40.059 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.059 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.059 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:40.059 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135990+0000", 2024-02-05T04:53:40.059 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2830000000000007e-06 2024-02-05T04:53:40.059 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.059 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.059 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:40.060 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.136086+0000", 2024-02-05T04:53:40.060 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6457000000000002e-05 2024-02-05T04:53:40.060 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.060 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.060 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:40.060 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.136096+0000", 2024-02-05T04:53:40.060 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0138e-05 2024-02-05T04:53:40.060 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.060 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.060 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:40.060 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.139087+0000", 2024-02-05T04:53:40.060 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0029904200000000001 2024-02-05T04:53:40.060 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.060 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.061 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:40.061 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.139119+0000", 2024-02-05T04:53:40.061 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2487000000000003e-05 2024-02-05T04:53:40.061 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:40.061 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:40.061 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:40.061 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.061 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.061 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.3s3 37/31 ECSubWrite(tid=33, reqid=client.4634.0:220, at_version=0'0, trim_to=0'0, roll_forward_to=37'14))", 2024-02-05T04:53:40.061 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:29.144162+0000", 2024-02-05T04:53:40.061 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 10.905174174999999, 2024-02-05T04:53:40.061 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0036064249999999999, 2024-02-05T04:53:40.061 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:40.061 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:40.061 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:40.061 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.062 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:40.062 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.144162+0000", 2024-02-05T04:53:40.062 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:40.062 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.062 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.062 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:40.062 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.144162+0000", 2024-02-05T04:53:40.062 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:40.062 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.062 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.062 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:40.062 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.144163+0000", 2024-02-05T04:53:40.062 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1399999999999995e-07 2024-02-05T04:53:40.063 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.063 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.063 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:40.063 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.144176+0000", 2024-02-05T04:53:40.063 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3200999999999999e-05 2024-02-05T04:53:40.063 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.063 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.063 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:40.063 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.144195+0000", 2024-02-05T04:53:40.063 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9514999999999999e-05 2024-02-05T04:53:40.063 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.063 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.063 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:40.063 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.144205+0000", 2024-02-05T04:53:40.064 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.9830000000000008e-06 2024-02-05T04:53:40.064 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.064 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.064 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:40.064 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.144356+0000", 2024-02-05T04:53:40.064 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000150898 2024-02-05T04:53:40.064 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.064 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.064 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:40.064 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.144366+0000", 2024-02-05T04:53:40.064 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0183999999999999e-05 2024-02-05T04:53:40.064 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.064 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.064 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:40.064 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.147723+0000", 2024-02-05T04:53:40.065 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0033561839999999999 2024-02-05T04:53:40.065 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.065 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.065 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:40.065 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.147768+0000", 2024-02-05T04:53:40.065 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5546000000000001e-05 2024-02-05T04:53:40.065 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:40.065 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:40.065 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:40.065 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.065 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.065 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.2s4 37/31 ECSubWrite(tid=20, reqid=client.4634.0:252, at_version=37'12, trim_to=0'0, roll_forward_to=34'8))", 2024-02-05T04:53:40.065 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:29.201419+0000", 2024-02-05T04:53:40.065 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 10.847917054, 2024-02-05T04:53:40.065 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0028812999999999998, 2024-02-05T04:53:40.065 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:40.066 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:40.066 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:40.066 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.066 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:40.066 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.201419+0000", 2024-02-05T04:53:40.066 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:40.066 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.066 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.066 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:40.066 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.201419+0000", 2024-02-05T04:53:40.066 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:40.066 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.066 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.066 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:40.067 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.201419+0000", 2024-02-05T04:53:40.067 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2099999999999996e-07 2024-02-05T04:53:40.067 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.067 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.067 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:40.067 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.201435+0000", 2024-02-05T04:53:40.067 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5973000000000001e-05 2024-02-05T04:53:40.067 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.067 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.067 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:40.067 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.201449+0000", 2024-02-05T04:53:40.067 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3903000000000001e-05 2024-02-05T04:53:40.067 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.067 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.067 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:40.068 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.201463+0000", 2024-02-05T04:53:40.068 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4022999999999999e-05 2024-02-05T04:53:40.068 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.068 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.068 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:40.068 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.201521+0000", 2024-02-05T04:53:40.068 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7737e-05 2024-02-05T04:53:40.068 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.068 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.068 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:40.068 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.201530+0000", 2024-02-05T04:53:40.068 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2180000000000005e-06 2024-02-05T04:53:40.068 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.069 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.069 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:40.069 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.204274+0000", 2024-02-05T04:53:40.069 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0027433420000000002 2024-02-05T04:53:40.069 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.069 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.069 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:40.069 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.204300+0000", 2024-02-05T04:53:40.069 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.6383000000000001e-05 2024-02-05T04:53:40.069 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:40.069 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:40.070 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:40.070 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.070 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.070 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.2s4 37/31 ECSubWrite(tid=21, reqid=client.4634.0:253, at_version=37'14, trim_to=0'0, roll_forward_to=37'10))", 2024-02-05T04:53:40.070 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:29.206768+0000", 2024-02-05T04:53:40.070 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 10.842567836000001, 2024-02-05T04:53:40.070 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0041686320000000002, 2024-02-05T04:53:40.071 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:40.071 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:40.071 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:40.071 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.071 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:40.071 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.206768+0000", 2024-02-05T04:53:40.071 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:40.071 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.071 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.071 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:40.071 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.206768+0000", 2024-02-05T04:53:40.071 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:40.071 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.071 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.071 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:40.071 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.206769+0000", 2024-02-05T04:53:40.072 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4e-07 2024-02-05T04:53:40.072 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.072 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.072 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:40.072 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.207503+0000", 2024-02-05T04:53:40.072 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00073375399999999998 2024-02-05T04:53:40.072 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.072 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.072 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:40.072 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.207517+0000", 2024-02-05T04:53:40.073 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4746000000000001e-05 2024-02-05T04:53:40.073 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.073 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.073 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:40.073 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.207526+0000", 2024-02-05T04:53:40.073 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.9989999999999997e-06 2024-02-05T04:53:40.073 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.073 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.073 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:40.073 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.207602+0000", 2024-02-05T04:53:40.073 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5783000000000003e-05 2024-02-05T04:53:40.073 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.073 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.074 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:40.074 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.207622+0000", 2024-02-05T04:53:40.074 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0159000000000001e-05 2024-02-05T04:53:40.074 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.074 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.074 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:40.074 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.210902+0000", 2024-02-05T04:53:40.074 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0032794450000000002 2024-02-05T04:53:40.074 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.074 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.074 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:40.074 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.210937+0000", 2024-02-05T04:53:40.074 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4906000000000001e-05 2024-02-05T04:53:40.075 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:40.075 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:40.075 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:40.075 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.075 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.075 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.2s4 37/31 ECSubWrite(tid=22, reqid=client.4634.0:254, at_version=37'15, trim_to=0'0, roll_forward_to=37'10))", 2024-02-05T04:53:40.075 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:29.209788+0000", 2024-02-05T04:53:40.075 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 10.839548161, 2024-02-05T04:53:40.075 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0035378919999999999, 2024-02-05T04:53:40.076 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:40.076 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:40.076 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:40.076 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.076 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:40.076 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.209788+0000", 2024-02-05T04:53:40.076 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:40.076 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.076 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.076 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:40.076 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.209788+0000", 2024-02-05T04:53:40.076 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:40.076 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.077 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.077 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:40.077 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.209788+0000", 2024-02-05T04:53:40.077 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.37e-07 2024-02-05T04:53:40.077 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.077 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.077 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:40.077 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.209975+0000", 2024-02-05T04:53:40.077 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00018678400000000001 2024-02-05T04:53:40.077 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.077 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.077 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:40.077 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.209985+0000", 2024-02-05T04:53:40.077 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.9219999999999992e-06 2024-02-05T04:53:40.077 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.078 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.078 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:40.078 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.209991+0000", 2024-02-05T04:53:40.078 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7980000000000002e-06 2024-02-05T04:53:40.078 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.078 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.078 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:40.078 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.210048+0000", 2024-02-05T04:53:40.078 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7031999999999999e-05 2024-02-05T04:53:40.078 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.079 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.079 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:40.079 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.210057+0000", 2024-02-05T04:53:40.079 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.149e-06 2024-02-05T04:53:40.079 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.080 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.080 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:40.081 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.213281+0000", 2024-02-05T04:53:40.081 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0032242350000000002 2024-02-05T04:53:40.081 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.081 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.081 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:40.081 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.213326+0000", 2024-02-05T04:53:40.081 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4335000000000002e-05 2024-02-05T04:53:40.081 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:40.081 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:40.081 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:40.081 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.081 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.081 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.2s4 37/31 ECSubWrite(tid=23, reqid=client.4634.0:255, at_version=37'16, trim_to=0'0, roll_forward_to=37'10))", 2024-02-05T04:53:40.081 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:29.210005+0000", 2024-02-05T04:53:40.081 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 10.839331061999999, 2024-02-05T04:53:40.081 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0033454790000000002, 2024-02-05T04:53:40.082 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:40.082 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:40.082 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:40.082 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.082 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:40.082 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.210005+0000", 2024-02-05T04:53:40.082 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:40.082 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.082 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.083 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:40.083 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.210005+0000", 2024-02-05T04:53:40.083 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:40.083 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.083 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.083 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:40.083 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.210005+0000", 2024-02-05T04:53:40.083 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8599999999999999e-07 2024-02-05T04:53:40.083 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.083 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.083 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:40.083 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.210017+0000", 2024-02-05T04:53:40.083 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2319e-05 2024-02-05T04:53:40.083 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.083 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.084 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:40.084 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.210022+0000", 2024-02-05T04:53:40.084 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.091e-06 2024-02-05T04:53:40.084 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.084 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.084 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:40.084 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.210026+0000", 2024-02-05T04:53:40.084 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7780000000000001e-06 2024-02-05T04:53:40.084 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.084 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.084 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:40.084 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.210682+0000", 2024-02-05T04:53:40.084 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00065532700000000004 2024-02-05T04:53:40.084 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.085 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.085 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:40.085 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.210691+0000", 2024-02-05T04:53:40.085 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.7389999999999994e-06 2024-02-05T04:53:40.085 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.085 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.085 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:40.085 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.213329+0000", 2024-02-05T04:53:40.085 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0026372909999999999 2024-02-05T04:53:40.085 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.085 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.085 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:40.085 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.213350+0000", 2024-02-05T04:53:40.085 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1648e-05 2024-02-05T04:53:40.085 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:40.086 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:40.086 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:40.086 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.086 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.086 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.2s4 37/31 ECSubWrite(tid=24, reqid=client.4634.0:257, at_version=37'18, trim_to=0'0, roll_forward_to=37'12))", 2024-02-05T04:53:40.086 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:29.211442+0000", 2024-02-05T04:53:40.086 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 10.837893770000001, 2024-02-05T04:53:40.086 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0045198679999999998, 2024-02-05T04:53:40.086 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:40.086 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:40.086 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:40.086 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.086 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:40.086 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.211442+0000", 2024-02-05T04:53:40.086 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:40.086 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.087 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.087 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:40.087 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.211442+0000", 2024-02-05T04:53:40.087 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:40.087 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.087 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.087 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:40.087 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.211443+0000", 2024-02-05T04:53:40.087 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.3799999999999996e-07 2024-02-05T04:53:40.087 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.087 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.087 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:40.087 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.211459+0000", 2024-02-05T04:53:40.087 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6405999999999998e-05 2024-02-05T04:53:40.087 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.087 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.088 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:40.088 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.211469+0000", 2024-02-05T04:53:40.088 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0168e-05 2024-02-05T04:53:40.088 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.088 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.088 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:40.088 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.211478+0000", 2024-02-05T04:53:40.088 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1160000000000005e-06 2024-02-05T04:53:40.088 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.088 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.088 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:40.088 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.211545+0000", 2024-02-05T04:53:40.088 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6080000000000004e-05 2024-02-05T04:53:40.089 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.089 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.089 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:40.089 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.211556+0000", 2024-02-05T04:53:40.089 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1080999999999999e-05 2024-02-05T04:53:40.089 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.089 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.089 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:40.089 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.215935+0000", 2024-02-05T04:53:40.089 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0043795750000000001 2024-02-05T04:53:40.089 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.089 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.089 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:40.089 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.215962+0000", 2024-02-05T04:53:40.089 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.6704e-05 2024-02-05T04:53:40.090 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:40.090 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:40.090 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:40.090 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.090 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.090 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.6s3 38/31 ECSubWrite(tid=90, reqid=osd.13.0:90, at_version=38'11, trim_to=0'0, roll_forward_to=37'8))", 2024-02-05T04:53:40.090 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:30.142148+0000", 2024-02-05T04:53:40.090 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 9.9071873260000007, 2024-02-05T04:53:40.090 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0032515859999999999, 2024-02-05T04:53:40.090 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:40.090 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:40.090 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:40.090 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.090 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:40.090 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.142148+0000", 2024-02-05T04:53:40.091 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:40.091 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.091 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.091 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:40.091 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.142148+0000", 2024-02-05T04:53:40.091 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:40.091 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.091 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.091 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:40.091 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.142149+0000", 2024-02-05T04:53:40.091 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.9299999999999996e-07 2024-02-05T04:53:40.091 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.091 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.091 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:40.091 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.142175+0000", 2024-02-05T04:53:40.091 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5211e-05 2024-02-05T04:53:40.092 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.092 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.092 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:40.092 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.142334+0000", 2024-02-05T04:53:40.092 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000159753 2024-02-05T04:53:40.092 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.092 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.092 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:40.092 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.142353+0000", 2024-02-05T04:53:40.092 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9069999999999999e-05 2024-02-05T04:53:40.092 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.092 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.092 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:40.092 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.142436+0000", 2024-02-05T04:53:40.093 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.2162000000000004e-05 2024-02-05T04:53:40.093 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.093 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.093 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:40.093 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.142451+0000", 2024-02-05T04:53:40.093 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5330000000000001e-05 2024-02-05T04:53:40.093 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.093 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.093 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:40.093 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.145360+0000", 2024-02-05T04:53:40.093 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0029090869999999999 2024-02-05T04:53:40.093 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.093 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.093 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:40.093 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.145400+0000", 2024-02-05T04:53:40.093 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0080000000000003e-05 2024-02-05T04:53:40.094 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:40.094 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:40.094 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:40.094 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.094 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.094 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.2s4 38/31 ECSubWrite(tid=27, reqid=osd.12.0:27, at_version=38'20, trim_to=0'0, roll_forward_to=37'18))", 2024-02-05T04:53:40.094 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:30.143682+0000", 2024-02-05T04:53:40.094 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 9.9056541540000005, 2024-02-05T04:53:40.094 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0047163500000000002, 2024-02-05T04:53:40.094 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:40.094 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:40.094 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:40.094 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.094 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:40.094 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.143682+0000", 2024-02-05T04:53:40.094 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:40.095 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.095 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.095 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:40.095 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.143682+0000", 2024-02-05T04:53:40.095 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:40.095 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.095 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.095 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:40.095 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.143682+0000", 2024-02-05T04:53:40.095 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.91e-07 2024-02-05T04:53:40.095 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.095 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.095 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:40.095 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.143721+0000", 2024-02-05T04:53:40.095 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9177000000000001e-05 2024-02-05T04:53:40.096 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.096 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.096 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:40.096 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.143735+0000", 2024-02-05T04:53:40.096 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3937000000000001e-05 2024-02-05T04:53:40.096 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.096 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.096 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:40.096 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.143744+0000", 2024-02-05T04:53:40.096 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8759999999999994e-06 2024-02-05T04:53:40.096 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.096 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.096 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:40.096 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.143828+0000", 2024-02-05T04:53:40.096 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4036999999999995e-05 2024-02-05T04:53:40.097 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.097 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.097 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:40.097 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.143839+0000", 2024-02-05T04:53:40.097 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.146e-05 2024-02-05T04:53:40.097 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.097 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.097 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:40.097 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.148350+0000", 2024-02-05T04:53:40.097 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0045110239999999998 2024-02-05T04:53:40.097 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.097 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.097 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:40.097 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.148398+0000", 2024-02-05T04:53:40.098 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7547999999999999e-05 2024-02-05T04:53:40.098 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:40.098 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:40.098 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:40.098 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.098 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.098 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.2s4 38/31 ECSubWrite(tid=28, reqid=osd.12.0:28, at_version=38'22, trim_to=0'0, roll_forward_to=37'18))", 2024-02-05T04:53:40.098 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:30.144323+0000", 2024-02-05T04:53:40.098 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 9.9050123380000006, 2024-02-05T04:53:40.098 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0044628350000000001, 2024-02-05T04:53:40.098 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:40.098 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:40.098 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:40.098 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.098 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:40.099 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.144323+0000", 2024-02-05T04:53:40.099 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:40.099 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.099 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.099 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:40.099 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.144323+0000", 2024-02-05T04:53:40.099 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:40.099 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.099 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.099 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:40.099 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.144324+0000", 2024-02-05T04:53:40.099 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0200000000000001e-07 2024-02-05T04:53:40.099 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.099 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.100 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:40.100 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.144345+0000", 2024-02-05T04:53:40.100 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0843999999999999e-05 2024-02-05T04:53:40.100 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.100 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.100 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:40.100 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.144359+0000", 2024-02-05T04:53:40.100 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4365999999999999e-05 2024-02-05T04:53:40.100 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.100 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.100 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:40.100 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.144376+0000", 2024-02-05T04:53:40.100 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7088999999999999e-05 2024-02-05T04:53:40.100 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.100 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.101 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:40.101 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.144448+0000", 2024-02-05T04:53:40.101 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1088999999999996e-05 2024-02-05T04:53:40.101 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.101 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.101 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:40.101 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.144458+0000", 2024-02-05T04:53:40.101 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0566e-05 2024-02-05T04:53:40.101 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.101 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.101 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:40.101 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.148748+0000", 2024-02-05T04:53:40.101 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0042896089999999998 2024-02-05T04:53:40.102 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.102 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.102 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:40.102 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.148786+0000", 2024-02-05T04:53:40.102 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8569999999999998e-05 2024-02-05T04:53:40.102 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:40.102 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:40.102 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:40.102 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.102 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.102 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.2s4 38/31 ECSubWrite(tid=30, reqid=osd.12.0:30, at_version=38'24, trim_to=0'0, roll_forward_to=38'22))", 2024-02-05T04:53:40.102 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:30.155267+0000", 2024-02-05T04:53:40.102 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 9.8940684139999995, 2024-02-05T04:53:40.102 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0044217589999999999, 2024-02-05T04:53:40.102 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:40.103 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:40.103 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:40.103 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.103 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:40.103 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.155267+0000", 2024-02-05T04:53:40.103 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:40.103 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.103 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.103 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:40.103 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.155267+0000", 2024-02-05T04:53:40.103 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:40.103 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.103 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.103 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:40.104 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.155268+0000", 2024-02-05T04:53:40.104 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8400000000000004e-07 2024-02-05T04:53:40.104 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.104 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.104 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:40.104 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.155286+0000", 2024-02-05T04:53:40.104 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8269000000000001e-05 2024-02-05T04:53:40.104 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.104 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.104 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:40.104 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.155298+0000", 2024-02-05T04:53:40.104 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1387e-05 2024-02-05T04:53:40.104 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.104 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.104 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:40.105 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.155304+0000", 2024-02-05T04:53:40.105 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.2750000000000001e-06 2024-02-05T04:53:40.105 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.105 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.105 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:40.105 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.155363+0000", 2024-02-05T04:53:40.105 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8847000000000001e-05 2024-02-05T04:53:40.105 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.105 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.105 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:40.105 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.155372+0000", 2024-02-05T04:53:40.105 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.7110000000000001e-06 2024-02-05T04:53:40.105 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.105 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.105 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:40.106 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.159663+0000", 2024-02-05T04:53:40.106 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0042909469999999998 2024-02-05T04:53:40.106 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.106 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.106 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:40.106 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.159689+0000", 2024-02-05T04:53:40.106 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.6639e-05 2024-02-05T04:53:40.106 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:40.106 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:40.106 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:40.106 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.106 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.106 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.8s5 42/31 ECSubWrite(tid=60, reqid=client.4634.0:390, at_version=42'17, trim_to=0'0, roll_forward_to=40'15))", 2024-02-05T04:53:40.106 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:34.181140+0000", 2024-02-05T04:53:40.106 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 5.8681954620000001, 2024-02-05T04:53:40.106 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0045097100000000001, 2024-02-05T04:53:40.107 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:40.107 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:40.107 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:40.107 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.107 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:40.107 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.181140+0000", 2024-02-05T04:53:40.107 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:40.107 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.107 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.107 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:40.107 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.181140+0000", 2024-02-05T04:53:40.107 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:40.107 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.107 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.107 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:40.107 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.181141+0000", 2024-02-05T04:53:40.108 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3900000000000004e-07 2024-02-05T04:53:40.108 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.108 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.108 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:40.108 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.181945+0000", 2024-02-05T04:53:40.108 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00080416699999999995 2024-02-05T04:53:40.108 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.108 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.108 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:40.108 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.181960+0000", 2024-02-05T04:53:40.108 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4809e-05 2024-02-05T04:53:40.108 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.108 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.108 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:40.108 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.181970+0000", 2024-02-05T04:53:40.109 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.4110000000000002e-06 2024-02-05T04:53:40.109 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.109 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.109 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:40.109 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.182039+0000", 2024-02-05T04:53:40.109 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9497000000000002e-05 2024-02-05T04:53:40.109 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.109 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.109 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:40.109 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.182047+0000", 2024-02-05T04:53:40.109 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3769999999999996e-06 2024-02-05T04:53:40.109 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.109 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.109 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:40.110 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.185615+0000", 2024-02-05T04:53:40.110 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.003567387 2024-02-05T04:53:40.110 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.110 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.110 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:40.110 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.185650+0000", 2024-02-05T04:53:40.110 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5222999999999998e-05 2024-02-05T04:53:40.110 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:40.110 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:40.110 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:40.110 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.110 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.110 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "pg_update_log_missing(3.2s4 epoch 43/31 rep_tid 47 entries 43'36 (0'0) error 3:4d05e2f1:::smithi19639362-34:head by client.4634.0:397 0.000000 -2 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 39'33)", 2024-02-05T04:53:40.110 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:35.164200+0000", 2024-02-05T04:53:40.110 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 4.8851354699999998, 2024-02-05T04:53:40.110 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0023764429999999998, 2024-02-05T04:53:40.111 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:40.111 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:40.111 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:40.111 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.111 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:40.111 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.164200+0000", 2024-02-05T04:53:40.111 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:40.111 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.111 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.111 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:40.111 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.164200+0000", 2024-02-05T04:53:40.112 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:40.112 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.112 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.112 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:40.112 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.164201+0000", 2024-02-05T04:53:40.112 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7199999999999998e-07 2024-02-05T04:53:40.112 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.112 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.112 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:40.112 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.164210+0000", 2024-02-05T04:53:40.112 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.3989999999999993e-06 2024-02-05T04:53:40.112 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.113 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.113 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:40.113 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.164231+0000", 2024-02-05T04:53:40.113 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0267000000000001e-05 2024-02-05T04:53:40.113 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.113 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.113 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:40.113 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.164245+0000", 2024-02-05T04:53:40.113 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.464e-05 2024-02-05T04:53:40.113 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.113 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.113 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:40.114 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.164342+0000", 2024-02-05T04:53:40.114 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6951000000000001e-05 2024-02-05T04:53:40.114 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.114 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.114 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:40.114 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.166577+0000", 2024-02-05T04:53:40.114 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0022345139999999999 2024-02-05T04:53:40.114 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:40.114 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:40.114 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:40.114 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.114 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.114 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.2s4 43/31 ECSubWrite(tid=48, reqid=osd.12.0:48, at_version=43'38, trim_to=0'0, roll_forward_to=42'35))", 2024-02-05T04:53:40.114 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:35.164721+0000", 2024-02-05T04:53:40.114 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 4.8846148810000001, 2024-02-05T04:53:40.115 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0046293300000000001, 2024-02-05T04:53:40.115 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:40.115 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:40.115 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:40.115 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.115 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:40.115 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.164721+0000", 2024-02-05T04:53:40.115 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:40.115 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.115 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.115 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:40.115 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.164721+0000", 2024-02-05T04:53:40.115 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:40.115 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.115 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.115 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:40.116 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.164722+0000", 2024-02-05T04:53:40.116 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5099999999999999e-07 2024-02-05T04:53:40.116 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.116 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.116 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:40.116 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.164747+0000", 2024-02-05T04:53:40.116 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.4861000000000001e-05 2024-02-05T04:53:40.116 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.116 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.116 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:40.116 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.164758+0000", 2024-02-05T04:53:40.116 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1644999999999999e-05 2024-02-05T04:53:40.116 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.116 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.116 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:40.116 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.164767+0000", 2024-02-05T04:53:40.117 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1609999999999992e-06 2024-02-05T04:53:40.117 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.117 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.117 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:40.117 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.164827+0000", 2024-02-05T04:53:40.117 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0078000000000003e-05 2024-02-05T04:53:40.117 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.117 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.117 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:40.117 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.164837+0000", 2024-02-05T04:53:40.118 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8609999999999993e-06 2024-02-05T04:53:40.118 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.118 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.118 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:40.118 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.169298+0000", 2024-02-05T04:53:40.118 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0044605340000000004 2024-02-05T04:53:40.118 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.118 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.118 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:40.118 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.169350+0000", 2024-02-05T04:53:40.118 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2438999999999997e-05 2024-02-05T04:53:40.118 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:40.118 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:40.118 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:40.118 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.118 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.119 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.8s5 43/31 ECSubWrite(tid=62, reqid=osd.4.0:62, at_version=43'19, trim_to=0'0, roll_forward_to=42'17))", 2024-02-05T04:53:40.119 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:35.172528+0000", 2024-02-05T04:53:40.119 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 4.8768081350000001, 2024-02-05T04:53:40.119 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0048987559999999998, 2024-02-05T04:53:40.119 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:40.119 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:40.119 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:40.119 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.119 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:40.119 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.172528+0000", 2024-02-05T04:53:40.119 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:40.119 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.119 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.119 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:40.119 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.172528+0000", 2024-02-05T04:53:40.120 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:40.120 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.120 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.120 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:40.120 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.172528+0000", 2024-02-05T04:53:40.120 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.86e-07 2024-02-05T04:53:40.120 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.120 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.120 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:40.120 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.172551+0000", 2024-02-05T04:53:40.120 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3411999999999999e-05 2024-02-05T04:53:40.120 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.120 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.120 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:40.120 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.172564+0000", 2024-02-05T04:53:40.121 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3117e-05 2024-02-05T04:53:40.121 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.121 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.121 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:40.121 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.172576+0000", 2024-02-05T04:53:40.121 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1925e-05 2024-02-05T04:53:40.121 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.121 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.121 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:40.121 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.172645+0000", 2024-02-05T04:53:40.121 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8647999999999997e-05 2024-02-05T04:53:40.121 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.121 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.121 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:40.121 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.172655+0000", 2024-02-05T04:53:40.122 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0295999999999999e-05 2024-02-05T04:53:40.122 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.122 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.122 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:40.122 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.177348+0000", 2024-02-05T04:53:40.122 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0046929420000000003 2024-02-05T04:53:40.122 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.122 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.122 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:40.122 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.177426+0000", 2024-02-05T04:53:40.122 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8230000000000001e-05 2024-02-05T04:53:40.122 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:40.122 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:40.122 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:40.123 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.123 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.123 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.6s3 43/31 ECSubWrite(tid=138, reqid=osd.13.0:138, at_version=43'24, trim_to=0'0, roll_forward_to=40'22))", 2024-02-05T04:53:40.123 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:35.172529+0000", 2024-02-05T04:53:40.123 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 4.8768063550000003, 2024-02-05T04:53:40.123 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0049249749999999998, 2024-02-05T04:53:40.123 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:40.123 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:40.123 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:40.123 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.123 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:40.123 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.172529+0000", 2024-02-05T04:53:40.123 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:40.123 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.123 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.124 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:40.124 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.172529+0000", 2024-02-05T04:53:40.124 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:40.124 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.124 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.124 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:40.124 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.172530+0000", 2024-02-05T04:53:40.124 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3799999999999999e-07 2024-02-05T04:53:40.124 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.124 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.124 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:40.124 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.172550+0000", 2024-02-05T04:53:40.124 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9952000000000001e-05 2024-02-05T04:53:40.124 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.125 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.125 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:40.125 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.172566+0000", 2024-02-05T04:53:40.125 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6005999999999999e-05 2024-02-05T04:53:40.125 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.125 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.125 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:40.125 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.172579+0000", 2024-02-05T04:53:40.125 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3234000000000001e-05 2024-02-05T04:53:40.125 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.125 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.125 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:40.125 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.172649+0000", 2024-02-05T04:53:40.125 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0038999999999998e-05 2024-02-05T04:53:40.125 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.125 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.126 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:40.126 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.172659+0000", 2024-02-05T04:53:40.126 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0295999999999999e-05 2024-02-05T04:53:40.126 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.126 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.126 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:40.126 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.177376+0000", 2024-02-05T04:53:40.126 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0047169619999999999 2024-02-05T04:53:40.126 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.126 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.126 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:40.127 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.177454+0000", 2024-02-05T04:53:40.127 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8348000000000003e-05 2024-02-05T04:53:40.127 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:40.127 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:40.127 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:40.127 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.127 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.127 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.3s3 43/31 ECSubWrite(tid=63, reqid=osd.4.0:63, at_version=43'27, trim_to=0'0, roll_forward_to=40'25))", 2024-02-05T04:53:40.127 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:35.173378+0000", 2024-02-05T04:53:40.128 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 4.8759582009999995, 2024-02-05T04:53:40.128 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0066975339999999998, 2024-02-05T04:53:40.128 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:40.128 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:40.128 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:40.128 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.128 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:40.128 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.173378+0000", 2024-02-05T04:53:40.128 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:40.128 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.128 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.128 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:40.128 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.173378+0000", 2024-02-05T04:53:40.128 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:40.129 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.129 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.129 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:40.129 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.173378+0000", 2024-02-05T04:53:40.129 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9599999999999999e-07 2024-02-05T04:53:40.129 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.129 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.129 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:40.129 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.173394+0000", 2024-02-05T04:53:40.129 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6269e-05 2024-02-05T04:53:40.129 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.129 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.129 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:40.129 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.173405+0000", 2024-02-05T04:53:40.129 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0372e-05 2024-02-05T04:53:40.130 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.130 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.130 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:40.130 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.173411+0000", 2024-02-05T04:53:40.130 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0549999999999996e-06 2024-02-05T04:53:40.130 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.130 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.130 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:40.130 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.173465+0000", 2024-02-05T04:53:40.130 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3665000000000002e-05 2024-02-05T04:53:40.130 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.130 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.130 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:40.130 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.173471+0000", 2024-02-05T04:53:40.130 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1079999999999998e-06 2024-02-05T04:53:40.130 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.131 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.131 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:40.131 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.180035+0000", 2024-02-05T04:53:40.131 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0065645019999999998 2024-02-05T04:53:40.131 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.131 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.131 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:40.131 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.180075+0000", 2024-02-05T04:53:40.131 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9966999999999999e-05 2024-02-05T04:53:40.131 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:40.131 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:40.131 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:40.131 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.131 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.131 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.3s3 43/31 ECSubWrite(tid=65, reqid=osd.4.0:63, at_version=0'0, trim_to=0'0, roll_forward_to=43'27))", 2024-02-05T04:53:40.132 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:35.182154+0000", 2024-02-05T04:53:40.132 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 4.8671816530000003, 2024-02-05T04:53:40.132 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0035578089999999999, 2024-02-05T04:53:40.132 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:40.132 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:40.132 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:40.132 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.132 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:40.132 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.182154+0000", 2024-02-05T04:53:40.132 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:40.132 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.132 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.132 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:40.132 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.182154+0000", 2024-02-05T04:53:40.132 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:40.133 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.133 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.133 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:40.133 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.182155+0000", 2024-02-05T04:53:40.133 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.1100000000000005e-07 2024-02-05T04:53:40.133 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.133 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.133 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:40.133 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.182167+0000", 2024-02-05T04:53:40.133 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1749000000000001e-05 2024-02-05T04:53:40.134 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.134 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.134 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:40.134 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.182182+0000", 2024-02-05T04:53:40.134 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5332000000000002e-05 2024-02-05T04:53:40.134 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.134 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.134 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:40.134 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.182191+0000", 2024-02-05T04:53:40.134 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.0589999999999992e-06 2024-02-05T04:53:40.134 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.134 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.134 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:40.134 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.182255+0000", 2024-02-05T04:53:40.135 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3621000000000006e-05 2024-02-05T04:53:40.135 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.135 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.135 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:40.135 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.182265+0000", 2024-02-05T04:53:40.135 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.012e-05 2024-02-05T04:53:40.135 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.135 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.135 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:40.135 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.185679+0000", 2024-02-05T04:53:40.135 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.003413953 2024-02-05T04:53:40.135 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.135 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.135 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:40.135 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.185712+0000", 2024-02-05T04:53:40.135 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3164000000000003e-05 2024-02-05T04:53:40.136 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:40.136 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:40.136 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:40.136 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.136 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.136 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.6s3 43/31 ECSubWrite(tid=141, reqid=osd.13.0:138, at_version=0'0, trim_to=0'0, roll_forward_to=43'24))", 2024-02-05T04:53:40.136 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:35.183155+0000", 2024-02-05T04:53:40.136 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 4.8661807000000001, 2024-02-05T04:53:40.136 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0055237740000000004, 2024-02-05T04:53:40.136 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:40.136 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:40.136 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:40.136 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.136 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:40.137 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.183155+0000", 2024-02-05T04:53:40.137 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:40.137 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.137 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.137 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:40.137 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.183155+0000", 2024-02-05T04:53:40.137 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:40.137 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.137 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.137 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:40.137 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.183156+0000", 2024-02-05T04:53:40.137 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.7000000000000003e-07 2024-02-05T04:53:40.137 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.137 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.137 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:40.138 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.183168+0000", 2024-02-05T04:53:40.138 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1644999999999999e-05 2024-02-05T04:53:40.138 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.138 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.138 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:40.138 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.183184+0000", 2024-02-05T04:53:40.138 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.63e-05 2024-02-05T04:53:40.138 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.138 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.138 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:40.138 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.183195+0000", 2024-02-05T04:53:40.138 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1456e-05 2024-02-05T04:53:40.138 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.138 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.138 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:40.139 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.183257+0000", 2024-02-05T04:53:40.139 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1734000000000002e-05 2024-02-05T04:53:40.139 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.139 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.139 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:40.139 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.183267+0000", 2024-02-05T04:53:40.139 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8379999999999997e-06 2024-02-05T04:53:40.139 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.139 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.139 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:40.139 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.188639+0000", 2024-02-05T04:53:40.139 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0053717189999999996 2024-02-05T04:53:40.139 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:40.139 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:40.139 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:40.139 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.188679+0000", 2024-02-05T04:53:40.140 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0212000000000001e-05 2024-02-05T04:53:40.140 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:40.140 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:40.140 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:40.140 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:40.140 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:40.140 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T04:53:40.142 INFO:tasks.ceph.osd.15.smithi196.stderr:2024-02-05T04:53:40.140+0000 7f1facc45640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 34611) UID: 0 2024-02-05T04:53:40.142 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-02-05T04:53:40.242 INFO:tasks.ceph.osd.4.smithi062.stderr:2024-02-05T04:53:40.242+0000 7f708c308640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34689) UID: 0 2024-02-05T04:53:40.255 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-02-05T04:53:40.256 INFO:teuthology.orchestra.run.smithi062.stdout: "ops": [], 2024-02-05T04:53:40.256 INFO:teuthology.orchestra.run.smithi062.stdout: "num_ops": 0 2024-02-05T04:53:40.256 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-02-05T04:53:40.265 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-02-05T04:53:40.351 INFO:tasks.ceph.osd.4.smithi062.stderr:2024-02-05T04:53:40.348+0000 7f708c308640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34689) UID: 0 2024-02-05T04:53:40.413 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-02-05T04:53:40.413 INFO:teuthology.orchestra.run.smithi062.stdout: "ops": [], 2024-02-05T04:53:40.413 INFO:teuthology.orchestra.run.smithi062.stdout: "complaint_time": 30, 2024-02-05T04:53:40.413 INFO:teuthology.orchestra.run.smithi062.stdout: "num_blocked_ops": 0 2024-02-05T04:53:40.413 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-02-05T04:53:40.421 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-02-05T04:53:40.445 INFO:tasks.ceph.osd.6.smithi150.stderr:2024-02-05T04:53:40.444+0000 7f6bea9f0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34015) UID: 0 2024-02-05T04:53:40.545 INFO:tasks.ceph.osd.3.smithi196.stderr:2024-02-05T04:53:40.545+0000 7fc8817c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34605) UID: 0 2024-02-05T04:53:40.572 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-02-05T04:53:40.572 INFO:teuthology.orchestra.run.smithi062.stdout: "size": 20, 2024-02-05T04:53:40.572 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 600, 2024-02-05T04:53:40.572 INFO:teuthology.orchestra.run.smithi062.stdout: "ops": [ 2024-02-05T04:53:40.572 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.572 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.8s3 37/31 ECSubWrite(tid=25, reqid=osd.4.0:25, at_version=37'10, trim_to=0'0, roll_forward_to=35'8))", 2024-02-05T04:53:40.572 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:29.129097+0000", 2024-02-05T04:53:40.572 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 11.441378457999999, 2024-02-05T04:53:40.572 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0016204780000000001, 2024-02-05T04:53:40.573 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:40.573 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:40.573 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:40.573 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.573 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:40.573 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.129097+0000", 2024-02-05T04:53:40.573 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:40.573 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.573 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.573 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:40.573 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.129097+0000", 2024-02-05T04:53:40.573 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:40.573 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.574 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.574 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:40.574 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.129097+0000", 2024-02-05T04:53:40.574 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.0100000000000005e-07 2024-02-05T04:53:40.574 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.574 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.574 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:40.574 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.129147+0000", 2024-02-05T04:53:40.574 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.9901000000000002e-05 2024-02-05T04:53:40.574 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.574 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.574 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:40.574 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.129178+0000", 2024-02-05T04:53:40.574 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.0639000000000003e-05 2024-02-05T04:53:40.574 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.574 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.575 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:40.575 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.129193+0000", 2024-02-05T04:53:40.575 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.5194999999999999e-05 2024-02-05T04:53:40.575 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.575 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.575 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:40.575 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.129285+0000", 2024-02-05T04:53:40.575 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.1779000000000003e-05 2024-02-05T04:53:40.575 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.575 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.575 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:40.575 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.129299+0000", 2024-02-05T04:53:40.575 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.4622e-05 2024-02-05T04:53:40.575 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.575 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.576 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:40.576 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.130678+0000", 2024-02-05T04:53:40.576 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0013788310000000001 2024-02-05T04:53:40.576 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.576 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.576 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:40.576 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.130717+0000", 2024-02-05T04:53:40.576 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.8909999999999998e-05 2024-02-05T04:53:40.576 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:40.576 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:40.576 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:40.576 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.576 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.576 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.8s3 37/31 ECSubWrite(tid=28, reqid=osd.4.0:25, at_version=0'0, trim_to=0'0, roll_forward_to=37'10))", 2024-02-05T04:53:40.577 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:29.131644+0000", 2024-02-05T04:53:40.577 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 11.438830665999999, 2024-02-05T04:53:40.577 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00092228799999999995, 2024-02-05T04:53:40.577 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:40.577 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:40.577 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:40.577 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.577 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:40.577 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.131644+0000", 2024-02-05T04:53:40.577 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:40.577 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.577 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.577 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:40.577 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.131644+0000", 2024-02-05T04:53:40.578 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:40.578 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.578 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.578 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:40.578 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.131645+0000", 2024-02-05T04:53:40.578 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.029e-06 2024-02-05T04:53:40.578 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.578 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.578 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:40.578 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.131670+0000", 2024-02-05T04:53:40.578 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.4794e-05 2024-02-05T04:53:40.578 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.578 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.578 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:40.578 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.131686+0000", 2024-02-05T04:53:40.579 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.5472999999999999e-05 2024-02-05T04:53:40.579 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.579 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.579 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:40.579 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.131696+0000", 2024-02-05T04:53:40.579 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.9119999999999993e-06 2024-02-05T04:53:40.579 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.579 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.579 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:40.579 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.131758+0000", 2024-02-05T04:53:40.579 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.2593000000000002e-05 2024-02-05T04:53:40.579 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.579 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.579 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:40.579 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.131768+0000", 2024-02-05T04:53:40.580 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.5959999999999993e-06 2024-02-05T04:53:40.580 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.580 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.580 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:40.580 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.132531+0000", 2024-02-05T04:53:40.580 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00076315200000000001 2024-02-05T04:53:40.580 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.580 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.580 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:40.580 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.132567+0000", 2024-02-05T04:53:40.580 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.5738999999999998e-05 2024-02-05T04:53:40.580 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:40.580 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:40.580 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:40.580 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.581 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.581 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpRead(3.bs3 37/31 ECSubRead(tid=76, to_read={3:db93bf34:::smithi19639362-24:head=0,98304,0}, subchunks={3:db93bf34:::smithi19639362-24:head=[0,1]}, attrs_to_read=))", 2024-02-05T04:53:40.581 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:29.150071+0000", 2024-02-05T04:53:40.581 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 11.420404241, 2024-02-05T04:53:40.581 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00064157799999999996, 2024-02-05T04:53:40.581 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:40.581 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:40.581 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:40.581 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.581 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:40.581 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.150071+0000", 2024-02-05T04:53:40.581 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:40.581 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.582 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.582 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:40.582 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.150071+0000", 2024-02-05T04:53:40.582 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:40.582 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.582 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.582 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:40.582 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.150071+0000", 2024-02-05T04:53:40.582 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.3599999999999999e-07 2024-02-05T04:53:40.582 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.582 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.582 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:40.582 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.150081+0000", 2024-02-05T04:53:40.582 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0317e-05 2024-02-05T04:53:40.582 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.583 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.583 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:40.583 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.150110+0000", 2024-02-05T04:53:40.583 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.8129000000000001e-05 2024-02-05T04:53:40.583 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.583 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.583 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:40.583 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.150122+0000", 2024-02-05T04:53:40.583 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.2308e-05 2024-02-05T04:53:40.583 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.583 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.583 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:40.583 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.150213+0000", 2024-02-05T04:53:40.583 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.0698999999999993e-05 2024-02-05T04:53:40.583 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.584 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.584 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:40.584 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.150712+0000", 2024-02-05T04:53:40.584 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00049978900000000005 2024-02-05T04:53:40.584 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:40.584 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:40.584 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:40.584 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.584 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.584 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpRead(3.bs3 37/31 ECSubRead(tid=77, to_read={3:db93bf34:::smithi19639362-24:head=0,98304,0}, subchunks={3:db93bf34:::smithi19639362-24:head=[0,1]}, attrs_to_read=))", 2024-02-05T04:53:40.584 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:29.150463+0000", 2024-02-05T04:53:40.584 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 11.420011619, 2024-02-05T04:53:40.584 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000453003, 2024-02-05T04:53:40.584 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:40.584 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:40.585 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:40.585 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.585 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:40.585 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.150463+0000", 2024-02-05T04:53:40.585 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:40.585 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.585 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.585 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:40.585 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.150463+0000", 2024-02-05T04:53:40.585 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:40.585 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.585 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.586 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:40.586 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.150464+0000", 2024-02-05T04:53:40.586 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.7300000000000004e-07 2024-02-05T04:53:40.586 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.586 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.586 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:40.586 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.150474+0000", 2024-02-05T04:53:40.586 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.6679999999999996e-06 2024-02-05T04:53:40.586 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.586 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.586 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:40.586 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.150491+0000", 2024-02-05T04:53:40.586 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.7288000000000001e-05 2024-02-05T04:53:40.586 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.586 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.587 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:40.587 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.150505+0000", 2024-02-05T04:53:40.587 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.3531e-05 2024-02-05T04:53:40.587 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.587 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.587 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:40.587 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.150773+0000", 2024-02-05T04:53:40.587 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000267796 2024-02-05T04:53:40.587 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.587 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.587 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:40.587 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.150916+0000", 2024-02-05T04:53:40.587 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000143747 2024-02-05T04:53:40.587 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:40.588 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:40.588 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:40.588 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.588 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.588 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpRead(3.bs3 37/31 ECSubRead(tid=78, to_read={3:db93bf34:::smithi19639362-24:head=0,98304,0}, subchunks={3:db93bf34:::smithi19639362-24:head=[0,1]}, attrs_to_read=))", 2024-02-05T04:53:40.588 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:29.150556+0000", 2024-02-05T04:53:40.588 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 11.419919061, 2024-02-05T04:53:40.588 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00053283099999999999, 2024-02-05T04:53:40.588 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:40.588 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:40.588 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:40.588 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.588 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:40.588 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.150556+0000", 2024-02-05T04:53:40.588 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:40.589 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.589 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.589 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:40.589 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.150556+0000", 2024-02-05T04:53:40.589 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:40.589 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.589 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.589 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:40.589 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.150557+0000", 2024-02-05T04:53:40.589 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.7999999999999995e-07 2024-02-05T04:53:40.589 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.589 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.589 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:40.589 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.150563+0000", 2024-02-05T04:53:40.589 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.5590000000000002e-06 2024-02-05T04:53:40.589 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.590 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.590 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:40.590 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.150588+0000", 2024-02-05T04:53:40.590 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.4513000000000001e-05 2024-02-05T04:53:40.590 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.590 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.590 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:40.590 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.150598+0000", 2024-02-05T04:53:40.590 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0587e-05 2024-02-05T04:53:40.590 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.590 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.590 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:40.590 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.150956+0000", 2024-02-05T04:53:40.590 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00035813799999999998 2024-02-05T04:53:40.591 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.591 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.591 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:40.591 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.151089+0000", 2024-02-05T04:53:40.591 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000132454 2024-02-05T04:53:40.591 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:40.591 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:40.591 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:40.591 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.591 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.591 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpRead(3.bs3 37/31 ECSubRead(tid=80, to_read={3:de65491d:::smithi19639362-2:head=0,8192,0}, subchunks={3:de65491d:::smithi19639362-2:head=[0,1]}, attrs_to_read=))", 2024-02-05T04:53:40.591 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:29.152960+0000", 2024-02-05T04:53:40.591 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 11.417514699, 2024-02-05T04:53:40.591 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00041001500000000001, 2024-02-05T04:53:40.591 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:40.591 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:40.592 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:40.592 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.592 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:40.592 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.152960+0000", 2024-02-05T04:53:40.592 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:40.592 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.592 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.592 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:40.592 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.152960+0000", 2024-02-05T04:53:40.592 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:40.592 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.592 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.592 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:40.592 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.152961+0000", 2024-02-05T04:53:40.592 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.9899999999999999e-07 2024-02-05T04:53:40.593 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.593 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.593 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:40.593 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.152974+0000", 2024-02-05T04:53:40.593 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.2663999999999999e-05 2024-02-05T04:53:40.593 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.593 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.593 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:40.593 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.152990+0000", 2024-02-05T04:53:40.593 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.5631e-05 2024-02-05T04:53:40.593 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.593 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.593 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:40.593 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.153001+0000", 2024-02-05T04:53:40.593 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1001e-05 2024-02-05T04:53:40.594 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.594 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.594 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:40.594 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.153076+0000", 2024-02-05T04:53:40.594 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.5737999999999999e-05 2024-02-05T04:53:40.594 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.594 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.594 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:40.594 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.153370+0000", 2024-02-05T04:53:40.594 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00029408199999999998 2024-02-05T04:53:40.594 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:40.594 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:40.594 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:40.594 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.594 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.595 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.fs4 38/31 ECSubWrite(tid=51, reqid=client.4634.0:290, at_version=38'17, trim_to=0'0, roll_forward_to=34'16))", 2024-02-05T04:53:40.595 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:30.387948+0000", 2024-02-05T04:53:40.595 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 10.182527392000001, 2024-02-05T04:53:40.595 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0017840239999999999, 2024-02-05T04:53:40.595 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:40.595 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:40.595 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:40.595 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.595 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:40.595 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.387948+0000", 2024-02-05T04:53:40.595 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:40.595 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.595 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.595 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:40.595 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.387948+0000", 2024-02-05T04:53:40.596 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:40.596 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.596 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.596 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:40.596 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.387949+0000", 2024-02-05T04:53:40.596 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.159e-06 2024-02-05T04:53:40.596 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.596 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.596 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:40.596 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.388411+0000", 2024-02-05T04:53:40.596 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00046226999999999998 2024-02-05T04:53:40.596 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.597 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.597 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:40.597 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.388431+0000", 2024-02-05T04:53:40.597 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.0390999999999998e-05 2024-02-05T04:53:40.597 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.597 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.597 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:40.597 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.388442+0000", 2024-02-05T04:53:40.597 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0902e-05 2024-02-05T04:53:40.597 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.597 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.597 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:40.597 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.388517+0000", 2024-02-05T04:53:40.597 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.4648999999999996e-05 2024-02-05T04:53:40.597 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.598 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.598 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:40.598 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.388528+0000", 2024-02-05T04:53:40.598 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.131e-05 2024-02-05T04:53:40.598 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.598 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.598 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:40.598 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.389707+0000", 2024-02-05T04:53:40.598 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0011782629999999999 2024-02-05T04:53:40.598 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.598 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.598 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:40.598 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.389732+0000", 2024-02-05T04:53:40.598 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.508e-05 2024-02-05T04:53:40.598 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:40.599 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:40.599 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:40.599 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.599 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.599 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.fs4 38/31 ECSubWrite(tid=52, reqid=client.4634.0:291, at_version=38'18, trim_to=0'0, roll_forward_to=34'16))", 2024-02-05T04:53:40.599 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:30.389019+0000", 2024-02-05T04:53:40.599 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 10.181456353, 2024-02-05T04:53:40.599 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.001372809, 2024-02-05T04:53:40.599 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:40.599 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:40.599 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:40.599 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.599 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:40.599 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.389019+0000", 2024-02-05T04:53:40.600 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:40.600 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.600 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.600 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:40.600 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.389019+0000", 2024-02-05T04:53:40.600 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:40.600 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.600 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.600 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:40.600 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.389019+0000", 2024-02-05T04:53:40.600 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.2300000000000001e-07 2024-02-05T04:53:40.600 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.600 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.600 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:40.601 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.389319+0000", 2024-02-05T04:53:40.601 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00029977500000000001 2024-02-05T04:53:40.601 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.601 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.601 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:40.601 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.389331+0000", 2024-02-05T04:53:40.601 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.2146e-05 2024-02-05T04:53:40.601 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.601 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.601 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:40.601 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.389339+0000", 2024-02-05T04:53:40.601 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.5129999999999999e-06 2024-02-05T04:53:40.601 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.601 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.602 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:40.602 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.389391+0000", 2024-02-05T04:53:40.602 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.2703e-05 2024-02-05T04:53:40.602 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.602 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.602 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:40.602 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.389401+0000", 2024-02-05T04:53:40.602 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.4050000000000006e-06 2024-02-05T04:53:40.602 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.602 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.602 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:40.602 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.390362+0000", 2024-02-05T04:53:40.602 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00096129699999999995 2024-02-05T04:53:40.602 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.602 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.603 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:40.603 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.390391+0000", 2024-02-05T04:53:40.603 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.9346999999999999e-05 2024-02-05T04:53:40.603 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:40.603 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:40.603 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:40.603 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.603 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.603 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.fs4 38/31 ECSubWrite(tid=53, reqid=client.4634.0:292, at_version=38'19, trim_to=0'0, roll_forward_to=34'16))", 2024-02-05T04:53:40.603 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:30.389890+0000", 2024-02-05T04:53:40.603 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 10.180585376, 2024-02-05T04:53:40.603 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.001022018, 2024-02-05T04:53:40.603 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:40.603 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:40.603 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:40.604 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.604 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:40.604 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.389890+0000", 2024-02-05T04:53:40.604 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:40.604 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.604 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.604 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:40.604 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.389890+0000", 2024-02-05T04:53:40.604 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:40.604 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.604 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.604 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:40.604 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.389890+0000", 2024-02-05T04:53:40.604 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.7100000000000002e-07 2024-02-05T04:53:40.604 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.604 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.605 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:40.605 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.389997+0000", 2024-02-05T04:53:40.605 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000106426 2024-02-05T04:53:40.605 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.605 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.605 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:40.605 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.390011+0000", 2024-02-05T04:53:40.605 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.4124000000000001e-05 2024-02-05T04:53:40.605 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.605 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.605 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:40.605 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.390021+0000", 2024-02-05T04:53:40.605 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0149999999999999e-05 2024-02-05T04:53:40.606 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.606 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.606 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:40.606 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.390069+0000", 2024-02-05T04:53:40.606 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.7901000000000001e-05 2024-02-05T04:53:40.606 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.606 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.606 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:40.606 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.390074+0000", 2024-02-05T04:53:40.606 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.186e-06 2024-02-05T04:53:40.606 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.606 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.606 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:40.606 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.390883+0000", 2024-02-05T04:53:40.606 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00080893899999999995 2024-02-05T04:53:40.606 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.607 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.607 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:40.607 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.390912+0000", 2024-02-05T04:53:40.607 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.8721000000000002e-05 2024-02-05T04:53:40.607 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:40.607 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:40.607 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:40.607 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.607 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.607 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.fs4 38/31 ECSubWrite(tid=54, reqid=client.4634.0:293, at_version=38'20, trim_to=0'0, roll_forward_to=34'16))", 2024-02-05T04:53:40.607 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:30.390622+0000", 2024-02-05T04:53:40.607 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 10.179852899, 2024-02-05T04:53:40.607 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0011679749999999999, 2024-02-05T04:53:40.607 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:40.608 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:40.608 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:40.608 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.608 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:40.608 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.390622+0000", 2024-02-05T04:53:40.608 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:40.608 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.608 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.608 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:40.608 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.390622+0000", 2024-02-05T04:53:40.608 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:40.608 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.608 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.608 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:40.608 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.390623+0000", 2024-02-05T04:53:40.609 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.1600000000000001e-07 2024-02-05T04:53:40.609 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.609 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.609 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:40.609 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.390634+0000", 2024-02-05T04:53:40.609 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1669e-05 2024-02-05T04:53:40.609 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.609 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.609 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:40.609 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.390644+0000", 2024-02-05T04:53:40.609 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.5480000000000007e-06 2024-02-05T04:53:40.609 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.609 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.609 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:40.609 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.390649+0000", 2024-02-05T04:53:40.609 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.5929999999999996e-06 2024-02-05T04:53:40.610 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.610 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.610 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:40.610 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.390694+0000", 2024-02-05T04:53:40.610 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.4348999999999998e-05 2024-02-05T04:53:40.610 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.610 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.610 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:40.610 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.390704+0000", 2024-02-05T04:53:40.610 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0111e-05 2024-02-05T04:53:40.610 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.610 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.610 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:40.610 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.391761+0000", 2024-02-05T04:53:40.610 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0010575409999999999 2024-02-05T04:53:40.611 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.611 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.611 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:40.611 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.391790+0000", 2024-02-05T04:53:40.611 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.8648000000000001e-05 2024-02-05T04:53:40.611 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:40.611 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:40.611 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:40.611 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.611 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.611 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.fs4 38/31 ECSubWrite(tid=55, reqid=client.4634.0:293, at_version=0'0, trim_to=0'0, roll_forward_to=38'20))", 2024-02-05T04:53:40.611 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:30.395478+0000", 2024-02-05T04:53:40.611 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 10.174997089, 2024-02-05T04:53:40.611 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00056105200000000004, 2024-02-05T04:53:40.611 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:40.611 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:40.612 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:40.612 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.612 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:40.612 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.395478+0000", 2024-02-05T04:53:40.612 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:40.612 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.612 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.612 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:40.612 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.395478+0000", 2024-02-05T04:53:40.612 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:40.612 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.612 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.612 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:40.612 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.395479+0000", 2024-02-05T04:53:40.613 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.7700000000000004e-07 2024-02-05T04:53:40.613 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.613 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.613 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:40.613 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.395487+0000", 2024-02-05T04:53:40.613 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.1240000000000005e-06 2024-02-05T04:53:40.613 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.613 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.613 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:40.613 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.395497+0000", 2024-02-05T04:53:40.613 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0736e-05 2024-02-05T04:53:40.613 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.613 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.613 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:40.613 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.395506+0000", 2024-02-05T04:53:40.614 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.3469999999999999e-06 2024-02-05T04:53:40.614 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.614 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.614 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:40.614 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.395548+0000", 2024-02-05T04:53:40.614 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.2030000000000002e-05 2024-02-05T04:53:40.614 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.614 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.614 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:40.614 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.395553+0000", 2024-02-05T04:53:40.614 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.2100000000000001e-06 2024-02-05T04:53:40.614 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.614 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.614 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:40.614 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.396015+0000", 2024-02-05T04:53:40.614 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00046247399999999999 2024-02-05T04:53:40.615 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.615 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.615 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:40.615 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.396039+0000", 2024-02-05T04:53:40.615 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.3553999999999998e-05 2024-02-05T04:53:40.615 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:40.615 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:40.615 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:40.615 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.615 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.615 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.8s3 42/31 ECSubWrite(tid=60, reqid=client.4634.0:390, at_version=42'17, trim_to=0'0, roll_forward_to=40'15))", 2024-02-05T04:53:40.615 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:34.180486+0000", 2024-02-05T04:53:40.615 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 6.3899892390000002, 2024-02-05T04:53:40.615 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0019622329999999999, 2024-02-05T04:53:40.616 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:40.616 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:40.616 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:40.616 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.616 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:40.616 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.180486+0000", 2024-02-05T04:53:40.616 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:40.616 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.616 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.616 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:40.616 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.180486+0000", 2024-02-05T04:53:40.616 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:40.616 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.616 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.617 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:40.617 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.180486+0000", 2024-02-05T04:53:40.617 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.46e-07 2024-02-05T04:53:40.617 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.617 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.617 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:40.617 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.180737+0000", 2024-02-05T04:53:40.617 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00025076299999999998 2024-02-05T04:53:40.617 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.617 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.617 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:40.617 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.180755+0000", 2024-02-05T04:53:40.617 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.789e-05 2024-02-05T04:53:40.617 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.617 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.618 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:40.618 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.180766+0000", 2024-02-05T04:53:40.618 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1547e-05 2024-02-05T04:53:40.618 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.618 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.618 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:40.618 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.180839+0000", 2024-02-05T04:53:40.618 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.2297000000000003e-05 2024-02-05T04:53:40.618 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.618 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.618 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:40.618 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.180851+0000", 2024-02-05T04:53:40.618 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1888e-05 2024-02-05T04:53:40.618 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.618 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.619 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:40.619 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.182415+0000", 2024-02-05T04:53:40.619 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.001564385 2024-02-05T04:53:40.619 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.619 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.619 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:40.619 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.182448+0000", 2024-02-05T04:53:40.619 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.3016999999999999e-05 2024-02-05T04:53:40.619 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:40.619 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:40.619 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:40.619 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.619 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.619 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.8s3 42/31 ECSubWrite(tid=61, reqid=client.4634.0:390, at_version=0'0, trim_to=0'0, roll_forward_to=42'17))", 2024-02-05T04:53:40.620 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:34.186149+0000", 2024-02-05T04:53:40.620 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 6.3843264380000004, 2024-02-05T04:53:40.620 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00080371399999999997, 2024-02-05T04:53:40.620 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:40.620 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:40.620 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:40.620 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.620 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:40.620 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.186149+0000", 2024-02-05T04:53:40.620 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:40.620 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.621 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.621 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:40.621 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.186149+0000", 2024-02-05T04:53:40.621 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:40.621 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.621 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.621 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:40.621 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.186150+0000", 2024-02-05T04:53:40.621 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0499999999999999e-06 2024-02-05T04:53:40.621 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.621 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.621 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:40.621 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.186164+0000", 2024-02-05T04:53:40.621 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.4263999999999999e-05 2024-02-05T04:53:40.622 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.622 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.622 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:40.622 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.186182+0000", 2024-02-05T04:53:40.622 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.7886000000000001e-05 2024-02-05T04:53:40.622 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.622 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.622 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:40.622 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.186192+0000", 2024-02-05T04:53:40.622 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0494e-05 2024-02-05T04:53:40.622 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.622 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.622 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:40.622 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.186277+0000", 2024-02-05T04:53:40.623 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.4598999999999994e-05 2024-02-05T04:53:40.623 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.623 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.623 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:40.623 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.186290+0000", 2024-02-05T04:53:40.623 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.2938999999999999e-05 2024-02-05T04:53:40.623 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.623 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.623 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:40.623 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.186925+0000", 2024-02-05T04:53:40.623 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00063520000000000004 2024-02-05T04:53:40.623 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.623 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.623 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:40.624 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.186952+0000", 2024-02-05T04:53:40.624 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.7282000000000001e-05 2024-02-05T04:53:40.624 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:40.624 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:40.624 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:40.624 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.624 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.624 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.bs3 43/31 ECSubWrite(tid=137, reqid=osd.13.0:137, at_version=43'28, trim_to=0'0, roll_forward_to=41'26))", 2024-02-05T04:53:40.624 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:35.171779+0000", 2024-02-05T04:53:40.624 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 5.3986959309999998, 2024-02-05T04:53:40.624 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0021022329999999998, 2024-02-05T04:53:40.624 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:40.624 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:40.624 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:40.625 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.625 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:40.625 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.171779+0000", 2024-02-05T04:53:40.625 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:40.625 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.625 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.625 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:40.625 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.171779+0000", 2024-02-05T04:53:40.625 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:40.625 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.625 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.625 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:40.625 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.171779+0000", 2024-02-05T04:53:40.625 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.8299999999999998e-07 2024-02-05T04:53:40.625 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.626 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.626 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:40.626 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.171803+0000", 2024-02-05T04:53:40.626 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.3748e-05 2024-02-05T04:53:40.626 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.626 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.626 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:40.626 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.171827+0000", 2024-02-05T04:53:40.626 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.3434999999999998e-05 2024-02-05T04:53:40.626 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.626 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.626 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:40.626 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.171839+0000", 2024-02-05T04:53:40.626 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.2774e-05 2024-02-05T04:53:40.626 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.627 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.627 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:40.627 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.171916+0000", 2024-02-05T04:53:40.627 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.7031000000000002e-05 2024-02-05T04:53:40.627 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.627 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.627 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:40.627 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.171925+0000", 2024-02-05T04:53:40.627 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.0420000000000008e-06 2024-02-05T04:53:40.627 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.627 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.627 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:40.627 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.173840+0000", 2024-02-05T04:53:40.627 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0019142059999999999 2024-02-05T04:53:40.627 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.628 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.628 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:40.628 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.173881+0000", 2024-02-05T04:53:40.628 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.1714000000000001e-05 2024-02-05T04:53:40.628 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:40.628 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:40.628 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:40.628 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.628 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.628 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.8s3 43/31 ECSubWrite(tid=62, reqid=osd.4.0:62, at_version=43'19, trim_to=0'0, roll_forward_to=42'17))", 2024-02-05T04:53:40.628 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:35.172477+0000", 2024-02-05T04:53:40.628 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 5.3979984050000001, 2024-02-05T04:53:40.628 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0022630319999999999, 2024-02-05T04:53:40.628 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:40.628 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:40.629 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:40.629 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.629 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:40.629 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.172477+0000", 2024-02-05T04:53:40.629 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:40.629 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.629 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.629 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:40.629 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.172477+0000", 2024-02-05T04:53:40.629 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:40.629 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.629 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.629 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:40.629 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.172477+0000", 2024-02-05T04:53:40.629 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.0600000000000001e-07 2024-02-05T04:53:40.630 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.630 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.630 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:40.630 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.172499+0000", 2024-02-05T04:53:40.630 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.1738000000000001e-05 2024-02-05T04:53:40.630 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.630 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.630 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:40.630 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.172508+0000", 2024-02-05T04:53:40.630 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.3239999999999992e-06 2024-02-05T04:53:40.630 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.630 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.630 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:40.630 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.172516+0000", 2024-02-05T04:53:40.630 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.7870000000000002e-06 2024-02-05T04:53:40.631 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.631 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.631 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:40.631 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.172581+0000", 2024-02-05T04:53:40.631 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.4904000000000001e-05 2024-02-05T04:53:40.631 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.631 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.631 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:40.631 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.172589+0000", 2024-02-05T04:53:40.631 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.7229999999999993e-06 2024-02-05T04:53:40.631 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.631 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.631 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:40.631 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.174705+0000", 2024-02-05T04:53:40.631 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.002115722 2024-02-05T04:53:40.632 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.632 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.632 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:40.632 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.174740+0000", 2024-02-05T04:53:40.632 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.4527999999999999e-05 2024-02-05T04:53:40.632 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:40.632 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:40.632 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:40.632 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.632 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.632 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.bs3 43/31 ECSubWrite(tid=139, reqid=osd.13.0:137, at_version=0'0, trim_to=0'0, roll_forward_to=43'28))", 2024-02-05T04:53:40.632 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:35.180865+0000", 2024-02-05T04:53:40.632 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 5.38961047, 2024-02-05T04:53:40.632 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0037505139999999999, 2024-02-05T04:53:40.632 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:40.633 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:40.633 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:40.633 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.633 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:40.633 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.180865+0000", 2024-02-05T04:53:40.633 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:40.633 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.633 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.633 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:40.633 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.180865+0000", 2024-02-05T04:53:40.633 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:40.633 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.633 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.633 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:40.633 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.180865+0000", 2024-02-05T04:53:40.634 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.0200000000000001e-07 2024-02-05T04:53:40.634 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.634 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.634 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:40.634 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.180875+0000", 2024-02-05T04:53:40.634 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.8959999999999998e-06 2024-02-05T04:53:40.634 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.634 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.634 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:40.634 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.180892+0000", 2024-02-05T04:53:40.634 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.6531000000000001e-05 2024-02-05T04:53:40.634 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.634 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.634 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:40.634 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.180901+0000", 2024-02-05T04:53:40.635 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.2890000000000003e-06 2024-02-05T04:53:40.635 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.635 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.635 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:40.635 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.180961+0000", 2024-02-05T04:53:40.635 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.0012000000000001e-05 2024-02-05T04:53:40.635 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.635 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.635 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:40.635 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.180971+0000", 2024-02-05T04:53:40.635 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0190999999999999e-05 2024-02-05T04:53:40.635 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.635 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.635 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:40.635 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.184573+0000", 2024-02-05T04:53:40.636 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00360181 2024-02-05T04:53:40.636 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.636 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.636 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:40.636 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.184615+0000", 2024-02-05T04:53:40.636 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.1983000000000001e-05 2024-02-05T04:53:40.636 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:40.636 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:40.636 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:40.636 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.636 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.636 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.8s3 43/31 ECSubWrite(tid=64, reqid=osd.4.0:62, at_version=0'0, trim_to=0'0, roll_forward_to=43'19))", 2024-02-05T04:53:40.636 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:35.181173+0000", 2024-02-05T04:53:40.636 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 5.3893021509999999, 2024-02-05T04:53:40.637 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0033436339999999998, 2024-02-05T04:53:40.637 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:40.637 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:40.637 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:40.637 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.637 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:40.637 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.181173+0000", 2024-02-05T04:53:40.637 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:40.637 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.637 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.637 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:40.637 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.181173+0000", 2024-02-05T04:53:40.637 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:40.637 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.637 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.638 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:40.638 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.181174+0000", 2024-02-05T04:53:40.638 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.85e-07 2024-02-05T04:53:40.638 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.638 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.638 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:40.638 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.181183+0000", 2024-02-05T04:53:40.638 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.8019999999999998e-06 2024-02-05T04:53:40.638 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.638 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.638 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:40.638 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.181195+0000", 2024-02-05T04:53:40.638 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.223e-05 2024-02-05T04:53:40.638 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.638 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.639 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:40.639 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.181205+0000", 2024-02-05T04:53:40.639 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.7569999999999999e-06 2024-02-05T04:53:40.639 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.639 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.639 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:40.639 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.181269+0000", 2024-02-05T04:53:40.639 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.4759999999999997e-05 2024-02-05T04:53:40.639 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.639 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.639 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:40.639 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.181278+0000", 2024-02-05T04:53:40.639 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.9500000000000007e-06 2024-02-05T04:53:40.639 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.639 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.640 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:40.640 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.184483+0000", 2024-02-05T04:53:40.640 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0032044019999999999 2024-02-05T04:53:40.640 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.640 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.640 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:40.640 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.184517+0000", 2024-02-05T04:53:40.640 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.3847999999999998e-05 2024-02-05T04:53:40.640 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:40.640 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:40.640 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:40.640 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.640 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.640 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpRead(3.bs3 47/31 ECSubRead(tid=159, to_read={3:df6528f1:::smithi19639362-33:head=0,135168,0}, subchunks={3:df6528f1:::smithi19639362-33:head=[0,1]}, attrs_to_read=))", 2024-02-05T04:53:40.640 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:38.947219+0000", 2024-02-05T04:53:40.641 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 1.6232564819999999, 2024-02-05T04:53:40.641 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00025446699999999997, 2024-02-05T04:53:40.641 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:40.641 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:40.641 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:40.641 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.641 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:40.641 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:38.947219+0000", 2024-02-05T04:53:40.641 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:40.641 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.641 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.641 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:40.641 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:38.947219+0000", 2024-02-05T04:53:40.641 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:40.642 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.642 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.642 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:40.642 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:38.947219+0000", 2024-02-05T04:53:40.642 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.3900000000000001e-07 2024-02-05T04:53:40.642 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.642 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.642 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:40.642 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:38.947227+0000", 2024-02-05T04:53:40.642 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.5790000000000004e-06 2024-02-05T04:53:40.642 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.642 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.642 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:40.642 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:38.947241+0000", 2024-02-05T04:53:40.642 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.3613e-05 2024-02-05T04:53:40.643 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.643 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.643 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:40.643 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:38.947252+0000", 2024-02-05T04:53:40.643 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1102e-05 2024-02-05T04:53:40.643 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.643 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.643 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:40.643 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:38.947326+0000", 2024-02-05T04:53:40.643 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.4333999999999997e-05 2024-02-05T04:53:40.643 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.643 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.643 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:40.644 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:38.947473+0000", 2024-02-05T04:53:40.644 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00014660000000000001 2024-02-05T04:53:40.644 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:40.644 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:40.644 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:40.644 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.644 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.644 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpRead(3.bs3 47/31 ECSubRead(tid=160, to_read={3:df6528f1:::smithi19639362-33:head=0,135168,0}, subchunks={3:df6528f1:::smithi19639362-33:head=[0,1]}, attrs_to_read=))", 2024-02-05T04:53:40.644 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:38.947822+0000", 2024-02-05T04:53:40.644 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 1.6226531070000001, 2024-02-05T04:53:40.644 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000247125, 2024-02-05T04:53:40.644 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:40.644 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:40.644 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:40.644 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.645 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:40.645 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:38.947822+0000", 2024-02-05T04:53:40.645 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:40.645 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.645 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.645 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:40.645 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:38.947822+0000", 2024-02-05T04:53:40.645 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:40.645 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.645 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.645 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:40.645 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:38.947822+0000", 2024-02-05T04:53:40.645 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.7000000000000005e-07 2024-02-05T04:53:40.645 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.645 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.646 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:40.646 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:38.947831+0000", 2024-02-05T04:53:40.646 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.4449999999999996e-06 2024-02-05T04:53:40.646 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.646 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.646 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:40.646 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:38.947843+0000", 2024-02-05T04:53:40.646 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1962e-05 2024-02-05T04:53:40.646 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.646 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.646 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:40.646 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:38.947859+0000", 2024-02-05T04:53:40.646 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.6257999999999999e-05 2024-02-05T04:53:40.646 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.646 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.647 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:40.647 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:38.947935+0000", 2024-02-05T04:53:40.647 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.5457000000000006e-05 2024-02-05T04:53:40.647 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.647 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.647 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:40.647 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:38.948069+0000", 2024-02-05T04:53:40.647 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00013443299999999999 2024-02-05T04:53:40.647 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:40.647 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:40.647 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:40.647 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.648 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.648 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpRead(3.bs3 47/31 ECSubRead(tid=161, to_read={3:df6528f1:::smithi19639362-33:head=0,135168,0}, subchunks={3:df6528f1:::smithi19639362-33:head=[0,1]}, attrs_to_read=))", 2024-02-05T04:53:40.648 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:38.948821+0000", 2024-02-05T04:53:40.648 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 1.621653738, 2024-02-05T04:53:40.648 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000419679, 2024-02-05T04:53:40.648 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:40.648 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:40.648 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:40.648 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.648 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:40.648 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:38.948821+0000", 2024-02-05T04:53:40.648 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:40.648 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.649 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.649 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:40.649 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:38.948821+0000", 2024-02-05T04:53:40.649 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:40.649 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.649 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.649 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:40.649 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:38.948822+0000", 2024-02-05T04:53:40.649 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.9599999999999999e-07 2024-02-05T04:53:40.649 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.649 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.649 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:40.649 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:38.948829+0000", 2024-02-05T04:53:40.649 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.9430000000000003e-06 2024-02-05T04:53:40.650 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.650 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.650 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:40.650 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:38.948840+0000", 2024-02-05T04:53:40.650 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1491000000000001e-05 2024-02-05T04:53:40.650 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.650 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.650 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:40.650 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:38.948850+0000", 2024-02-05T04:53:40.650 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.3929999999999997e-06 2024-02-05T04:53:40.650 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.650 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.650 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:40.650 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:38.949129+0000", 2024-02-05T04:53:40.650 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00027895699999999998 2024-02-05T04:53:40.650 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:40.651 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:40.651 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:40.651 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:38.949241+0000", 2024-02-05T04:53:40.651 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000112299 2024-02-05T04:53:40.651 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:40.651 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:40.651 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:40.651 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:40.651 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:40.651 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-02-05T04:53:40.652 INFO:tasks.ceph.osd.15.smithi196.stderr:2024-02-05T04:53:40.651+0000 7f1facc45640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 34611) UID: 0 2024-02-05T04:53:40.653 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-02-05T04:53:40.712 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T04:53:40.753 INFO:tasks.ceph.osd.14.smithi150.stderr:2024-02-05T04:53:40.752+0000 7f5f81083640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 34020) UID: 0 2024-02-05T04:53:40.761 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-02-05T04:53:40.762 INFO:teuthology.orchestra.run.smithi062.stdout: "ops": [], 2024-02-05T04:53:40.762 INFO:teuthology.orchestra.run.smithi062.stdout: "num_ops": 0 2024-02-05T04:53:40.762 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-02-05T04:53:40.771 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-02-05T04:53:40.838 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0: {} 2024-02-05T04:53:40.838 INFO:teuthology.orchestra.run.smithi062.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T04:53:40.855 INFO:tasks.ceph.osd.10.smithi150.stderr:2024-02-05T04:53:40.853+0000 7f3075a4d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 34018) UID: 0 2024-02-05T04:53:40.857 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1: {} 2024-02-05T04:53:40.857 INFO:teuthology.orchestra.run.smithi062.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T04:53:40.875 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2: {} 2024-02-05T04:53:40.875 INFO:teuthology.orchestra.run.smithi062.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T04:53:40.880 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-02-05T04:53:40.880 INFO:teuthology.orchestra.run.smithi062.stdout: "ops": [], 2024-02-05T04:53:40.880 INFO:teuthology.orchestra.run.smithi062.stdout: "complaint_time": 30, 2024-02-05T04:53:40.880 INFO:teuthology.orchestra.run.smithi062.stdout: "num_blocked_ops": 0 2024-02-05T04:53:40.880 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-02-05T04:53:40.889 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-02-05T04:53:40.893 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3: {} 2024-02-05T04:53:40.893 INFO:teuthology.orchestra.run.smithi062.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T04:53:40.910 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4: {} 2024-02-05T04:53:40.911 INFO:teuthology.orchestra.run.smithi062.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T04:53:40.928 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5: {} 2024-02-05T04:53:40.929 INFO:teuthology.orchestra.run.smithi062.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T04:53:40.946 INFO:teuthology.orchestra.run.smithi062.stdout:osd.6: {} 2024-02-05T04:53:40.946 INFO:teuthology.orchestra.run.smithi062.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T04:53:40.951 INFO:tasks.rados.rados.0.smithi196.stdout:163: done (0 left) 2024-02-05T04:53:40.951 INFO:tasks.rados.rados.0.smithi196.stdout:164: read oid 26 snap -1 2024-02-05T04:53:40.951 INFO:tasks.rados.rados.0.smithi196.stdout:164: expect (ObjNum 77 snap 4 seq_num 77) 2024-02-05T04:53:40.951 INFO:tasks.rados.rados.0.smithi196.stdout:append oid 2 current snap is 10 2024-02-05T04:53:40.951 INFO:tasks.rados.rados.0.smithi196.stdout:165: seq_num 90 ranges {1294336=753664,2048000=720896,2768896=49152} 2024-02-05T04:53:40.954 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-02-05T04:53:40.956 INFO:tasks.ceph.osd.12.smithi062.stderr:2024-02-05T04:53:40.955+0000 7f5e5847e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 34695) UID: 0 2024-02-05T04:53:40.962 INFO:tasks.rados.rados.0.smithi196.stdout:165: writing smithi19639362-2 from 1294336 to 2048000 tid 1 2024-02-05T04:53:40.967 INFO:teuthology.orchestra.run.smithi062.stdout:osd.7: {} 2024-02-05T04:53:40.968 INFO:teuthology.orchestra.run.smithi062.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T04:53:40.968 INFO:tasks.rados.rados.0.smithi196.stdout:165: writing smithi19639362-2 from 2048000 to 2768896 tid 2 2024-02-05T04:53:40.968 INFO:tasks.rados.rados.0.smithi196.stdout:165: writing smithi19639362-2 from 2768896 to 2818048 tid 3 2024-02-05T04:53:40.968 INFO:tasks.rados.rados.0.smithi196.stdout:166: delete oid 38 current snap is 10 2024-02-05T04:53:40.968 INFO:tasks.rados.rados.0.smithi196.stdout:164: expect (ObjNum 77 snap 4 seq_num 77) 2024-02-05T04:53:40.988 INFO:teuthology.orchestra.run.smithi062.stdout:osd.8: {} 2024-02-05T04:53:40.988 INFO:teuthology.orchestra.run.smithi062.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-05T04:53:41.003 INFO:teuthology.orchestra.run.smithi062.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:41.003 INFO:teuthology.orchestra.run.smithi062.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:41.006 INFO:teuthology.orchestra.run.smithi062.stdout:osd.9: {} 2024-02-05T04:53:41.006 INFO:teuthology.orchestra.run.smithi062.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-05T04:53:41.012 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-02-05T04:53:41.025 INFO:teuthology.orchestra.run.smithi062.stdout:osd.10: {} 2024-02-05T04:53:41.025 INFO:teuthology.orchestra.run.smithi062.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-05T04:53:41.042 INFO:teuthology.orchestra.run.smithi062.stdout:osd.11: {} 2024-02-05T04:53:41.043 INFO:teuthology.orchestra.run.smithi062.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-05T04:53:41.055 INFO:tasks.rados.rados.0.smithi196.stdout:165: finishing write tid 1 to smithi19639362-2 2024-02-05T04:53:41.055 INFO:tasks.rados.rados.0.smithi196.stdout:165: finishing write tid 2 to smithi19639362-2 2024-02-05T04:53:41.056 INFO:tasks.rados.rados.0.smithi196.stdout:165: finishing write tid 3 to smithi19639362-2 2024-02-05T04:53:41.056 INFO:tasks.rados.rados.0.smithi196.stdout:165: finishing write tid 4 to smithi19639362-2 2024-02-05T04:53:41.056 INFO:tasks.rados.rados.0.smithi196.stdout:165: finishing write tid 5 to smithi19639362-2 2024-02-05T04:53:41.056 INFO:tasks.rados.rados.0.smithi196.stdout:165: oid 2 updating version 0 to 39 2024-02-05T04:53:41.056 INFO:tasks.rados.rados.0.smithi196.stdout:165: oid 2 version 39 is already newer than 36 2024-02-05T04:53:41.056 INFO:tasks.rados.rados.0.smithi196.stdout:165: oid 2 version 39 is already newer than 37 2024-02-05T04:53:41.056 INFO:tasks.rados.rados.0.smithi196.stdout:165: oid 2 version 39 is already newer than 38 2024-02-05T04:53:41.056 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 2 v 39 (ObjNum 90 snap 10 seq_num 90) dirty exists 2024-02-05T04:53:41.056 INFO:tasks.rados.rados.0.smithi196.stdout:165: left oid 2 (ObjNum 90 snap 10 seq_num 90) 2024-02-05T04:53:41.056 INFO:tasks.rados.rados.0.smithi196.stdout:164: done (2 left) 2024-02-05T04:53:41.056 INFO:tasks.rados.rados.0.smithi196.stdout:165: done (1 left) 2024-02-05T04:53:41.056 INFO:tasks.rados.rados.0.smithi196.stdout:166: done (0 left) 2024-02-05T04:53:41.056 INFO:tasks.rados.rados.0.smithi196.stdout:167: copy_from oid 22 from oid 38 current snap is 10 2024-02-05T04:53:41.056 INFO:tasks.rados.rados.0.smithi196.stdout:168: snap_create 2024-02-05T04:53:41.057 INFO:tasks.rados.rados.0.smithi196.stdout:167: finishing copy_from to smithi19639362-22 2024-02-05T04:53:41.057 INFO:tasks.rados.rados.0.smithi196.stdout:167: got expected ENOENT (src dne) 2024-02-05T04:53:41.057 INFO:tasks.rados.rados.0.smithi196.stdout:167: finishing copy_from racing read to smithi19639362-22 2024-02-05T04:53:41.058 INFO:tasks.ceph.osd.8.smithi062.stderr:2024-02-05T04:53:41.057+0000 7f9385bf2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 34693) UID: 0 2024-02-05T04:53:41.059 INFO:teuthology.orchestra.run.smithi062.stdout:osd.12: {} 2024-02-05T04:53:41.060 INFO:teuthology.orchestra.run.smithi062.stderr:osd.12: osd_enable_op_tracker = '' 2024-02-05T04:53:41.078 INFO:teuthology.orchestra.run.smithi062.stdout:osd.13: {} 2024-02-05T04:53:41.079 INFO:teuthology.orchestra.run.smithi062.stderr:osd.13: osd_enable_op_tracker = '' 2024-02-05T04:53:41.097 INFO:teuthology.orchestra.run.smithi062.stdout:osd.14: {} 2024-02-05T04:53:41.098 INFO:teuthology.orchestra.run.smithi062.stderr:osd.14: osd_enable_op_tracker = '' 2024-02-05T04:53:41.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-02-05T04:53:41.097+0000 7f5ec6541640 0 -- 172.21.15.62:0/2496877067 >> [v2:172.21.15.196:6826/3187639683,v1:172.21.15.196:6827/3187639683] conn(0x7f5ec021fcb0 msgr2=0x7f5ec02220a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-02-05T04:53:41.116 INFO:teuthology.orchestra.run.smithi062.stdout:osd.15: {} 2024-02-05T04:53:41.116 INFO:teuthology.orchestra.run.smithi062.stderr:osd.15: osd_enable_op_tracker = '' 2024-02-05T04:53:41.145 INFO:teuthology.orchestra.run.smithi062.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:41.145 INFO:teuthology.orchestra.run.smithi062.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:41.154 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-02-05T04:53:41.158 INFO:tasks.ceph.osd.3.smithi196.stderr:2024-02-05T04:53:41.158+0000 7fc8817c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34605) UID: 0 2024-02-05T04:53:41.261 INFO:tasks.ceph.osd.4.smithi062.stderr:2024-02-05T04:53:41.259+0000 7f708c308640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34689) UID: 0 2024-02-05T04:53:41.301 INFO:teuthology.orchestra.run.smithi062.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:41.302 INFO:teuthology.orchestra.run.smithi062.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:41.310 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-02-05T04:53:41.360 INFO:tasks.ceph.osd.5.smithi091.stderr:2024-02-05T04:53:41.359+0000 7f63983b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34277) UID: 0 2024-02-05T04:53:41.458 INFO:teuthology.orchestra.run.smithi062.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:41.459 INFO:teuthology.orchestra.run.smithi062.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:41.461 INFO:tasks.ceph.osd.11.smithi196.stderr:2024-02-05T04:53:41.460+0000 7f85a8d86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 34609) UID: 0 2024-02-05T04:53:41.467 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-02-05T04:53:41.562 INFO:tasks.ceph.osd.7.smithi196.stderr:2024-02-05T04:53:41.561+0000 7fb281419640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34606) UID: 0 2024-02-05T04:53:41.574 INFO:teuthology.orchestra.run.smithi062.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:41.574 INFO:teuthology.orchestra.run.smithi062.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:41.583 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-02-05T04:53:41.663 INFO:tasks.ceph.osd.9.smithi091.stderr:2024-02-05T04:53:41.662+0000 7efc18827640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 34279) UID: 0 2024-02-05T04:53:41.691 INFO:teuthology.orchestra.run.smithi062.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:41.691 INFO:teuthology.orchestra.run.smithi062.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:41.699 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-02-05T04:53:41.764 INFO:tasks.ceph.osd.15.smithi196.stderr:2024-02-05T04:53:41.763+0000 7f1facc45640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 34611) UID: 0 2024-02-05T04:53:41.806 INFO:teuthology.orchestra.run.smithi062.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:41.807 INFO:teuthology.orchestra.run.smithi062.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:41.815 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-02-05T04:53:41.865 INFO:tasks.ceph.osd.10.smithi150.stderr:2024-02-05T04:53:41.864+0000 7f3075a4d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 34018) UID: 0 2024-02-05T04:53:41.932 INFO:teuthology.orchestra.run.smithi091.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:41.932 INFO:teuthology.orchestra.run.smithi091.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:41.941 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-02-05T04:53:41.946 INFO:teuthology.orchestra.run.smithi062.stderr:nodeep-scrub is unset 2024-02-05T04:53:41.952 INFO:tasks.rados.rados.0.smithi196.stdout:167: done (1 left) 2024-02-05T04:53:41.952 INFO:tasks.rados.rados.0.smithi196.stdout:168: done (0 left) 2024-02-05T04:53:41.952 INFO:tasks.rados.rados.0.smithi196.stdout:169: setattr oid 34 current snap is 11 2024-02-05T04:53:41.953 INFO:tasks.rados.rados.0.smithi196.stdout:append oid 27 current snap is 11 2024-02-05T04:53:41.953 INFO:tasks.rados.rados.0.smithi196.stdout:170: seq_num 92 ranges {442368=786432,1228800=491520,1720320=278528} 2024-02-05T04:53:41.960 INFO:tasks.rados.rados.0.smithi196.stdout:170: writing smithi19639362-27 from 442368 to 1228800 tid 1 2024-02-05T04:53:41.963 INFO:tasks.rados.rados.0.smithi196.stdout:170: writing smithi19639362-27 from 1228800 to 1720320 tid 2 2024-02-05T04:53:41.966 INFO:tasks.rados.rados.0.smithi196.stdout:170: writing smithi19639362-27 from 1720320 to 1998848 tid 3 2024-02-05T04:53:41.966 INFO:tasks.rados.rados.0.smithi196.stdout:171: copy_from oid 19 from oid 24 current snap is 11 2024-02-05T04:53:41.966 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 34 v 45 (ObjNum 65 snap 0 seq_num 3157353872) dirty exists 2024-02-05T04:53:41.966 INFO:tasks.rados.rados.0.smithi196.stdout:169: done (2 left) 2024-02-05T04:53:41.966 INFO:tasks.rados.rados.0.smithi196.stdout:172: rollback oid 48 current snap is 11 2024-02-05T04:53:41.966 INFO:tasks.rados.rados.0.smithi196.stdout:rollback oid 48 to 5 2024-02-05T04:53:41.966 INFO:tasks.rados.rados.0.smithi196.stdout:173: copy_from oid 4 from oid 33 current snap is 11 2024-02-05T04:53:41.966 INFO:tasks.rados.rados.0.smithi196.stdout:append oid 35 current snap is 11 2024-02-05T04:53:41.967 INFO:tasks.rados.rados.0.smithi196.stdout:174: seq_num 93 ranges {1736704=802816,2539520=638976,3178496=131072} 2024-02-05T04:53:41.967 INFO:tasks.ceph.osd.10.smithi150.stderr:2024-02-05T04:53:41.965+0000 7f3075a4d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 34018) UID: 0 2024-02-05T04:53:41.980 INFO:tasks.rados.rados.0.smithi196.stdout:174: writing smithi19639362-35 from 1736704 to 2539520 tid 1 2024-02-05T04:53:41.983 INFO:tasks.rados.rados.0.smithi196.stdout:174: writing smithi19639362-35 from 2539520 to 3178496 tid 2 2024-02-05T04:53:41.988 INFO:tasks.rados.rados.0.smithi196.stdout:174: writing smithi19639362-35 from 3178496 to 3309568 tid 3 2024-02-05T04:53:41.988 INFO:tasks.rados.rados.0.smithi196.stdout:175: rollback oid 8 current snap is 11 2024-02-05T04:53:41.988 INFO:tasks.rados.rados.0.smithi196.stdout:rollback oid 8 to 10 2024-02-05T04:53:41.988 INFO:tasks.rados.rados.0.smithi196.stdout:170: finishing write tid 1 to smithi19639362-27 2024-02-05T04:53:41.988 INFO:tasks.rados.rados.0.smithi196.stdout:170: finishing write tid 2 to smithi19639362-27 2024-02-05T04:53:41.988 INFO:tasks.rados.rados.0.smithi196.stdout:173: finishing copy_from racing read to smithi19639362-4 2024-02-05T04:53:41.988 INFO:tasks.rados.rados.0.smithi196.stdout:170: finishing write tid 3 to smithi19639362-27 2024-02-05T04:53:41.988 INFO:tasks.rados.rados.0.smithi196.stdout:170: finishing write tid 4 to smithi19639362-27 2024-02-05T04:53:41.988 INFO:tasks.rados.rados.0.smithi196.stdout:171: finishing copy_from racing read to smithi19639362-19 2024-02-05T04:53:41.988 INFO:tasks.rados.rados.0.smithi196.stdout:172: finishing rollback tid 0 to smithi19639362-48 2024-02-05T04:53:41.988 INFO:tasks.rados.rados.0.smithi196.stdout:170: finishing write tid 5 to smithi19639362-27 2024-02-05T04:53:41.988 INFO:tasks.rados.rados.0.smithi196.stdout:170: oid 27 updating version 0 to 13 2024-02-05T04:53:41.989 INFO:tasks.rados.rados.0.smithi196.stdout:170: oid 27 version 13 is already newer than 11 2024-02-05T04:53:41.989 INFO:tasks.rados.rados.0.smithi196.stdout:170: oid 27 version 13 is already newer than 10 2024-02-05T04:53:41.989 INFO:tasks.rados.rados.0.smithi196.stdout:170: oid 27 version 13 is already newer than 12 2024-02-05T04:53:41.989 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 27 v 13 (ObjNum 92 snap 11 seq_num 92) dirty exists 2024-02-05T04:53:41.989 INFO:tasks.rados.rados.0.smithi196.stdout:170: left oid 27 (ObjNum 92 snap 11 seq_num 92) 2024-02-05T04:53:41.989 INFO:tasks.rados.rados.0.smithi196.stdout:172: finishing rollback tid 1 to smithi19639362-48 2024-02-05T04:53:41.989 INFO:tasks.rados.rados.0.smithi196.stdout:172: finishing rollback tid 2 to smithi19639362-48 2024-02-05T04:53:41.989 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 48 v 15 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-02-05T04:53:41.989 INFO:tasks.rados.rados.0.smithi196.stdout:173: finishing copy_from to smithi19639362-4 2024-02-05T04:53:41.989 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 4 v 8 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-02-05T04:53:41.989 INFO:tasks.rados.rados.0.smithi196.stdout:171: finishing copy_from to smithi19639362-19 2024-02-05T04:53:41.989 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 19 v 6 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-02-05T04:53:41.989 INFO:tasks.rados.rados.0.smithi196.stdout:170: done (5 left) 2024-02-05T04:53:41.989 INFO:tasks.rados.rados.0.smithi196.stdout:171: done (4 left) 2024-02-05T04:53:41.990 INFO:tasks.rados.rados.0.smithi196.stdout:172: done (3 left) 2024-02-05T04:53:41.990 INFO:tasks.rados.rados.0.smithi196.stdout:173: done (2 left) 2024-02-05T04:53:41.990 INFO:tasks.rados.rados.0.smithi196.stdout:append oid 33 current snap is 11 2024-02-05T04:53:41.990 INFO:tasks.rados.rados.0.smithi196.stdout:176: seq_num 94 ranges {540672=425984,966656=786432,1753088=376832} 2024-02-05T04:53:41.990 INFO:tasks.rados.rados.0.smithi196.stdout:176: writing smithi19639362-33 from 540672 to 966656 tid 1 2024-02-05T04:53:41.994 INFO:tasks.rados.rados.0.smithi196.stdout:176: writing smithi19639362-33 from 966656 to 1753088 tid 2 2024-02-05T04:53:41.997 INFO:tasks.rados.rados.0.smithi196.stdout:176: writing smithi19639362-33 from 1753088 to 2129920 tid 3 2024-02-05T04:53:41.997 INFO:tasks.rados.rados.0.smithi196.stdout:177: read oid 34 snap -1 2024-02-05T04:53:41.997 INFO:tasks.rados.rados.0.smithi196.stdout:177: expect (ObjNum 0 snap 0 seq_num 0) 2024-02-05T04:53:41.997 INFO:tasks.rados.rados.0.smithi196.stdout:178: rmattr oid 22 current snap is 11 2024-02-05T04:53:41.997 INFO:tasks.rados.rados.0.smithi196.stdout:178: done (4 left) 2024-02-05T04:53:41.997 INFO:tasks.rados.rados.0.smithi196.stdout:append oid 42 current snap is 11 2024-02-05T04:53:41.997 INFO:tasks.rados.rados.0.smithi196.stdout:174: finishing write tid 1 to smithi19639362-35 2024-02-05T04:53:41.997 INFO:tasks.rados.rados.0.smithi196.stdout:179: seq_num 95 ranges {1687552=442368,2129920=540672,2670592=622592} 2024-02-05T04:53:42.008 INFO:tasks.rados.rados.0.smithi196.stdout:179: writing smithi19639362-42 from 1687552 to 2129920 tid 1 2024-02-05T04:53:42.011 INFO:tasks.rados.rados.0.smithi196.stdout:179: writing smithi19639362-42 from 2129920 to 2670592 tid 2 2024-02-05T04:53:42.015 INFO:tasks.rados.rados.0.smithi196.stdout:179: writing smithi19639362-42 from 2670592 to 3293184 tid 3 2024-02-05T04:53:42.015 INFO:tasks.rados.rados.0.smithi196.stdout:append oid 1 current snap is 11 2024-02-05T04:53:42.016 INFO:tasks.rados.rados.0.smithi196.stdout:175: finishing rollback tid 0 to smithi19639362-8 2024-02-05T04:53:42.016 INFO:tasks.rados.rados.0.smithi196.stdout:175: finishing rollback tid 1 to smithi19639362-8 2024-02-05T04:53:42.016 INFO:tasks.rados.rados.0.smithi196.stdout:174: finishing write tid 2 to smithi19639362-35 2024-02-05T04:53:42.016 INFO:tasks.rados.rados.0.smithi196.stdout:174: finishing write tid 3 to smithi19639362-35 2024-02-05T04:53:42.016 INFO:tasks.rados.rados.0.smithi196.stdout:174: finishing write tid 4 to smithi19639362-35 2024-02-05T04:53:42.016 INFO:tasks.rados.rados.0.smithi196.stdout:180: seq_num 96 ranges {1245184=458752,1703936=688128,2392064=475136} 2024-02-05T04:53:42.025 INFO:tasks.rados.rados.0.smithi196.stdout:180: writing smithi19639362-1 from 1245184 to 1703936 tid 1 2024-02-05T04:53:42.029 INFO:tasks.rados.rados.0.smithi196.stdout:180: writing smithi19639362-1 from 1703936 to 2392064 tid 2 2024-02-05T04:53:42.032 INFO:tasks.rados.rados.0.smithi196.stdout:180: writing smithi19639362-1 from 2392064 to 2867200 tid 3 2024-02-05T04:53:42.033 INFO:tasks.rados.rados.0.smithi196.stdout:181: rmattr oid 29 current snap is 11 2024-02-05T04:53:42.033 INFO:tasks.rados.rados.0.smithi196.stdout:175: finishing rollback tid 2 to smithi19639362-8 2024-02-05T04:53:42.033 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 8 v 41 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-02-05T04:53:42.033 INFO:tasks.rados.rados.0.smithi196.stdout:174: finishing write tid 5 to smithi19639362-35 2024-02-05T04:53:42.033 INFO:tasks.rados.rados.0.smithi196.stdout:174: oid 35 updating version 0 to 40 2024-02-05T04:53:42.033 INFO:tasks.rados.rados.0.smithi196.stdout:174: oid 35 updating version 40 to 43 2024-02-05T04:53:42.033 INFO:tasks.rados.rados.0.smithi196.stdout:174: oid 35 version 43 is already newer than 41 2024-02-05T04:53:42.033 INFO:tasks.rados.rados.0.smithi196.stdout:174: oid 35 version 43 is already newer than 42 2024-02-05T04:53:42.033 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 35 v 43 (ObjNum 93 snap 11 seq_num 93) dirty exists 2024-02-05T04:53:42.033 INFO:tasks.rados.rados.0.smithi196.stdout:174: left oid 35 (ObjNum 93 snap 11 seq_num 93) 2024-02-05T04:53:42.034 INFO:tasks.rados.rados.0.smithi196.stdout:176: finishing write tid 1 to smithi19639362-33 2024-02-05T04:53:42.034 INFO:tasks.rados.rados.0.smithi196.stdout:174: done (6 left) 2024-02-05T04:53:42.034 INFO:tasks.rados.rados.0.smithi196.stdout:175: done (5 left) 2024-02-05T04:53:42.034 INFO:tasks.rados.rados.0.smithi196.stdout:append oid 35 current snap is 11 2024-02-05T04:53:42.034 INFO:tasks.rados.rados.0.smithi196.stdout:177: expect (ObjNum 0 snap 0 seq_num 0) 2024-02-05T04:53:42.036 INFO:tasks.rados.rados.0.smithi196.stdout:176: finishing write tid 2 to smithi19639362-33 2024-02-05T04:53:42.036 INFO:tasks.rados.rados.0.smithi196.stdout:176: finishing write tid 3 to smithi19639362-33 2024-02-05T04:53:42.036 INFO:tasks.rados.rados.0.smithi196.stdout:182: seq_num 97 ranges {3309568=475136,3784704=442368,4227072=442368,4669440=278528} 2024-02-05T04:53:42.056 INFO:tasks.rados.rados.0.smithi196.stdout:182: writing smithi19639362-35 from 3309568 to 3784704 tid 1 2024-02-05T04:53:42.059 INFO:tasks.rados.rados.0.smithi196.stdout:182: writing smithi19639362-35 from 3784704 to 4227072 tid 2 2024-02-05T04:53:42.061 INFO:tasks.rados.rados.0.smithi196.stdout:182: writing smithi19639362-35 from 4227072 to 4669440 tid 3 2024-02-05T04:53:42.064 INFO:tasks.rados.rados.0.smithi196.stdout:182: writing smithi19639362-35 from 4669440 to 4947968 tid 4 2024-02-05T04:53:42.064 INFO:tasks.rados.rados.0.smithi196.stdout:177: done (5 left) 2024-02-05T04:53:42.064 INFO:tasks.rados.rados.0.smithi196.stdout:183: read oid 12 snap 9 2024-02-05T04:53:42.064 INFO:tasks.rados.rados.0.smithi196.stdout:183: expect deleted 2024-02-05T04:53:42.064 INFO:tasks.rados.rados.0.smithi196.stdout:184: snap_remove snap 8 2024-02-05T04:53:42.067 INFO:tasks.ceph.osd.11.smithi196.stderr:2024-02-05T04:53:42.066+0000 7f85a8d86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 34609) UID: 0 2024-02-05T04:53:42.069 INFO:teuthology.orchestra.run.smithi091.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:42.069 INFO:teuthology.orchestra.run.smithi091.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:42.078 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-02-05T04:53:42.169 INFO:tasks.ceph.osd.12.smithi062.stderr:2024-02-05T04:53:42.168+0000 7f5e5847e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 34695) UID: 0 2024-02-05T04:53:42.184 INFO:teuthology.orchestra.run.smithi091.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:42.184 INFO:teuthology.orchestra.run.smithi091.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:42.193 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-02-05T04:53:42.269 INFO:tasks.ceph.osd.1.smithi091.stderr:2024-02-05T04:53:42.268+0000 7ff9dc283640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34275) UID: 0 2024-02-05T04:53:42.298 INFO:teuthology.orchestra.run.smithi091.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:42.299 INFO:teuthology.orchestra.run.smithi091.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:42.307 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-02-05T04:53:42.370 INFO:tasks.ceph.osd.10.smithi150.stderr:2024-02-05T04:53:42.369+0000 7f3075a4d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 34018) UID: 0 2024-02-05T04:53:42.453 INFO:teuthology.orchestra.run.smithi091.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:42.454 INFO:teuthology.orchestra.run.smithi091.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:42.462 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-02-05T04:53:42.506 INFO:tasks.ceph.osd.1.smithi091.stderr:2024-02-05T04:53:42.505+0000 7ff9dc283640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34275) UID: 0 2024-02-05T04:53:42.567 INFO:teuthology.orchestra.run.smithi091.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:42.567 INFO:teuthology.orchestra.run.smithi091.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:42.571 INFO:tasks.ceph.osd.3.smithi196.stderr:2024-02-05T04:53:42.570+0000 7fc8817c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34605) UID: 0 2024-02-05T04:53:42.576 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-02-05T04:53:42.673 INFO:tasks.ceph.osd.15.smithi196.stderr:2024-02-05T04:53:42.672+0000 7f1facc45640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 34611) UID: 0 2024-02-05T04:53:42.721 INFO:teuthology.orchestra.run.smithi091.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:42.721 INFO:teuthology.orchestra.run.smithi091.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:42.730 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-02-05T04:53:42.773 INFO:tasks.ceph.osd.2.smithi150.stderr:2024-02-05T04:53:42.772+0000 7f7337bf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 34014) UID: 0 2024-02-05T04:53:42.835 INFO:teuthology.orchestra.run.smithi091.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:42.835 INFO:teuthology.orchestra.run.smithi091.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:42.845 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-02-05T04:53:42.875 INFO:tasks.ceph.osd.14.smithi150.stderr:2024-02-05T04:53:42.874+0000 7f5f81083640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 34020) UID: 0 2024-02-05T04:53:42.950 INFO:teuthology.orchestra.run.smithi091.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:42.951 INFO:teuthology.orchestra.run.smithi091.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:42.954 INFO:tasks.rados.rados.0.smithi196.stdout:184: done (6 left) 2024-02-05T04:53:42.954 INFO:tasks.rados.rados.0.smithi196.stdout:185: copy_from oid 31 from oid 11 current snap is 11 2024-02-05T04:53:42.954 INFO:tasks.rados.rados.0.smithi196.stdout:176: finishing write tid 4 to smithi19639362-33 2024-02-05T04:53:42.954 INFO:tasks.rados.rados.0.smithi196.stdout:176: finishing write tid 5 to smithi19639362-33 2024-02-05T04:53:42.955 INFO:tasks.rados.rados.0.smithi196.stdout:176: oid 33 updating version 0 to 44 2024-02-05T04:53:42.955 INFO:tasks.rados.rados.0.smithi196.stdout:176: oid 33 version 44 is already newer than 41 2024-02-05T04:53:42.955 INFO:tasks.rados.rados.0.smithi196.stdout:176: oid 33 version 44 is already newer than 43 2024-02-05T04:53:42.955 INFO:tasks.rados.rados.0.smithi196.stdout:176: oid 33 version 44 is already newer than 42 2024-02-05T04:53:42.955 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 33 v 44 (ObjNum 94 snap 11 seq_num 94) dirty exists 2024-02-05T04:53:42.955 INFO:tasks.rados.rados.0.smithi196.stdout:176: left oid 33 (ObjNum 94 snap 11 seq_num 94) 2024-02-05T04:53:42.955 INFO:tasks.rados.rados.0.smithi196.stdout:179: finishing write tid 1 to smithi19639362-42 2024-02-05T04:53:42.955 INFO:tasks.rados.rados.0.smithi196.stdout:179: finishing write tid 2 to smithi19639362-42 2024-02-05T04:53:42.955 INFO:tasks.rados.rados.0.smithi196.stdout:179: finishing write tid 3 to smithi19639362-42 2024-02-05T04:53:42.955 INFO:tasks.rados.rados.0.smithi196.stdout:179: finishing write tid 4 to smithi19639362-42 2024-02-05T04:53:42.955 INFO:tasks.rados.rados.0.smithi196.stdout:179: finishing write tid 5 to smithi19639362-42 2024-02-05T04:53:42.955 INFO:tasks.rados.rados.0.smithi196.stdout:179: oid 42 updating version 0 to 26 2024-02-05T04:53:42.955 INFO:tasks.rados.rados.0.smithi196.stdout:179: oid 42 version 26 is already newer than 23 2024-02-05T04:53:42.955 INFO:tasks.rados.rados.0.smithi196.stdout:179: oid 42 version 26 is already newer than 24 2024-02-05T04:53:42.955 INFO:tasks.rados.rados.0.smithi196.stdout:179: oid 42 version 26 is already newer than 25 2024-02-05T04:53:42.956 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 42 v 26 (ObjNum 95 snap 11 seq_num 95) dirty exists 2024-02-05T04:53:42.956 INFO:tasks.rados.rados.0.smithi196.stdout:179: left oid 42 (ObjNum 95 snap 11 seq_num 95) 2024-02-05T04:53:42.956 INFO:tasks.rados.rados.0.smithi196.stdout:180: finishing write tid 1 to smithi19639362-1 2024-02-05T04:53:42.956 INFO:tasks.rados.rados.0.smithi196.stdout:180: finishing write tid 2 to smithi19639362-1 2024-02-05T04:53:42.956 INFO:tasks.rados.rados.0.smithi196.stdout:180: finishing write tid 3 to smithi19639362-1 2024-02-05T04:53:42.956 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 29 v 44 (ObjNum 78 snap 4 seq_num 78) dirty exists 2024-02-05T04:53:42.956 INFO:tasks.rados.rados.0.smithi196.stdout:180: finishing write tid 4 to smithi19639362-1 2024-02-05T04:53:42.956 INFO:tasks.rados.rados.0.smithi196.stdout:180: finishing write tid 5 to smithi19639362-1 2024-02-05T04:53:42.956 INFO:tasks.rados.rados.0.smithi196.stdout:180: oid 1 updating version 0 to 31 2024-02-05T04:53:42.956 INFO:tasks.rados.rados.0.smithi196.stdout:180: oid 1 version 31 is already newer than 29 2024-02-05T04:53:42.956 INFO:tasks.rados.rados.0.smithi196.stdout:180: oid 1 updating version 31 to 32 2024-02-05T04:53:42.956 INFO:tasks.rados.rados.0.smithi196.stdout:180: oid 1 version 32 is already newer than 30 2024-02-05T04:53:42.956 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 1 v 32 (ObjNum 96 snap 11 seq_num 96) dirty exists 2024-02-05T04:53:42.956 INFO:tasks.rados.rados.0.smithi196.stdout:180: left oid 1 (ObjNum 96 snap 11 seq_num 96) 2024-02-05T04:53:42.956 INFO:tasks.rados.rados.0.smithi196.stdout:182: finishing write tid 1 to smithi19639362-35 2024-02-05T04:53:42.957 INFO:tasks.rados.rados.0.smithi196.stdout:176: done (6 left) 2024-02-05T04:53:42.957 INFO:tasks.rados.rados.0.smithi196.stdout:179: done (5 left) 2024-02-05T04:53:42.957 INFO:tasks.rados.rados.0.smithi196.stdout:180: done (4 left) 2024-02-05T04:53:42.957 INFO:tasks.rados.rados.0.smithi196.stdout:181: done (3 left) 2024-02-05T04:53:42.957 INFO:tasks.rados.rados.0.smithi196.stdout:186: rmattr oid 26 current snap is 11 2024-02-05T04:53:42.957 INFO:tasks.rados.rados.0.smithi196.stdout:186: done (3 left) 2024-02-05T04:53:42.957 INFO:tasks.rados.rados.0.smithi196.stdout:187: read oid 1 snap 5 2024-02-05T04:53:42.957 INFO:tasks.rados.rados.0.smithi196.stdout:187: expect (ObjNum 72 snap 3 seq_num 72) 2024-02-05T04:53:42.957 INFO:tasks.rados.rados.0.smithi196.stdout:188: delete oid 28 current snap is 11 2024-02-05T04:53:42.957 INFO:tasks.rados.rados.0.smithi196.stdout:182: finishing write tid 2 to smithi19639362-35 2024-02-05T04:53:42.957 INFO:tasks.rados.rados.0.smithi196.stdout:182: finishing write tid 3 to smithi19639362-35 2024-02-05T04:53:42.957 INFO:tasks.rados.rados.0.smithi196.stdout:182: finishing write tid 4 to smithi19639362-35 2024-02-05T04:53:42.957 INFO:tasks.rados.rados.0.smithi196.stdout:182: finishing write tid 5 to smithi19639362-35 2024-02-05T04:53:42.957 INFO:tasks.rados.rados.0.smithi196.stdout:182: finishing write tid 6 to smithi19639362-35 2024-02-05T04:53:42.957 INFO:tasks.rados.rados.0.smithi196.stdout:182: oid 35 updating version 0 to 46 2024-02-05T04:53:42.957 INFO:tasks.rados.rados.0.smithi196.stdout:182: oid 35 version 46 is already newer than 44 2024-02-05T04:53:42.958 INFO:tasks.rados.rados.0.smithi196.stdout:182: oid 35 updating version 46 to 47 2024-02-05T04:53:42.958 INFO:tasks.rados.rados.0.smithi196.stdout:182: oid 35 updating version 47 to 48 2024-02-05T04:53:42.958 INFO:tasks.rados.rados.0.smithi196.stdout:182: oid 35 version 48 is already newer than 45 2024-02-05T04:53:42.958 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 35 v 48 (ObjNum 97 snap 11 seq_num 97) dirty exists 2024-02-05T04:53:42.958 INFO:tasks.rados.rados.0.smithi196.stdout:182: left oid 35 (ObjNum 97 snap 11 seq_num 97) 2024-02-05T04:53:42.965 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-02-05T04:53:42.968 INFO:tasks.rados.rados.0.smithi196.stdout:185: finishing copy_from racing read to smithi19639362-31 2024-02-05T04:53:42.968 INFO:tasks.rados.rados.0.smithi196.stdout:182: done (4 left) 2024-02-05T04:53:42.968 INFO:tasks.rados.rados.0.smithi196.stdout:183: done (3 left) 2024-02-05T04:53:42.968 INFO:tasks.rados.rados.0.smithi196.stdout:188: done (2 left) 2024-02-05T04:53:42.968 INFO:tasks.rados.rados.0.smithi196.stdout:189: snap_remove snap 10 2024-02-05T04:53:43.008 INFO:tasks.ceph.osd.9.smithi091.stderr:2024-02-05T04:53:43.007+0000 7efc18827640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 34279) UID: 0 2024-02-05T04:53:43.072 INFO:teuthology.orchestra.run.smithi091.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:43.072 INFO:teuthology.orchestra.run.smithi091.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:43.076 INFO:tasks.ceph.osd.10.smithi150.stderr:2024-02-05T04:53:43.075+0000 7f3075a4d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 34018) UID: 0 2024-02-05T04:53:43.080 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-02-05T04:53:43.128 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T04:53:43.177 INFO:tasks.ceph.osd.12.smithi062.stderr:2024-02-05T04:53:43.176+0000 7f5e5847e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 34695) UID: 0 2024-02-05T04:53:43.226 INFO:teuthology.orchestra.run.smithi091.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:43.226 INFO:teuthology.orchestra.run.smithi091.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:43.235 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-02-05T04:53:43.265 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0: {} 2024-02-05T04:53:43.266 INFO:teuthology.orchestra.run.smithi062.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T04:53:43.278 INFO:tasks.ceph.osd.5.smithi091.stderr:2024-02-05T04:53:43.277+0000 7f63983b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34277) UID: 0 2024-02-05T04:53:43.284 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1: {} 2024-02-05T04:53:43.284 INFO:teuthology.orchestra.run.smithi062.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T04:53:43.302 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2: {} 2024-02-05T04:53:43.303 INFO:teuthology.orchestra.run.smithi062.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T04:53:43.321 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3: {} 2024-02-05T04:53:43.321 INFO:teuthology.orchestra.run.smithi062.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T04:53:43.338 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4: {} 2024-02-05T04:53:43.338 INFO:teuthology.orchestra.run.smithi062.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T04:53:43.339 INFO:teuthology.orchestra.run.smithi091.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:43.339 INFO:teuthology.orchestra.run.smithi091.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:43.348 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-02-05T04:53:43.356 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5: {} 2024-02-05T04:53:43.356 INFO:teuthology.orchestra.run.smithi062.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T04:53:43.374 INFO:teuthology.orchestra.run.smithi062.stdout:osd.6: {} 2024-02-05T04:53:43.374 INFO:teuthology.orchestra.run.smithi062.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T04:53:43.379 INFO:tasks.ceph.osd.0.smithi062.stderr:2024-02-05T04:53:43.378+0000 7fece8e03640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34688) UID: 0 2024-02-05T04:53:43.392 INFO:teuthology.orchestra.run.smithi062.stdout:osd.7: {} 2024-02-05T04:53:43.392 INFO:teuthology.orchestra.run.smithi062.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T04:53:43.408 INFO:teuthology.orchestra.run.smithi062.stdout:osd.8: {} 2024-02-05T04:53:43.409 INFO:teuthology.orchestra.run.smithi062.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-05T04:53:43.426 INFO:teuthology.orchestra.run.smithi062.stdout:osd.9: {} 2024-02-05T04:53:43.426 INFO:teuthology.orchestra.run.smithi062.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-05T04:53:43.444 INFO:teuthology.orchestra.run.smithi062.stdout:osd.10: {} 2024-02-05T04:53:43.445 INFO:teuthology.orchestra.run.smithi062.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-05T04:53:43.462 INFO:teuthology.orchestra.run.smithi062.stdout:osd.11: {} 2024-02-05T04:53:43.463 INFO:teuthology.orchestra.run.smithi062.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-05T04:53:43.464 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-02-05T04:53:43.464 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [], 2024-02-05T04:53:43.464 INFO:teuthology.orchestra.run.smithi150.stdout: "num_ops": 0 2024-02-05T04:53:43.464 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-02-05T04:53:43.473 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-02-05T04:53:43.479 INFO:teuthology.orchestra.run.smithi062.stdout:osd.12: {} 2024-02-05T04:53:43.479 INFO:teuthology.orchestra.run.smithi062.stderr:osd.12: osd_enable_op_tracker = '' 2024-02-05T04:53:43.480 INFO:tasks.ceph.osd.15.smithi196.stderr:2024-02-05T04:53:43.479+0000 7f1facc45640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 34611) UID: 0 2024-02-05T04:53:43.496 INFO:teuthology.orchestra.run.smithi062.stdout:osd.13: {} 2024-02-05T04:53:43.497 INFO:teuthology.orchestra.run.smithi062.stderr:osd.13: osd_enable_op_tracker = '' 2024-02-05T04:53:43.513 INFO:teuthology.orchestra.run.smithi062.stdout:osd.14: {} 2024-02-05T04:53:43.514 INFO:teuthology.orchestra.run.smithi062.stderr:osd.14: osd_enable_op_tracker = '' 2024-02-05T04:53:43.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-02-05T04:53:43.513+0000 7f0eb5c2d640 0 -- 172.21.15.62:0/1375880773 >> [v2:172.21.15.196:6826/3187639683,v1:172.21.15.196:6827/3187639683] conn(0x7f0eb02220a0 msgr2=0x7f0eb0224490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-02-05T04:53:43.579 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-02-05T04:53:43.580 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [], 2024-02-05T04:53:43.580 INFO:teuthology.orchestra.run.smithi150.stdout: "complaint_time": 30, 2024-02-05T04:53:43.580 INFO:teuthology.orchestra.run.smithi150.stdout: "num_blocked_ops": 0 2024-02-05T04:53:43.580 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-02-05T04:53:43.581 INFO:tasks.ceph.osd.10.smithi150.stderr:2024-02-05T04:53:43.580+0000 7f3075a4d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 34018) UID: 0 2024-02-05T04:53:43.588 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-02-05T04:53:43.682 INFO:tasks.ceph.osd.8.smithi062.stderr:2024-02-05T04:53:43.681+0000 7f9385bf2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 34693) UID: 0 2024-02-05T04:53:43.733 INFO:teuthology.orchestra.run.smithi062.stdout:osd.15: {} 2024-02-05T04:53:43.734 INFO:teuthology.orchestra.run.smithi062.stderr:osd.15: osd_enable_op_tracker = '' 2024-02-05T04:53:43.736 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-02-05T04:53:43.736 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 20, 2024-02-05T04:53:43.737 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 600, 2024-02-05T04:53:43.737 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [ 2024-02-05T04:53:43.737 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.737 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.4209.0:85 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'55, mlcod=24'50)", 2024-02-05T04:53:43.737 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:09.697543+0000", 2024-02-05T04:53:43.737 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 34.037223279999999, 2024-02-05T04:53:43.737 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0081615030000000005, 2024-02-05T04:53:43.737 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:43.737 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:43.737 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:43.737 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.737 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:43.737 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.697543+0000", 2024-02-05T04:53:43.737 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:43.738 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.738 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.738 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:43.738 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.697543+0000", 2024-02-05T04:53:43.738 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:43.738 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.738 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.738 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:43.738 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.697544+0000", 2024-02-05T04:53:43.738 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.7899999999999999e-07 2024-02-05T04:53:43.738 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.738 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.738 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:43.738 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.697546+0000", 2024-02-05T04:53:43.739 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.7970000000000001e-06 2024-02-05T04:53:43.739 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.739 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.739 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:43.739 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.697553+0000", 2024-02-05T04:53:43.739 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.2929999999999997e-06 2024-02-05T04:53:43.739 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.739 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.739 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:43.739 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.697556+0000", 2024-02-05T04:53:43.739 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.4309999999999998e-06 2024-02-05T04:53:43.739 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.739 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.740 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:43.740 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.697596+0000", 2024-02-05T04:53:43.740 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.977e-05 2024-02-05T04:53:43.740 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.740 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.740 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:53:43.740 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.697607+0000", 2024-02-05T04:53:43.740 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1459e-05 2024-02-05T04:53:43.740 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.740 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.740 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-02-05T04:53:43.740 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.705669+0000", 2024-02-05T04:53:43.740 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0080620009999999992 2024-02-05T04:53:43.741 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.741 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.741 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:43.741 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.705705+0000", 2024-02-05T04:53:43.741 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.5373000000000002e-05 2024-02-05T04:53:43.741 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:43.741 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:43.741 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:43.741 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.741 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.741 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.4209.0:86 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'56, mlcod=24'52)", 2024-02-05T04:53:43.741 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:09.698014+0000", 2024-02-05T04:53:43.741 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 34.036752180000001, 2024-02-05T04:53:43.741 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0077071750000000001, 2024-02-05T04:53:43.741 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:43.742 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:43.742 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:43.742 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.742 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:43.742 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698014+0000", 2024-02-05T04:53:43.742 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:43.742 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.742 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.742 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:43.742 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698014+0000", 2024-02-05T04:53:43.742 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:43.742 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.742 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.742 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:43.742 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698015+0000", 2024-02-05T04:53:43.743 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.9900000000000002e-07 2024-02-05T04:53:43.743 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.743 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.743 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:43.743 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698018+0000", 2024-02-05T04:53:43.743 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.0019999999999998e-06 2024-02-05T04:53:43.743 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.743 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.743 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:43.743 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698024+0000", 2024-02-05T04:53:43.743 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.251e-06 2024-02-05T04:53:43.743 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.743 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.743 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:43.743 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698028+0000", 2024-02-05T04:53:43.743 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.76e-06 2024-02-05T04:53:43.744 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.744 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.744 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:43.744 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698066+0000", 2024-02-05T04:53:43.744 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.8556999999999997e-05 2024-02-05T04:53:43.744 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.744 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.744 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:53:43.744 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698078+0000", 2024-02-05T04:53:43.744 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1620999999999999e-05 2024-02-05T04:53:43.744 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.744 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.744 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-02-05T04:53:43.744 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.705706+0000", 2024-02-05T04:53:43.745 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0076284400000000002 2024-02-05T04:53:43.745 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.745 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.745 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:43.745 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.705721+0000", 2024-02-05T04:53:43.745 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.5245000000000001e-05 2024-02-05T04:53:43.745 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:43.745 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:43.745 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:43.745 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.745 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.745 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.4209.0:87 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'57, mlcod=24'52)", 2024-02-05T04:53:43.745 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:09.698307+0000", 2024-02-05T04:53:43.745 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 34.036459252, 2024-02-05T04:53:43.745 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0074282530000000001, 2024-02-05T04:53:43.746 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:43.746 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:43.746 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:43.746 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.746 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:43.746 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698307+0000", 2024-02-05T04:53:43.746 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:43.746 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.746 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.746 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:43.746 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698307+0000", 2024-02-05T04:53:43.746 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:43.746 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.746 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.746 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:43.747 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698308+0000", 2024-02-05T04:53:43.747 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.9900000000000001e-07 2024-02-05T04:53:43.747 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.747 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.747 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:43.747 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698358+0000", 2024-02-05T04:53:43.747 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.0649000000000002e-05 2024-02-05T04:53:43.747 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.747 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.747 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:43.747 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698366+0000", 2024-02-05T04:53:43.747 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.9000000000000006e-06 2024-02-05T04:53:43.747 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.747 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.747 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:43.748 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698370+0000", 2024-02-05T04:53:43.748 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.7270000000000001e-06 2024-02-05T04:53:43.748 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.748 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.748 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:43.748 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698410+0000", 2024-02-05T04:53:43.748 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.0117000000000002e-05 2024-02-05T04:53:43.748 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.748 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.748 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:53:43.748 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698423+0000", 2024-02-05T04:53:43.748 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2531000000000001e-05 2024-02-05T04:53:43.748 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.748 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.748 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-02-05T04:53:43.749 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.705722+0000", 2024-02-05T04:53:43.749 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0072995270000000001 2024-02-05T04:53:43.749 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.749 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.749 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:43.749 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.705735+0000", 2024-02-05T04:53:43.749 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.3403e-05 2024-02-05T04:53:43.749 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:43.749 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:43.749 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:43.749 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.749 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.749 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.4209.0:88 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'58, mlcod=24'52)", 2024-02-05T04:53:43.749 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:09.698756+0000", 2024-02-05T04:53:43.750 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 34.036010163999997, 2024-02-05T04:53:43.750 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0072080540000000002, 2024-02-05T04:53:43.750 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:43.750 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:43.750 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:43.750 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.750 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:43.750 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698756+0000", 2024-02-05T04:53:43.750 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:43.750 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.750 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.750 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:43.750 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698756+0000", 2024-02-05T04:53:43.751 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:43.751 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.751 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.751 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:43.751 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698757+0000", 2024-02-05T04:53:43.751 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.6899999999999997e-07 2024-02-05T04:53:43.751 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.751 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.751 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:43.751 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698762+0000", 2024-02-05T04:53:43.751 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.8790000000000002e-06 2024-02-05T04:53:43.751 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.751 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.751 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:43.751 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698773+0000", 2024-02-05T04:53:43.752 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1e-05 2024-02-05T04:53:43.752 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.752 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.752 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:43.752 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698780+0000", 2024-02-05T04:53:43.752 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.8920000000000003e-06 2024-02-05T04:53:43.752 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.752 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.752 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:43.752 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.700009+0000", 2024-02-05T04:53:43.752 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.001229665 2024-02-05T04:53:43.752 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.752 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.752 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:53:43.752 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.700022+0000", 2024-02-05T04:53:43.753 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2699e-05 2024-02-05T04:53:43.753 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.753 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.753 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-02-05T04:53:43.753 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.705943+0000", 2024-02-05T04:53:43.753 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0059209759999999997 2024-02-05T04:53:43.753 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.753 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.753 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:43.753 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.705964+0000", 2024-02-05T04:53:43.753 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.1274e-05 2024-02-05T04:53:43.753 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:43.753 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:43.753 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:43.754 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.754 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.754 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.4209.0:110 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'72, mlcod=24'67)", 2024-02-05T04:53:43.754 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:09.853702+0000", 2024-02-05T04:53:43.754 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 33.881063982000001, 2024-02-05T04:53:43.754 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0063171329999999999, 2024-02-05T04:53:43.754 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:43.754 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:43.754 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:43.754 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.754 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:43.754 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.853702+0000", 2024-02-05T04:53:43.754 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:43.754 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.754 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.755 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:43.755 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.853702+0000", 2024-02-05T04:53:43.755 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:43.755 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.755 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.755 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:43.755 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.853703+0000", 2024-02-05T04:53:43.755 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.1100000000000002e-07 2024-02-05T04:53:43.755 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.755 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.755 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:43.755 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.853706+0000", 2024-02-05T04:53:43.755 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.3629999999999998e-06 2024-02-05T04:53:43.755 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.755 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.756 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:43.756 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.853713+0000", 2024-02-05T04:53:43.756 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.8410000000000002e-06 2024-02-05T04:53:43.756 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.756 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.756 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:43.756 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.853717+0000", 2024-02-05T04:53:43.756 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.777e-06 2024-02-05T04:53:43.756 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.756 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.756 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:43.756 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.856746+0000", 2024-02-05T04:53:43.756 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0030292800000000001 2024-02-05T04:53:43.756 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.756 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.757 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:53:43.757 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.856762+0000", 2024-02-05T04:53:43.757 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.5651999999999999e-05 2024-02-05T04:53:43.757 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.757 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.757 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-02-05T04:53:43.757 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.859984+0000", 2024-02-05T04:53:43.757 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0032218289999999998 2024-02-05T04:53:43.757 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.757 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.757 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:43.757 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.860020+0000", 2024-02-05T04:53:43.757 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.608e-05 2024-02-05T04:53:43.757 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:43.757 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:43.758 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:43.758 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.758 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.758 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.4209.0:111 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'73, mlcod=24'68)", 2024-02-05T04:53:43.758 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:09.854221+0000", 2024-02-05T04:53:43.758 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 33.880545650999998, 2024-02-05T04:53:43.758 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0058190960000000002, 2024-02-05T04:53:43.758 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:43.758 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:43.758 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:43.758 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.758 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:43.758 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.854221+0000", 2024-02-05T04:53:43.758 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:43.758 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.759 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.759 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:43.759 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.854221+0000", 2024-02-05T04:53:43.759 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:43.759 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.759 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.759 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:43.759 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.854221+0000", 2024-02-05T04:53:43.759 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.72e-07 2024-02-05T04:53:43.759 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.759 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.759 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:43.759 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.854224+0000", 2024-02-05T04:53:43.759 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.1779999999999999e-06 2024-02-05T04:53:43.759 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.760 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.760 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:43.760 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.854231+0000", 2024-02-05T04:53:43.760 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.7440000000000001e-06 2024-02-05T04:53:43.760 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.760 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.760 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:43.760 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.854235+0000", 2024-02-05T04:53:43.760 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.8349999999999997e-06 2024-02-05T04:53:43.760 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.760 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.760 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:43.760 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.856952+0000", 2024-02-05T04:53:43.760 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0027175839999999999 2024-02-05T04:53:43.761 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.761 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.761 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:53:43.761 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.856966+0000", 2024-02-05T04:53:43.761 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.3739e-05 2024-02-05T04:53:43.761 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.761 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.761 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-02-05T04:53:43.761 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.860023+0000", 2024-02-05T04:53:43.761 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0030564979999999999 2024-02-05T04:53:43.761 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.761 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.761 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:43.761 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.860040+0000", 2024-02-05T04:53:43.761 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.7146000000000002e-05 2024-02-05T04:53:43.761 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:43.762 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:43.762 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:43.762 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.762 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.762 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.4209.0:115 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'77, mlcod=24'69)", 2024-02-05T04:53:43.762 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:09.856748+0000", 2024-02-05T04:53:43.762 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 33.878018765, 2024-02-05T04:53:43.762 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0058814849999999997, 2024-02-05T04:53:43.762 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:43.762 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:43.762 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:43.762 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.762 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:43.762 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.856748+0000", 2024-02-05T04:53:43.762 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:43.763 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.763 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.763 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:43.763 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.856748+0000", 2024-02-05T04:53:43.763 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:43.763 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.763 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.763 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:43.763 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.856748+0000", 2024-02-05T04:53:43.763 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.4099999999999998e-07 2024-02-05T04:53:43.763 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.763 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.763 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:43.763 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.856781+0000", 2024-02-05T04:53:43.763 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.3056999999999999e-05 2024-02-05T04:53:43.764 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.764 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.764 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:43.764 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.856791+0000", 2024-02-05T04:53:43.764 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.5659999999999996e-06 2024-02-05T04:53:43.764 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.764 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.764 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:43.764 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.856799+0000", 2024-02-05T04:53:43.764 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.8720000000000002e-06 2024-02-05T04:53:43.764 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.764 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.764 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:43.764 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.857805+0000", 2024-02-05T04:53:43.765 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0010061180000000001 2024-02-05T04:53:43.765 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.765 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.765 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:53:43.765 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.857820+0000", 2024-02-05T04:53:43.765 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.4883e-05 2024-02-05T04:53:43.765 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.765 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.765 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-02-05T04:53:43.765 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.862596+0000", 2024-02-05T04:53:43.765 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0047763249999999997 2024-02-05T04:53:43.765 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.765 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.765 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:43.765 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.862629+0000", 2024-02-05T04:53:43.765 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.2923000000000003e-05 2024-02-05T04:53:43.766 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:43.766 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:43.766 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:43.766 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.766 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.766 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.4209.0:117 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'79, mlcod=24'69)", 2024-02-05T04:53:43.766 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:09.857803+0000", 2024-02-05T04:53:43.766 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 33.87696304, 2024-02-05T04:53:43.766 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0076301729999999996, 2024-02-05T04:53:43.766 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:43.766 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:43.767 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:43.767 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.767 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:43.767 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.857803+0000", 2024-02-05T04:53:43.767 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:43.767 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.767 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.767 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:43.767 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.857803+0000", 2024-02-05T04:53:43.767 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:43.767 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.767 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.767 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:43.767 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.857804+0000", 2024-02-05T04:53:43.767 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.1999999999999999e-07 2024-02-05T04:53:43.768 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.768 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.768 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:43.768 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.857810+0000", 2024-02-05T04:53:43.768 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.8690000000000001e-06 2024-02-05T04:53:43.768 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.768 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.768 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:43.768 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.857821+0000", 2024-02-05T04:53:43.768 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1059e-05 2024-02-05T04:53:43.768 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.768 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.768 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:43.768 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.857827+0000", 2024-02-05T04:53:43.768 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.1430000000000003e-06 2024-02-05T04:53:43.769 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.769 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.769 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:43.769 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.860138+0000", 2024-02-05T04:53:43.769 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0023111160000000002 2024-02-05T04:53:43.769 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.769 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.769 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:53:43.769 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.860152+0000", 2024-02-05T04:53:43.769 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.3896999999999999e-05 2024-02-05T04:53:43.769 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.769 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.769 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-02-05T04:53:43.769 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.865400+0000", 2024-02-05T04:53:43.769 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0052475040000000001 2024-02-05T04:53:43.769 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.770 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.770 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:43.770 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.865434+0000", 2024-02-05T04:53:43.770 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.3865000000000002e-05 2024-02-05T04:53:43.770 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:43.770 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:43.770 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:43.770 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.770 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.770 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.4634.0:245 3.es0 3:796382fe:::smithi19639362-29:head [rollback 3] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:53:43.770 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:29.186870+0000", 2024-02-05T04:53:43.770 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 14.547896264, 2024-02-05T04:53:43.770 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0071777610000000004, 2024-02-05T04:53:43.770 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:43.770 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:43.771 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-02-05T04:53:43.771 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.4634", 2024-02-05T04:53:43.771 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:43.771 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 245 2024-02-05T04:53:43.771 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.771 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:43.771 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.771 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:43.771 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186870+0000", 2024-02-05T04:53:43.771 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:43.771 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.772 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.772 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:43.772 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186870+0000", 2024-02-05T04:53:43.772 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:43.772 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.772 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.772 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:43.772 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186870+0000", 2024-02-05T04:53:43.772 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.0400000000000002e-07 2024-02-05T04:53:43.772 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.772 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.772 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:43.772 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186874+0000", 2024-02-05T04:53:43.772 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.348e-06 2024-02-05T04:53:43.772 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.773 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.773 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:43.773 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186878+0000", 2024-02-05T04:53:43.773 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.481e-06 2024-02-05T04:53:43.773 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.773 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.773 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:43.773 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186881+0000", 2024-02-05T04:53:43.773 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.1879999999999998e-06 2024-02-05T04:53:43.773 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.773 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.773 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:43.773 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.188893+0000", 2024-02-05T04:53:43.773 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.002011231 2024-02-05T04:53:43.773 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.774 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.774 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:53:43.774 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.189032+0000", 2024-02-05T04:53:43.774 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000139467 2024-02-05T04:53:43.774 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.774 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.774 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:43.774 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.189572+0000", 2024-02-05T04:53:43.774 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00053986300000000002 2024-02-05T04:53:43.774 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.774 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.774 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:43.774 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.192597+0000", 2024-02-05T04:53:43.774 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0030245659999999998 2024-02-05T04:53:43.774 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.774 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.775 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-02-05T04:53:43.775 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.193877+0000", 2024-02-05T04:53:43.775 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.001280109 2024-02-05T04:53:43.775 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.775 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.775 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:43.775 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.194048+0000", 2024-02-05T04:53:43.775 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000171204 2024-02-05T04:53:43.775 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:43.775 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:43.775 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:43.775 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.775 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.775 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.4634.0:246 3.es0 3:796382fe:::smithi19639362-29:head [append 0~0] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:53:43.775 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:29.186925+0000", 2024-02-05T04:53:43.775 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 14.547841719999999, 2024-02-05T04:53:43.776 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0088703129999999995, 2024-02-05T04:53:43.776 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:43.776 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:43.776 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-02-05T04:53:43.776 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.4634", 2024-02-05T04:53:43.776 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:43.776 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 246 2024-02-05T04:53:43.776 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.776 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:43.776 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.776 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:43.776 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186925+0000", 2024-02-05T04:53:43.776 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:43.776 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.777 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.777 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:43.777 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186925+0000", 2024-02-05T04:53:43.777 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:43.777 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.777 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.777 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:43.777 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186925+0000", 2024-02-05T04:53:43.777 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.53e-07 2024-02-05T04:53:43.777 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.777 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.777 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:43.777 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186928+0000", 2024-02-05T04:53:43.777 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.6900000000000001e-06 2024-02-05T04:53:43.777 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.778 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.778 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:43.778 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186933+0000", 2024-02-05T04:53:43.778 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.502e-06 2024-02-05T04:53:43.778 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.778 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.778 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:43.778 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186937+0000", 2024-02-05T04:53:43.778 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.6200000000000001e-06 2024-02-05T04:53:43.778 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.778 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.778 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:43.778 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.190223+0000", 2024-02-05T04:53:43.778 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0032863050000000002 2024-02-05T04:53:43.778 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.779 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.779 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:53:43.779 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.190312+0000", 2024-02-05T04:53:43.779 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.8821999999999998e-05 2024-02-05T04:53:43.779 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.779 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.779 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:43.779 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.190713+0000", 2024-02-05T04:53:43.779 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00040077400000000001 2024-02-05T04:53:43.779 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.779 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.779 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:43.779 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.193148+0000", 2024-02-05T04:53:43.779 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.002435544 2024-02-05T04:53:43.779 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.780 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.780 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-02-05T04:53:43.780 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.195751+0000", 2024-02-05T04:53:43.780 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0026025000000000002 2024-02-05T04:53:43.780 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.780 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.780 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:43.780 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.195795+0000", 2024-02-05T04:53:43.780 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.4202999999999997e-05 2024-02-05T04:53:43.780 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:43.780 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:43.780 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:43.780 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.780 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.780 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.4634.0:249 3.es0 3:70a5b9dd:::smithi19639362-46:head [delete] snapc 3=[3] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:53:43.781 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:29.186943+0000", 2024-02-05T04:53:43.781 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 14.54782376, 2024-02-05T04:53:43.781 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.011398484, 2024-02-05T04:53:43.781 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:43.781 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:43.781 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-02-05T04:53:43.781 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.4634", 2024-02-05T04:53:43.781 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:43.781 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 249 2024-02-05T04:53:43.781 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.781 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:43.781 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.781 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:43.781 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186943+0000", 2024-02-05T04:53:43.782 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:43.782 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.782 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.782 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:43.782 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186943+0000", 2024-02-05T04:53:43.782 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:43.782 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.782 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.782 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:43.782 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186943+0000", 2024-02-05T04:53:43.782 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.1899999999999999e-07 2024-02-05T04:53:43.782 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.782 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.782 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:43.782 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186944+0000", 2024-02-05T04:53:43.783 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.207e-06 2024-02-05T04:53:43.783 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.783 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.783 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:43.783 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186948+0000", 2024-02-05T04:53:43.783 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.7129999999999999e-06 2024-02-05T04:53:43.783 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.783 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.783 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:43.783 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186951+0000", 2024-02-05T04:53:43.783 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.9979999999999999e-06 2024-02-05T04:53:43.783 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.783 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.783 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:43.783 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.191572+0000", 2024-02-05T04:53:43.784 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0046207669999999996 2024-02-05T04:53:43.784 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.784 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.784 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:53:43.784 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.191630+0000", 2024-02-05T04:53:43.784 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.8774999999999999e-05 2024-02-05T04:53:43.784 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.784 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.784 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:43.784 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.192049+0000", 2024-02-05T04:53:43.784 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00041888 2024-02-05T04:53:43.784 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.784 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.784 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:43.784 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.194188+0000", 2024-02-05T04:53:43.785 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0021391359999999998 2024-02-05T04:53:43.785 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.785 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.785 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-02-05T04:53:43.785 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.198284+0000", 2024-02-05T04:53:43.785 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0040960509999999999 2024-02-05T04:53:43.785 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.785 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.785 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:43.785 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.198341+0000", 2024-02-05T04:53:43.785 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.6737999999999998e-05 2024-02-05T04:53:43.785 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:43.785 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:43.785 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:43.785 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.786 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.786 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.2s1 37/31 ECSubWrite(tid=19, reqid=client.4634.0:251, at_version=37'11, trim_to=0'0, roll_forward_to=34'8))", 2024-02-05T04:53:43.786 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:29.200609+0000", 2024-02-05T04:53:43.786 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 14.534157655, 2024-02-05T04:53:43.786 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0073060429999999999, 2024-02-05T04:53:43.786 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:43.786 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:53:43.786 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:43.786 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.786 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:43.786 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.200609+0000", 2024-02-05T04:53:43.786 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:43.786 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.786 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.786 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:43.787 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.200609+0000", 2024-02-05T04:53:43.787 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:43.787 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.787 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.787 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:43.787 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.200609+0000", 2024-02-05T04:53:43.787 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.9599999999999999e-07 2024-02-05T04:53:43.787 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.787 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.787 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:43.787 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.200622+0000", 2024-02-05T04:53:43.787 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2833000000000001e-05 2024-02-05T04:53:43.787 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.787 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.787 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:43.787 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.200633+0000", 2024-02-05T04:53:43.788 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0509999999999999e-05 2024-02-05T04:53:43.788 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.788 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.788 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:43.788 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.200640+0000", 2024-02-05T04:53:43.788 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.2150000000000004e-06 2024-02-05T04:53:43.788 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.788 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.788 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:43.788 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.200685+0000", 2024-02-05T04:53:43.788 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.5571000000000001e-05 2024-02-05T04:53:43.788 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.788 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.788 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:43.788 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.200691+0000", 2024-02-05T04:53:43.789 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.5990000000000001e-06 2024-02-05T04:53:43.789 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.789 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.789 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:43.789 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.207875+0000", 2024-02-05T04:53:43.789 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0071843130000000003 2024-02-05T04:53:43.789 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.789 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.789 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:43.789 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.207915+0000", 2024-02-05T04:53:43.789 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.9505999999999997e-05 2024-02-05T04:53:43.789 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:43.789 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:43.789 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:43.789 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.790 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.790 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.2s1 37/31 ECSubWrite(tid=20, reqid=client.4634.0:252, at_version=37'12, trim_to=0'0, roll_forward_to=34'8))", 2024-02-05T04:53:43.790 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:29.201260+0000", 2024-02-05T04:53:43.790 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 14.533506643999999, 2024-02-05T04:53:43.790 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0066905130000000004, 2024-02-05T04:53:43.790 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:43.790 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:53:43.790 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:43.790 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.790 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:43.790 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.201260+0000", 2024-02-05T04:53:43.790 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:43.790 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.790 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.790 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:43.790 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.201260+0000", 2024-02-05T04:53:43.791 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:43.791 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.791 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.791 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:43.791 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.201260+0000", 2024-02-05T04:53:43.791 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.2500000000000005e-07 2024-02-05T04:53:43.791 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.791 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.791 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:43.791 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.201272+0000", 2024-02-05T04:53:43.791 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1105e-05 2024-02-05T04:53:43.791 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.791 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.791 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:43.791 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.201282+0000", 2024-02-05T04:53:43.792 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0572e-05 2024-02-05T04:53:43.792 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.792 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.792 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:43.792 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.201288+0000", 2024-02-05T04:53:43.792 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.9699999999999996e-06 2024-02-05T04:53:43.792 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.792 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.792 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:43.792 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.202134+0000", 2024-02-05T04:53:43.792 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00084635299999999999 2024-02-05T04:53:43.792 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.792 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.792 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:43.792 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.202140+0000", 2024-02-05T04:53:43.793 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.054e-06 2024-02-05T04:53:43.793 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.793 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.793 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:43.793 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.207923+0000", 2024-02-05T04:53:43.793 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0057820989999999997 2024-02-05T04:53:43.793 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.793 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.793 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:43.793 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.207950+0000", 2024-02-05T04:53:43.793 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.7735000000000001e-05 2024-02-05T04:53:43.793 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:43.793 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:43.793 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:43.793 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.794 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.794 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.4634.0:382 3.es0 3:7f0ef6bd:::smithi19639362-19:head [append 0~655360] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-02-05T04:53:43.794 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:34.167434+0000", 2024-02-05T04:53:43.794 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 9.5673324710000003, 2024-02-05T04:53:43.794 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0077709989999999998, 2024-02-05T04:53:43.794 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:43.794 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:43.794 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-02-05T04:53:43.794 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.4634", 2024-02-05T04:53:43.794 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:43.794 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 382 2024-02-05T04:53:43.794 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.794 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:43.794 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.794 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:43.795 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.167434+0000", 2024-02-05T04:53:43.795 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:43.795 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.795 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.795 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:43.795 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.167434+0000", 2024-02-05T04:53:43.795 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:43.795 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.795 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.795 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:43.795 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.167436+0000", 2024-02-05T04:53:43.795 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.885e-06 2024-02-05T04:53:43.795 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.795 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.795 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:43.795 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.168595+0000", 2024-02-05T04:53:43.796 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0011590350000000001 2024-02-05T04:53:43.796 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.796 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.796 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:43.796 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.168610+0000", 2024-02-05T04:53:43.796 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.4778e-05 2024-02-05T04:53:43.796 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.796 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.796 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:43.796 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.168700+0000", 2024-02-05T04:53:43.796 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.0462000000000002e-05 2024-02-05T04:53:43.796 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.796 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.796 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:43.796 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.168791+0000", 2024-02-05T04:53:43.797 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.1015999999999997e-05 2024-02-05T04:53:43.797 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.797 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.797 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:53:43.797 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.168911+0000", 2024-02-05T04:53:43.797 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000119864 2024-02-05T04:53:43.797 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.797 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.797 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:43.797 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.170527+0000", 2024-02-05T04:53:43.797 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0016163589999999999 2024-02-05T04:53:43.797 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.797 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.797 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:43.797 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.174753+0000", 2024-02-05T04:53:43.798 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0042254429999999997 2024-02-05T04:53:43.798 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.798 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.798 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-02-05T04:53:43.798 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.175115+0000", 2024-02-05T04:53:43.798 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00036262499999999999 2024-02-05T04:53:43.798 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.798 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.798 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:43.798 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.175205+0000", 2024-02-05T04:53:43.798 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.9531999999999996e-05 2024-02-05T04:53:43.798 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:43.798 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:43.798 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:43.798 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.799 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.799 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.4634.0:384 3.es0 3:7f0ef6bd:::smithi19639362-19:head [append 0~131072] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-02-05T04:53:43.799 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:34.171647+0000", 2024-02-05T04:53:43.799 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 9.5631189380000006, 2024-02-05T04:53:43.799 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0054756500000000003, 2024-02-05T04:53:43.799 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:43.799 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:43.799 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-02-05T04:53:43.799 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.4634", 2024-02-05T04:53:43.799 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:43.799 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 384 2024-02-05T04:53:43.799 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.799 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:43.799 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.799 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:43.799 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.171647+0000", 2024-02-05T04:53:43.800 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:43.800 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.800 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.800 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:43.800 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.171647+0000", 2024-02-05T04:53:43.800 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:43.800 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.800 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.800 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:43.800 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.171649+0000", 2024-02-05T04:53:43.800 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0589999999999999e-06 2024-02-05T04:53:43.800 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.800 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.800 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:43.800 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.171860+0000", 2024-02-05T04:53:43.801 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000211471 2024-02-05T04:53:43.801 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.801 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.801 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:43.801 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.171870+0000", 2024-02-05T04:53:43.801 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.5750000000000007e-06 2024-02-05T04:53:43.801 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.801 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.801 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:43.801 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.171879+0000", 2024-02-05T04:53:43.801 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.7380000000000006e-06 2024-02-05T04:53:43.801 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.801 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.801 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:43.801 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.173576+0000", 2024-02-05T04:53:43.802 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.001696817 2024-02-05T04:53:43.802 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.802 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.802 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:53:43.802 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.173688+0000", 2024-02-05T04:53:43.802 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00011143 2024-02-05T04:53:43.802 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.802 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.802 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:43.802 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.174310+0000", 2024-02-05T04:53:43.802 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00062290999999999996 2024-02-05T04:53:43.802 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.802 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.802 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:43.802 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.175344+0000", 2024-02-05T04:53:43.803 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.001033747 2024-02-05T04:53:43.803 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.803 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.803 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-02-05T04:53:43.803 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.177056+0000", 2024-02-05T04:53:43.803 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0017118929999999999 2024-02-05T04:53:43.803 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.803 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.803 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:43.803 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.177123+0000", 2024-02-05T04:53:43.803 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.7009999999999997e-05 2024-02-05T04:53:43.803 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:43.803 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:43.803 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:43.803 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.804 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.804 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.4634.0:385 3.es0 3:7f0ef6bd:::smithi19639362-19:head [setxattr _header (56) in=7b] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-02-05T04:53:43.804 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:34.171897+0000", 2024-02-05T04:53:43.804 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 9.5628696939999998, 2024-02-05T04:53:43.804 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.005569114, 2024-02-05T04:53:43.804 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:43.804 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:43.804 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-02-05T04:53:43.804 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.4634", 2024-02-05T04:53:43.804 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:43.804 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 385 2024-02-05T04:53:43.804 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.804 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:43.804 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.804 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:43.805 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.171897+0000", 2024-02-05T04:53:43.805 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:43.805 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.805 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.805 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:43.805 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.171897+0000", 2024-02-05T04:53:43.805 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:43.805 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.805 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.805 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:43.805 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.171898+0000", 2024-02-05T04:53:43.805 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.6899999999999996e-07 2024-02-05T04:53:43.805 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.805 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.805 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:43.806 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.171901+0000", 2024-02-05T04:53:43.806 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.7699999999999999e-06 2024-02-05T04:53:43.806 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.806 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.806 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:43.806 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.171908+0000", 2024-02-05T04:53:43.806 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.1399999999999997e-06 2024-02-05T04:53:43.806 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.806 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.806 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:43.806 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.171913+0000", 2024-02-05T04:53:43.806 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.0509999999999996e-06 2024-02-05T04:53:43.806 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.806 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.806 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:43.807 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.175805+0000", 2024-02-05T04:53:43.807 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.003892674 2024-02-05T04:53:43.807 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.807 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.807 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:53:43.807 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.175908+0000", 2024-02-05T04:53:43.807 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000102518 2024-02-05T04:53:43.807 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.807 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.807 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:43.807 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.176312+0000", 2024-02-05T04:53:43.807 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00040448399999999999 2024-02-05T04:53:43.807 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.807 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.807 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:43.808 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.176961+0000", 2024-02-05T04:53:43.808 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00064888599999999999 2024-02-05T04:53:43.808 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.808 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.808 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-02-05T04:53:43.808 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.177414+0000", 2024-02-05T04:53:43.808 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000453292 2024-02-05T04:53:43.808 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.808 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.808 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:43.808 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.177466+0000", 2024-02-05T04:53:43.808 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.1430000000000001e-05 2024-02-05T04:53:43.808 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:43.808 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:43.808 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:43.809 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.809 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.809 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.4634.0:386 3.es0 3:7f0ef6bd:::smithi19639362-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-02-05T04:53:43.809 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:34.171921+0000", 2024-02-05T04:53:43.809 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 9.5628453869999994, 2024-02-05T04:53:43.809 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0068862569999999998, 2024-02-05T04:53:43.809 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:43.809 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "started", 2024-02-05T04:53:43.809 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-02-05T04:53:43.809 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.4634", 2024-02-05T04:53:43.809 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:43.809 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 386 2024-02-05T04:53:43.809 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.809 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:43.809 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.810 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:43.810 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.171921+0000", 2024-02-05T04:53:43.810 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:43.810 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.810 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.810 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:43.810 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.171921+0000", 2024-02-05T04:53:43.810 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:43.810 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.810 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.810 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:43.810 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.171922+0000", 2024-02-05T04:53:43.810 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.7399999999999998e-07 2024-02-05T04:53:43.810 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.810 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.810 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:43.811 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.171924+0000", 2024-02-05T04:53:43.811 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.1900000000000002e-06 2024-02-05T04:53:43.811 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.811 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.811 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:43.811 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.171930+0000", 2024-02-05T04:53:43.811 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.207e-06 2024-02-05T04:53:43.811 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.811 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.811 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:43.811 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.171936+0000", 2024-02-05T04:53:43.811 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.0630000000000002e-06 2024-02-05T04:53:43.811 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.811 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.811 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:43.811 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.176614+0000", 2024-02-05T04:53:43.812 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0046776270000000002 2024-02-05T04:53:43.812 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.812 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.812 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for rw locks", 2024-02-05T04:53:43.812 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.176696+0000", 2024-02-05T04:53:43.812 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.2084999999999998e-05 2024-02-05T04:53:43.812 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.812 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.812 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:43.812 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.177767+0000", 2024-02-05T04:53:43.812 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.001070809 2024-02-05T04:53:43.812 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.812 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.812 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:53:43.812 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.177813+0000", 2024-02-05T04:53:43.813 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.6635999999999999e-05 2024-02-05T04:53:43.813 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.813 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.813 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:43.813 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.178807+0000", 2024-02-05T04:53:43.813 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0009941660000000001 2024-02-05T04:53:43.813 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:43.813 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:43.813 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:43.813 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.813 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.813 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.8s2 43/31 ECSubWrite(tid=64, reqid=osd.4.0:62, at_version=0'0, trim_to=0'0, roll_forward_to=43'19))", 2024-02-05T04:53:43.813 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:35.181070+0000", 2024-02-05T04:53:43.813 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 8.5536959889999995, 2024-02-05T04:53:43.813 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0062151699999999999, 2024-02-05T04:53:43.813 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:43.814 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:53:43.814 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:43.814 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.814 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:43.814 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.181070+0000", 2024-02-05T04:53:43.814 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:43.814 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.814 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.814 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:43.814 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.181070+0000", 2024-02-05T04:53:43.814 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:43.814 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.814 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.814 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:43.814 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.181071+0000", 2024-02-05T04:53:43.814 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.85e-07 2024-02-05T04:53:43.815 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.815 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.815 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:43.815 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.181080+0000", 2024-02-05T04:53:43.815 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.7120000000000006e-06 2024-02-05T04:53:43.815 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.815 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.815 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:43.815 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.181093+0000", 2024-02-05T04:53:43.815 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2938e-05 2024-02-05T04:53:43.815 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.815 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.815 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:43.815 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.181101+0000", 2024-02-05T04:53:43.815 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.1629999999999996e-06 2024-02-05T04:53:43.816 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.816 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.816 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:43.816 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.181211+0000", 2024-02-05T04:53:43.816 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00010961800000000001 2024-02-05T04:53:43.816 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.816 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.816 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:43.816 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.181220+0000", 2024-02-05T04:53:43.816 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.7059999999999999e-06 2024-02-05T04:53:43.816 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.816 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.816 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:43.816 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.187235+0000", 2024-02-05T04:53:43.816 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0060150100000000003 2024-02-05T04:53:43.817 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.817 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.817 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:43.817 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.187286+0000", 2024-02-05T04:53:43.817 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.0238000000000002e-05 2024-02-05T04:53:43.817 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:43.817 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:43.817 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:43.817 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.817 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.817 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.1s5 43/31 ECSubWrite(tid=140, reqid=osd.13.0:140, at_version=43'29, trim_to=0'0, roll_forward_to=42'27))", 2024-02-05T04:53:43.817 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:35.181918+0000", 2024-02-05T04:53:43.817 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 8.5528489279999995, 2024-02-05T04:53:43.817 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0053704460000000001, 2024-02-05T04:53:43.817 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:43.818 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:53:43.818 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:43.818 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.818 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:43.818 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.181918+0000", 2024-02-05T04:53:43.818 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:43.818 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.818 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.818 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:43.818 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.181918+0000", 2024-02-05T04:53:43.818 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:43.818 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.818 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.818 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:43.818 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.181918+0000", 2024-02-05T04:53:43.818 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.4000000000000001e-07 2024-02-05T04:53:43.819 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.819 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.819 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:43.819 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.181934+0000", 2024-02-05T04:53:43.819 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.6229e-05 2024-02-05T04:53:43.819 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.819 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.819 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:43.819 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.181947+0000", 2024-02-05T04:53:43.819 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.204e-05 2024-02-05T04:53:43.819 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.819 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.819 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:43.819 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.181955+0000", 2024-02-05T04:53:43.819 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.6489999999999997e-06 2024-02-05T04:53:43.820 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.820 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.820 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:43.820 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.182021+0000", 2024-02-05T04:53:43.820 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.6298999999999997e-05 2024-02-05T04:53:43.820 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.820 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.820 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:43.820 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.182032+0000", 2024-02-05T04:53:43.820 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0139999999999999e-05 2024-02-05T04:53:43.820 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.820 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.820 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:43.820 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.187253+0000", 2024-02-05T04:53:43.820 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0052212109999999999 2024-02-05T04:53:43.820 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.821 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.821 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:43.821 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.187288+0000", 2024-02-05T04:53:43.821 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.5138000000000001e-05 2024-02-05T04:53:43.821 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:43.821 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:43.821 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:43.821 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.821 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.821 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.4s1 43/31 ECSubWrite(tid=46, reqid=osd.1.0:46, at_version=43'29, trim_to=0'0, roll_forward_to=42'27))", 2024-02-05T04:53:43.821 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:35.182368+0000", 2024-02-05T04:53:43.821 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 8.552398234, 2024-02-05T04:53:43.821 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0055476830000000003, 2024-02-05T04:53:43.821 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:43.821 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:53:43.822 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:43.822 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.822 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:43.822 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.182368+0000", 2024-02-05T04:53:43.822 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:43.822 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.822 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.822 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:43.822 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.182368+0000", 2024-02-05T04:53:43.822 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:43.822 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.822 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.822 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:43.822 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.182368+0000", 2024-02-05T04:53:43.822 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.1799999999999999e-07 2024-02-05T04:53:43.822 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.823 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.823 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:43.823 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.182390+0000", 2024-02-05T04:53:43.823 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.1206999999999999e-05 2024-02-05T04:53:43.823 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.823 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.823 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:43.823 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.182401+0000", 2024-02-05T04:53:43.823 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1674e-05 2024-02-05T04:53:43.823 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.823 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.823 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:43.823 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.182410+0000", 2024-02-05T04:53:43.823 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.7460000000000006e-06 2024-02-05T04:53:43.823 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.823 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.824 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:43.824 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.182472+0000", 2024-02-05T04:53:43.824 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.1871000000000004e-05 2024-02-05T04:53:43.824 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.824 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.824 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:43.824 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.182483+0000", 2024-02-05T04:53:43.824 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0787e-05 2024-02-05T04:53:43.824 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.824 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.824 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:43.824 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.187876+0000", 2024-02-05T04:53:43.824 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0053932800000000003 2024-02-05T04:53:43.824 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:43.824 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:43.825 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:43.825 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.187916+0000", 2024-02-05T04:53:43.825 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.9900000000000001e-05 2024-02-05T04:53:43.825 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:43.825 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:43.825 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:43.825 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:43.825 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:43.825 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-02-05T04:53:43.827 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-02-05T04:53:43.828 INFO:tasks.ceph.osd.14.smithi150.stderr:2024-02-05T04:53:43.825+0000 7f5f81083640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 34020) UID: 0 2024-02-05T04:53:43.927 INFO:tasks.ceph.osd.5.smithi091.stderr:2024-02-05T04:53:43.926+0000 7f63983b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34277) UID: 0 2024-02-05T04:53:43.951 INFO:tasks.rados.rados.0.smithi196.stdout:189: done (2 left) 2024-02-05T04:53:43.951 INFO:tasks.rados.rados.0.smithi196.stdout:append oid 27 current snap is 11 2024-02-05T04:53:43.952 INFO:tasks.rados.rados.0.smithi196.stdout:190: seq_num 99 ranges {1998848=507904,2506752=606208,3112960=557056} 2024-02-05T04:53:43.961 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-02-05T04:53:43.966 INFO:tasks.rados.rados.0.smithi196.stdout:190: writing smithi19639362-27 from 1998848 to 2506752 tid 1 2024-02-05T04:53:43.969 INFO:tasks.rados.rados.0.smithi196.stdout:190: writing smithi19639362-27 from 2506752 to 3112960 tid 2 2024-02-05T04:53:43.973 INFO:tasks.rados.rados.0.smithi196.stdout:190: writing smithi19639362-27 from 3112960 to 3670016 tid 3 2024-02-05T04:53:43.973 INFO:tasks.rados.rados.0.smithi196.stdout:191: snap_create 2024-02-05T04:53:43.973 INFO:tasks.rados.rados.0.smithi196.stdout:187: expect (ObjNum 72 snap 3 seq_num 72) 2024-02-05T04:53:43.981 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-02-05T04:53:43.981 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [], 2024-02-05T04:53:43.981 INFO:teuthology.orchestra.run.smithi150.stdout: "num_ops": 0 2024-02-05T04:53:43.981 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-02-05T04:53:43.990 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-02-05T04:53:44.006 INFO:tasks.rados.rados.0.smithi196.stdout:185: finishing copy_from to smithi19639362-31 2024-02-05T04:53:44.006 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 31 v 4 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-02-05T04:53:44.006 INFO:tasks.rados.rados.0.smithi196.stdout:190: finishing write tid 1 to smithi19639362-27 2024-02-05T04:53:44.006 INFO:tasks.rados.rados.0.smithi196.stdout:190: finishing write tid 2 to smithi19639362-27 2024-02-05T04:53:44.007 INFO:tasks.rados.rados.0.smithi196.stdout:190: finishing write tid 3 to smithi19639362-27 2024-02-05T04:53:44.007 INFO:tasks.rados.rados.0.smithi196.stdout:190: finishing write tid 4 to smithi19639362-27 2024-02-05T04:53:44.007 INFO:tasks.rados.rados.0.smithi196.stdout:190: finishing write tid 5 to smithi19639362-27 2024-02-05T04:53:44.007 INFO:tasks.rados.rados.0.smithi196.stdout:190: oid 27 updating version 0 to 22 2024-02-05T04:53:44.007 INFO:tasks.rados.rados.0.smithi196.stdout:190: oid 27 updating version 22 to 24 2024-02-05T04:53:44.007 INFO:tasks.rados.rados.0.smithi196.stdout:190: oid 27 version 24 is already newer than 23 2024-02-05T04:53:44.007 INFO:tasks.rados.rados.0.smithi196.stdout:190: oid 27 updating version 24 to 25 2024-02-05T04:53:44.007 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 27 v 25 (ObjNum 99 snap 11 seq_num 99) dirty exists 2024-02-05T04:53:44.007 INFO:tasks.rados.rados.0.smithi196.stdout:190: left oid 27 (ObjNum 99 snap 11 seq_num 99) 2024-02-05T04:53:44.034 INFO:tasks.ceph.osd.14.smithi150.stderr:2024-02-05T04:53:44.033+0000 7f5f81083640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 34020) UID: 0 2024-02-05T04:53:44.096 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-02-05T04:53:44.096 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [], 2024-02-05T04:53:44.097 INFO:teuthology.orchestra.run.smithi150.stdout: "complaint_time": 30, 2024-02-05T04:53:44.097 INFO:teuthology.orchestra.run.smithi150.stdout: "num_blocked_ops": 0 2024-02-05T04:53:44.097 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-02-05T04:53:44.105 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-02-05T04:53:44.129 INFO:tasks.ceph.osd.4.smithi062.stderr:2024-02-05T04:53:44.129+0000 7f708c308640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34689) UID: 0 2024-02-05T04:53:44.230 INFO:tasks.ceph.osd.2.smithi150.stderr:2024-02-05T04:53:44.229+0000 7f7337bf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 34014) UID: 0 2024-02-05T04:53:44.253 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-02-05T04:53:44.253 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 20, 2024-02-05T04:53:44.253 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 600, 2024-02-05T04:53:44.254 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [ 2024-02-05T04:53:44.254 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.254 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.es5 37/31 ECSubWrite(tid=20, reqid=client.4634.0:245, at_version=37'13, trim_to=0'0, roll_forward_to=34'10))", 2024-02-05T04:53:44.254 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:29.189558+0000", 2024-02-05T04:53:44.254 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 15.061757375999999, 2024-02-05T04:53:44.254 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0030151230000000002, 2024-02-05T04:53:44.254 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:44.254 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:53:44.254 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:44.254 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.254 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:44.254 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.189558+0000", 2024-02-05T04:53:44.254 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:44.254 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.255 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.255 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:44.255 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.189558+0000", 2024-02-05T04:53:44.255 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:44.255 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.255 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.255 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:44.255 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.189559+0000", 2024-02-05T04:53:44.255 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.5600000000000004e-07 2024-02-05T04:53:44.255 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.255 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.255 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:44.255 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.189581+0000", 2024-02-05T04:53:44.255 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.2469000000000001e-05 2024-02-05T04:53:44.255 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.255 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.256 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:44.256 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.189597+0000", 2024-02-05T04:53:44.256 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.5926e-05 2024-02-05T04:53:44.256 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.256 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.256 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:44.256 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.189608+0000", 2024-02-05T04:53:44.256 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0210999999999999e-05 2024-02-05T04:53:44.256 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.256 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.256 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:44.256 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.189682+0000", 2024-02-05T04:53:44.256 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.4025999999999999e-05 2024-02-05T04:53:44.256 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.256 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.257 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:44.257 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.189691+0000", 2024-02-05T04:53:44.257 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.5529999999999998e-06 2024-02-05T04:53:44.257 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.257 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.257 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:44.257 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.192545+0000", 2024-02-05T04:53:44.257 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0028542279999999999 2024-02-05T04:53:44.257 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.257 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.257 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:44.257 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.192573+0000", 2024-02-05T04:53:44.257 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.7753999999999999e-05 2024-02-05T04:53:44.257 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.257 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:44.257 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.257 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.258 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.258 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.2s3 37/31 ECSubWrite(tid=22, reqid=client.4634.0:254, at_version=37'15, trim_to=0'0, roll_forward_to=37'10))", 2024-02-05T04:53:44.258 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:29.208543+0000", 2024-02-05T04:53:44.258 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 15.042772009, 2024-02-05T04:53:44.258 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0047425410000000003, 2024-02-05T04:53:44.258 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:44.258 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:53:44.258 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:44.258 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.258 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:44.258 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.208543+0000", 2024-02-05T04:53:44.258 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:44.258 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.258 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.258 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:44.259 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.208543+0000", 2024-02-05T04:53:44.259 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:44.259 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.259 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.259 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:44.259 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.208544+0000", 2024-02-05T04:53:44.259 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.6700000000000003e-07 2024-02-05T04:53:44.259 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.259 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.259 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:44.259 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.208803+0000", 2024-02-05T04:53:44.259 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00025897499999999999 2024-02-05T04:53:44.259 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.259 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.259 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:44.259 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.208815+0000", 2024-02-05T04:53:44.260 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1630999999999999e-05 2024-02-05T04:53:44.260 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.260 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.260 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:44.260 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.208821+0000", 2024-02-05T04:53:44.260 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.331e-06 2024-02-05T04:53:44.260 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.260 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.260 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:44.260 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.208875+0000", 2024-02-05T04:53:44.260 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.3909999999999997e-05 2024-02-05T04:53:44.260 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.260 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.260 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:44.260 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.208884+0000", 2024-02-05T04:53:44.261 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.4169999999999998e-06 2024-02-05T04:53:44.261 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.261 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.261 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:44.261 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.213250+0000", 2024-02-05T04:53:44.261 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0043660360000000002 2024-02-05T04:53:44.261 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.261 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.261 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:44.261 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.213286+0000", 2024-02-05T04:53:44.261 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.5673999999999998e-05 2024-02-05T04:53:44.261 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.261 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:44.261 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.261 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.261 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.261 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.2s3 37/31 ECSubWrite(tid=23, reqid=client.4634.0:255, at_version=37'16, trim_to=0'0, roll_forward_to=37'10))", 2024-02-05T04:53:44.262 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:29.209472+0000", 2024-02-05T04:53:44.262 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 15.041843627, 2024-02-05T04:53:44.262 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0038365420000000001, 2024-02-05T04:53:44.262 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:44.262 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:53:44.262 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:44.262 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.262 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:44.262 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.209472+0000", 2024-02-05T04:53:44.262 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:44.262 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.262 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.262 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:44.262 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.209472+0000", 2024-02-05T04:53:44.262 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:44.263 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.263 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.263 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:44.263 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.209472+0000", 2024-02-05T04:53:44.263 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.4700000000000001e-07 2024-02-05T04:53:44.263 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.263 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.263 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:44.263 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.209484+0000", 2024-02-05T04:53:44.263 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2174000000000001e-05 2024-02-05T04:53:44.263 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.263 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.263 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:44.264 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.209494+0000", 2024-02-05T04:53:44.264 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.1309999999999995e-06 2024-02-05T04:53:44.264 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.264 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.264 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:44.264 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.209499+0000", 2024-02-05T04:53:44.264 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.8370000000000002e-06 2024-02-05T04:53:44.264 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.264 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.264 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:44.264 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.209557+0000", 2024-02-05T04:53:44.264 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.7207000000000003e-05 2024-02-05T04:53:44.264 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.264 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.265 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:44.265 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.209567+0000", 2024-02-05T04:53:44.265 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0424999999999999e-05 2024-02-05T04:53:44.265 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.265 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.265 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:44.265 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.213291+0000", 2024-02-05T04:53:44.265 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0037243939999999998 2024-02-05T04:53:44.265 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.265 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.265 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:44.265 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.213308+0000", 2024-02-05T04:53:44.265 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.6827e-05 2024-02-05T04:53:44.265 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.265 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:44.265 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.266 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.266 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.266 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.2s3 37/31 ECSubWrite(tid=24, reqid=client.4634.0:257, at_version=37'18, trim_to=0'0, roll_forward_to=37'12))", 2024-02-05T04:53:44.266 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:29.211345+0000", 2024-02-05T04:53:44.266 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 15.039970666, 2024-02-05T04:53:44.266 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0029126759999999999, 2024-02-05T04:53:44.266 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:44.266 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:53:44.266 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:44.266 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.266 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:44.266 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.211345+0000", 2024-02-05T04:53:44.266 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:44.267 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.267 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.267 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:44.267 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.211345+0000", 2024-02-05T04:53:44.267 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:44.267 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.267 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.267 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:44.267 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.211346+0000", 2024-02-05T04:53:44.267 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.0599999999999999e-07 2024-02-05T04:53:44.267 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.267 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.267 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:44.267 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.211369+0000", 2024-02-05T04:53:44.267 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.3587e-05 2024-02-05T04:53:44.268 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.268 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.268 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:44.268 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.211384+0000", 2024-02-05T04:53:44.268 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.5189e-05 2024-02-05T04:53:44.268 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.268 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.268 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:44.268 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.211394+0000", 2024-02-05T04:53:44.268 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.3870000000000001e-06 2024-02-05T04:53:44.268 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.268 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.268 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:44.268 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.211449+0000", 2024-02-05T04:53:44.269 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.4892000000000002e-05 2024-02-05T04:53:44.269 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.269 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.269 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:44.269 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.211456+0000", 2024-02-05T04:53:44.269 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.3869999999999997e-06 2024-02-05T04:53:44.269 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.269 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.269 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:44.269 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.214219+0000", 2024-02-05T04:53:44.269 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0027630150000000002 2024-02-05T04:53:44.269 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.269 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.269 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:44.270 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.214257+0000", 2024-02-05T04:53:44.270 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.8413e-05 2024-02-05T04:53:44.270 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.270 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:44.270 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.270 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.270 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.270 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.es5 38/31 ECSubWrite(tid=25, reqid=osd.2.0:25, at_version=38'20, trim_to=0'0, roll_forward_to=37'16))", 2024-02-05T04:53:44.270 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:30.134352+0000", 2024-02-05T04:53:44.270 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 14.116963075999999, 2024-02-05T04:53:44.270 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.003909442, 2024-02-05T04:53:44.270 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:44.270 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:53:44.270 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:44.271 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.271 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:44.271 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.134352+0000", 2024-02-05T04:53:44.271 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:44.271 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.271 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.271 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:44.271 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.134352+0000", 2024-02-05T04:53:44.271 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:44.271 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.271 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.271 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:44.271 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.134353+0000", 2024-02-05T04:53:44.271 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.7899999999999998e-07 2024-02-05T04:53:44.272 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.272 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.272 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:44.272 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.134381+0000", 2024-02-05T04:53:44.272 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.8187999999999998e-05 2024-02-05T04:53:44.272 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.272 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.272 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:44.272 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.134391+0000", 2024-02-05T04:53:44.272 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.6809999999999993e-06 2024-02-05T04:53:44.272 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.272 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.272 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:44.272 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.134398+0000", 2024-02-05T04:53:44.272 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.391e-06 2024-02-05T04:53:44.273 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.273 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.273 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:44.273 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.134469+0000", 2024-02-05T04:53:44.273 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.1200999999999998e-05 2024-02-05T04:53:44.273 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.273 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.273 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:44.273 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.134478+0000", 2024-02-05T04:53:44.273 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.4060000000000005e-06 2024-02-05T04:53:44.273 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.273 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.273 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:44.273 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.138231+0000", 2024-02-05T04:53:44.274 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0037529099999999999 2024-02-05T04:53:44.274 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.274 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.274 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:44.274 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.138262+0000", 2024-02-05T04:53:44.274 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.1086e-05 2024-02-05T04:53:44.274 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.274 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:44.274 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.274 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.274 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.274 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.2s3 38/31 ECSubWrite(tid=27, reqid=osd.12.0:27, at_version=38'20, trim_to=0'0, roll_forward_to=37'18))", 2024-02-05T04:53:44.274 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:30.143537+0000", 2024-02-05T04:53:44.274 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 14.107778193, 2024-02-05T04:53:44.274 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.002926611, 2024-02-05T04:53:44.275 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:44.275 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:53:44.275 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:44.275 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.275 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:44.275 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.143537+0000", 2024-02-05T04:53:44.275 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:44.275 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.275 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.275 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:44.275 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.143537+0000", 2024-02-05T04:53:44.275 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:44.275 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.275 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.275 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:44.276 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.143537+0000", 2024-02-05T04:53:44.276 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.3300000000000001e-07 2024-02-05T04:53:44.276 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.276 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.276 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:44.276 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.143555+0000", 2024-02-05T04:53:44.276 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.8026e-05 2024-02-05T04:53:44.276 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.276 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.276 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:44.276 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.143587+0000", 2024-02-05T04:53:44.276 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.1616999999999999e-05 2024-02-05T04:53:44.276 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.276 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.277 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:44.277 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.143599+0000", 2024-02-05T04:53:44.277 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2213e-05 2024-02-05T04:53:44.277 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.277 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.277 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:44.277 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.143722+0000", 2024-02-05T04:53:44.277 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00012242500000000001 2024-02-05T04:53:44.277 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.277 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.277 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:44.277 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.143732+0000", 2024-02-05T04:53:44.277 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0523000000000001e-05 2024-02-05T04:53:44.277 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.277 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.278 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:44.278 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.146438+0000", 2024-02-05T04:53:44.278 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.002705818 2024-02-05T04:53:44.278 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.278 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.278 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:44.278 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.146464+0000", 2024-02-05T04:53:44.278 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.5755999999999999e-05 2024-02-05T04:53:44.278 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.278 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:44.278 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.278 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.278 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.278 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.2s3 38/31 ECSubWrite(tid=28, reqid=osd.12.0:28, at_version=38'22, trim_to=0'0, roll_forward_to=37'18))", 2024-02-05T04:53:44.279 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:30.144203+0000", 2024-02-05T04:53:44.279 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 14.107112688000001, 2024-02-05T04:53:44.279 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0033111519999999999, 2024-02-05T04:53:44.279 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:44.279 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:53:44.279 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:44.279 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.279 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:44.279 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.144203+0000", 2024-02-05T04:53:44.279 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:44.279 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.279 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.279 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:44.279 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.144203+0000", 2024-02-05T04:53:44.279 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:44.279 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.280 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.280 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:44.280 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.144203+0000", 2024-02-05T04:53:44.280 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.6300000000000005e-07 2024-02-05T04:53:44.280 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.280 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.280 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:44.280 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.144235+0000", 2024-02-05T04:53:44.280 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.1686999999999997e-05 2024-02-05T04:53:44.280 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.280 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.280 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:44.280 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.144250+0000", 2024-02-05T04:53:44.280 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.4825e-05 2024-02-05T04:53:44.281 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.281 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.281 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:44.281 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.144266+0000", 2024-02-05T04:53:44.281 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.6123000000000001e-05 2024-02-05T04:53:44.281 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.281 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.281 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:44.281 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.144323+0000", 2024-02-05T04:53:44.281 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.7071000000000003e-05 2024-02-05T04:53:44.281 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.281 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.281 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:44.281 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.144333+0000", 2024-02-05T04:53:44.281 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0237e-05 2024-02-05T04:53:44.281 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.282 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.282 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:44.282 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.147486+0000", 2024-02-05T04:53:44.282 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0031526800000000002 2024-02-05T04:53:44.282 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.282 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.282 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:44.282 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.147514+0000", 2024-02-05T04:53:44.282 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.7866000000000001e-05 2024-02-05T04:53:44.282 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.282 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:44.282 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.282 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.283 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.283 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.8s6 42/31 ECSubWrite(tid=60, reqid=client.4634.0:390, at_version=42'17, trim_to=0'0, roll_forward_to=40'15))", 2024-02-05T04:53:44.283 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:34.180404+0000", 2024-02-05T04:53:44.283 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 10.07091185, 2024-02-05T04:53:44.283 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0025825100000000001, 2024-02-05T04:53:44.283 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:44.283 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:53:44.283 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:44.283 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.283 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:44.283 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.180404+0000", 2024-02-05T04:53:44.283 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:44.283 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.283 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.284 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:44.284 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.180404+0000", 2024-02-05T04:53:44.284 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:44.284 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.284 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.284 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:44.284 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.180404+0000", 2024-02-05T04:53:44.284 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.0299999999999999e-07 2024-02-05T04:53:44.284 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.284 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.284 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:44.284 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.181307+0000", 2024-02-05T04:53:44.284 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00090246300000000005 2024-02-05T04:53:44.284 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.285 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.285 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:44.285 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.181323+0000", 2024-02-05T04:53:44.285 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.6697999999999998e-05 2024-02-05T04:53:44.285 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.285 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.285 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:44.285 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.181333+0000", 2024-02-05T04:53:44.285 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.73e-06 2024-02-05T04:53:44.285 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.285 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.285 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:44.285 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.181402+0000", 2024-02-05T04:53:44.285 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.9411000000000003e-05 2024-02-05T04:53:44.285 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.286 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.286 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:44.286 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.181409+0000", 2024-02-05T04:53:44.286 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.567e-06 2024-02-05T04:53:44.286 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.286 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.286 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:44.286 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.182944+0000", 2024-02-05T04:53:44.286 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0015353420000000001 2024-02-05T04:53:44.286 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.286 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.286 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:44.286 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.182986+0000", 2024-02-05T04:53:44.286 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.1795999999999998e-05 2024-02-05T04:53:44.286 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.287 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:44.287 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.287 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.287 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.287 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.es5 43/31 ECSubWrite(tid=40, reqid=osd.2.0:40, at_version=43'34, trim_to=0'0, roll_forward_to=42'30))", 2024-02-05T04:53:44.287 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:35.169822+0000", 2024-02-05T04:53:44.287 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 9.0814938620000003, 2024-02-05T04:53:44.287 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.003348405, 2024-02-05T04:53:44.287 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:44.287 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:53:44.287 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:44.287 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.287 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:44.287 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.169822+0000", 2024-02-05T04:53:44.287 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:44.288 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.288 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.288 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:44.288 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.169822+0000", 2024-02-05T04:53:44.288 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:44.288 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.288 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.288 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:44.288 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.169822+0000", 2024-02-05T04:53:44.288 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.0800000000000004e-07 2024-02-05T04:53:44.288 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.288 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.288 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:44.288 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.169849+0000", 2024-02-05T04:53:44.289 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.6611999999999999e-05 2024-02-05T04:53:44.289 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.289 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.289 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:44.289 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.169861+0000", 2024-02-05T04:53:44.289 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1712000000000001e-05 2024-02-05T04:53:44.289 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.289 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.289 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:44.289 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.169867+0000", 2024-02-05T04:53:44.289 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.1850000000000001e-06 2024-02-05T04:53:44.289 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.289 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.289 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:44.289 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.169923+0000", 2024-02-05T04:53:44.290 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.6196999999999997e-05 2024-02-05T04:53:44.290 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.290 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.290 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:44.290 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.169933+0000", 2024-02-05T04:53:44.290 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0193999999999999e-05 2024-02-05T04:53:44.290 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.290 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.290 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:44.290 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.173126+0000", 2024-02-05T04:53:44.290 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0031932280000000002 2024-02-05T04:53:44.290 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.290 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.290 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:44.290 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.173170+0000", 2024-02-05T04:53:44.290 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.3469e-05 2024-02-05T04:53:44.290 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.291 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:44.291 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.291 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.291 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.291 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.2s3 43/31 ECSubWrite(tid=49, reqid=osd.12.0:48, at_version=0'0, trim_to=0'0, roll_forward_to=43'38))", 2024-02-05T04:53:44.291 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:35.169965+0000", 2024-02-05T04:53:44.291 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 9.0813499229999994, 2024-02-05T04:53:44.291 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0032122560000000001, 2024-02-05T04:53:44.291 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:44.291 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:53:44.291 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:44.291 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.291 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:44.291 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.169965+0000", 2024-02-05T04:53:44.291 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:44.291 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.292 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.292 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:44.292 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.169965+0000", 2024-02-05T04:53:44.292 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:44.292 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.292 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.292 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:44.292 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.169966+0000", 2024-02-05T04:53:44.292 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.6599999999999994e-07 2024-02-05T04:53:44.292 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.292 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.292 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:44.292 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.169979+0000", 2024-02-05T04:53:44.292 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2648e-05 2024-02-05T04:53:44.293 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.293 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.293 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:44.293 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.170011+0000", 2024-02-05T04:53:44.293 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.2299999999999999e-05 2024-02-05T04:53:44.293 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.293 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.293 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:44.293 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.170021+0000", 2024-02-05T04:53:44.293 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.6209999999999999e-06 2024-02-05T04:53:44.293 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.293 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.293 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:44.294 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.170089+0000", 2024-02-05T04:53:44.294 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.8356999999999996e-05 2024-02-05T04:53:44.294 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.294 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.294 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:44.294 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.170099+0000", 2024-02-05T04:53:44.294 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.007e-05 2024-02-05T04:53:44.294 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.294 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.294 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:44.294 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.173147+0000", 2024-02-05T04:53:44.294 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0030477170000000001 2024-02-05T04:53:44.294 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.294 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.294 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:44.295 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.173178+0000", 2024-02-05T04:53:44.295 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.0577000000000002e-05 2024-02-05T04:53:44.295 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.295 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:44.295 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.295 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.295 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.295 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.bs4 43/31 ECSubWrite(tid=137, reqid=osd.13.0:137, at_version=43'28, trim_to=0'0, roll_forward_to=41'26))", 2024-02-05T04:53:44.295 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:35.171683+0000", 2024-02-05T04:53:44.295 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 9.0796320210000001, 2024-02-05T04:53:44.295 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0045005519999999997, 2024-02-05T04:53:44.295 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:44.295 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:53:44.295 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:44.295 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.295 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:44.296 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.171683+0000", 2024-02-05T04:53:44.296 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:44.296 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.296 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.296 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:44.296 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.171683+0000", 2024-02-05T04:53:44.296 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:44.296 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.296 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.296 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:44.296 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.171683+0000", 2024-02-05T04:53:44.296 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.15e-07 2024-02-05T04:53:44.296 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.296 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.296 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:44.297 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.171700+0000", 2024-02-05T04:53:44.297 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.6855999999999999e-05 2024-02-05T04:53:44.297 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.297 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.297 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:44.297 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.171710+0000", 2024-02-05T04:53:44.297 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.3130000000000004e-06 2024-02-05T04:53:44.297 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.297 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.297 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:44.297 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.171716+0000", 2024-02-05T04:53:44.297 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.959e-06 2024-02-05T04:53:44.297 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.297 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.297 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:44.298 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.171766+0000", 2024-02-05T04:53:44.298 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.0550999999999998e-05 2024-02-05T04:53:44.298 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.298 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.298 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:44.298 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.171772+0000", 2024-02-05T04:53:44.298 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.2729999999999999e-06 2024-02-05T04:53:44.298 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.298 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.298 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:44.298 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.176147+0000", 2024-02-05T04:53:44.298 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0043745399999999997 2024-02-05T04:53:44.298 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.298 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.298 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:44.298 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.176184+0000", 2024-02-05T04:53:44.299 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.6944999999999999e-05 2024-02-05T04:53:44.299 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.299 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:44.299 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.299 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.299 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.299 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.8s6 43/31 ECSubWrite(tid=62, reqid=osd.4.0:62, at_version=43'19, trim_to=0'0, roll_forward_to=42'17))", 2024-02-05T04:53:44.299 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:35.172379+0000", 2024-02-05T04:53:44.299 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 9.0789366359999999, 2024-02-05T04:53:44.299 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0041952910000000003, 2024-02-05T04:53:44.299 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:44.299 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:53:44.299 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:44.299 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.299 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:44.299 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.172379+0000", 2024-02-05T04:53:44.299 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:44.300 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.300 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.300 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:44.300 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.172379+0000", 2024-02-05T04:53:44.300 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:44.300 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.300 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.300 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:44.300 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.172379+0000", 2024-02-05T04:53:44.300 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.5100000000000001e-07 2024-02-05T04:53:44.300 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.300 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.300 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:44.300 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.172404+0000", 2024-02-05T04:53:44.300 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.4725000000000001e-05 2024-02-05T04:53:44.301 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.301 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.301 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:44.301 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.172419+0000", 2024-02-05T04:53:44.301 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.5746000000000001e-05 2024-02-05T04:53:44.301 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.301 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.301 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:44.301 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.172430+0000", 2024-02-05T04:53:44.301 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0929e-05 2024-02-05T04:53:44.301 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.301 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.301 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:44.301 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.172513+0000", 2024-02-05T04:53:44.301 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.2513000000000001e-05 2024-02-05T04:53:44.301 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.301 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.301 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:44.302 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.172522+0000", 2024-02-05T04:53:44.302 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.4609999999999997e-06 2024-02-05T04:53:44.302 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.302 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.302 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:44.302 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.176520+0000", 2024-02-05T04:53:44.302 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0039978909999999999 2024-02-05T04:53:44.302 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.302 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.302 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:44.302 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.176574+0000", 2024-02-05T04:53:44.302 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.3774999999999999e-05 2024-02-05T04:53:44.302 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.302 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:44.302 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.303 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.303 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.303 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.8s6 43/31 ECSubWrite(tid=64, reqid=osd.4.0:62, at_version=0'0, trim_to=0'0, roll_forward_to=43'19))", 2024-02-05T04:53:44.303 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:35.181085+0000", 2024-02-05T04:53:44.303 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 9.0702300890000007, 2024-02-05T04:53:44.303 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0036911579999999999, 2024-02-05T04:53:44.303 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:44.303 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:53:44.303 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:44.303 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.303 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:44.303 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.181085+0000", 2024-02-05T04:53:44.303 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:44.303 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.303 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.304 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:44.304 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.181085+0000", 2024-02-05T04:53:44.304 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:44.304 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.304 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.304 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:44.304 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.181086+0000", 2024-02-05T04:53:44.304 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.8899999999999998e-07 2024-02-05T04:53:44.304 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.304 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.304 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:44.304 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.181098+0000", 2024-02-05T04:53:44.304 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1262e-05 2024-02-05T04:53:44.304 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.304 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.305 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:44.305 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.181115+0000", 2024-02-05T04:53:44.305 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.7447000000000001e-05 2024-02-05T04:53:44.305 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.305 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.305 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:44.305 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.181124+0000", 2024-02-05T04:53:44.305 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.8030000000000003e-06 2024-02-05T04:53:44.305 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.305 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.305 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:44.305 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.181186+0000", 2024-02-05T04:53:44.305 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.2663000000000006e-05 2024-02-05T04:53:44.305 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.305 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.305 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:44.306 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.181196+0000", 2024-02-05T04:53:44.306 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.7909999999999999e-06 2024-02-05T04:53:44.306 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.306 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.306 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:44.306 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.184734+0000", 2024-02-05T04:53:44.306 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.003537882 2024-02-05T04:53:44.306 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.306 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.306 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:44.306 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.184776+0000", 2024-02-05T04:53:44.306 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.2320999999999997e-05 2024-02-05T04:53:44.306 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.306 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:44.307 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.307 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.307 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.307 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.cs4 52/31 ECSubWrite(tid=83, reqid=osd.11.0:83, at_version=52'11, trim_to=0'0, roll_forward_to=51'9))", 2024-02-05T04:53:44.307 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:43.953850+0000", 2024-02-05T04:53:44.307 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 0.29746531999999998, 2024-02-05T04:53:44.307 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0031014770000000001, 2024-02-05T04:53:44.307 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:44.307 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:53:44.307 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:44.307 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.307 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:44.307 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.953850+0000", 2024-02-05T04:53:44.307 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:44.307 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.307 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.307 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:44.308 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.953850+0000", 2024-02-05T04:53:44.308 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:44.308 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.308 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.308 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:44.308 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.953850+0000", 2024-02-05T04:53:44.308 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.1199999999999999e-07 2024-02-05T04:53:44.308 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.308 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.308 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:44.308 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.953866+0000", 2024-02-05T04:53:44.308 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.5438999999999999e-05 2024-02-05T04:53:44.308 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.308 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.308 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:44.308 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.953875+0000", 2024-02-05T04:53:44.309 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.1139999999999995e-06 2024-02-05T04:53:44.309 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.309 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.309 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:44.309 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.953882+0000", 2024-02-05T04:53:44.309 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.4379999999999998e-06 2024-02-05T04:53:44.309 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.309 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.309 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:44.309 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.953935+0000", 2024-02-05T04:53:44.309 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.2247999999999998e-05 2024-02-05T04:53:44.309 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.309 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.309 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:44.309 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.953943+0000", 2024-02-05T04:53:44.310 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.7679999999999997e-06 2024-02-05T04:53:44.310 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.310 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.310 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:44.310 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.956915+0000", 2024-02-05T04:53:44.310 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.002971956 2024-02-05T04:53:44.310 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.310 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.310 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:44.310 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.956952+0000", 2024-02-05T04:53:44.310 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.6202000000000003e-05 2024-02-05T04:53:44.310 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.310 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:44.310 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.310 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.311 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.311 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.7s1 52/31 ECSubWrite(tid=201, reqid=osd.13.0:201, at_version=52'21, trim_to=0'0, roll_forward_to=51'17))", 2024-02-05T04:53:44.311 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:43.954949+0000", 2024-02-05T04:53:44.311 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 0.29636594700000002, 2024-02-05T04:53:44.311 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0038791429999999998, 2024-02-05T04:53:44.311 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:44.311 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:53:44.311 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:44.311 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.311 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:44.311 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.954949+0000", 2024-02-05T04:53:44.311 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:44.311 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.311 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.312 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:44.312 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.954949+0000", 2024-02-05T04:53:44.312 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:44.312 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.312 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.312 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:44.312 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.954950+0000", 2024-02-05T04:53:44.312 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.1600000000000001e-07 2024-02-05T04:53:44.312 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.312 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.312 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:44.312 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.954976+0000", 2024-02-05T04:53:44.312 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.6310999999999999e-05 2024-02-05T04:53:44.312 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.312 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.313 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:44.313 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.954992+0000", 2024-02-05T04:53:44.313 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.5460000000000001e-05 2024-02-05T04:53:44.313 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.313 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.313 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:44.313 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.955007+0000", 2024-02-05T04:53:44.313 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.4892000000000001e-05 2024-02-05T04:53:44.313 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.313 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.313 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:44.313 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.955213+0000", 2024-02-05T04:53:44.313 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000206197 2024-02-05T04:53:44.313 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.313 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.314 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:44.314 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.955219+0000", 2024-02-05T04:53:44.314 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.6579999999999999e-06 2024-02-05T04:53:44.314 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.314 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.314 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:44.314 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.958794+0000", 2024-02-05T04:53:44.314 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0035750489999999998 2024-02-05T04:53:44.314 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.314 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.314 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:44.314 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.958829+0000", 2024-02-05T04:53:44.314 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.4860000000000002e-05 2024-02-05T04:53:44.314 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.315 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:44.315 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.315 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.315 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.315 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.es5 52/31 ECSubWrite(tid=55, reqid=osd.2.0:54, at_version=0'0, trim_to=0'0, roll_forward_to=52'48))", 2024-02-05T04:53:44.315 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:43.956313+0000", 2024-02-05T04:53:44.315 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 0.29500199500000002, 2024-02-05T04:53:44.315 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.002517376, 2024-02-05T04:53:44.315 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:44.315 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:53:44.315 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:44.315 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.315 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:44.315 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.956313+0000", 2024-02-05T04:53:44.315 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:44.315 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.316 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.316 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:44.316 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.956313+0000", 2024-02-05T04:53:44.316 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:44.316 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.316 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.316 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:44.316 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.956314+0000", 2024-02-05T04:53:44.316 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.9400000000000004e-07 2024-02-05T04:53:44.316 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.316 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.316 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:44.316 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.956330+0000", 2024-02-05T04:53:44.316 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.6005999999999999e-05 2024-02-05T04:53:44.316 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.317 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.317 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:44.317 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.956345+0000", 2024-02-05T04:53:44.317 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.4416999999999999e-05 2024-02-05T04:53:44.317 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.317 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.317 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:44.317 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.956352+0000", 2024-02-05T04:53:44.317 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.8569999999999995e-06 2024-02-05T04:53:44.317 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.317 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.317 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:44.317 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.956477+0000", 2024-02-05T04:53:44.317 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000124277 2024-02-05T04:53:44.317 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.318 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.318 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:44.318 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.956490+0000", 2024-02-05T04:53:44.318 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2957e-05 2024-02-05T04:53:44.318 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.318 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.318 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:44.318 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.958795+0000", 2024-02-05T04:53:44.318 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.002305578 2024-02-05T04:53:44.318 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.318 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.318 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:44.318 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.958831+0000", 2024-02-05T04:53:44.318 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.5490000000000001e-05 2024-02-05T04:53:44.318 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.319 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:44.319 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.319 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.319 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.319 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.bs4 52/31 ECSubWrite(tid=202, reqid=osd.13.0:200, at_version=0'0, trim_to=0'0, roll_forward_to=52'50))", 2024-02-05T04:53:44.319 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:43.959100+0000", 2024-02-05T04:53:44.319 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 0.292215901, 2024-02-05T04:53:44.319 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0046049940000000003, 2024-02-05T04:53:44.319 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:44.319 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:53:44.319 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:44.319 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.319 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:44.319 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.959100+0000", 2024-02-05T04:53:44.320 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:44.320 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.320 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.320 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:44.320 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.959100+0000", 2024-02-05T04:53:44.320 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:44.320 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.320 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.320 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:44.320 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.959100+0000", 2024-02-05T04:53:44.320 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.0999999999999998e-07 2024-02-05T04:53:44.320 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.320 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.320 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:44.321 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.959108+0000", 2024-02-05T04:53:44.321 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.7389999999999999e-06 2024-02-05T04:53:44.321 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.321 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.321 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:44.321 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.959118+0000", 2024-02-05T04:53:44.321 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.8260000000000005e-06 2024-02-05T04:53:44.321 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.321 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.321 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:44.321 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.959125+0000", 2024-02-05T04:53:44.321 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.4950000000000002e-06 2024-02-05T04:53:44.321 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.322 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.322 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:44.322 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.959170+0000", 2024-02-05T04:53:44.322 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.5012000000000002e-05 2024-02-05T04:53:44.322 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.322 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.322 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:44.322 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.959177+0000", 2024-02-05T04:53:44.322 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.4119999999999998e-06 2024-02-05T04:53:44.322 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.322 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.322 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:44.322 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.963670+0000", 2024-02-05T04:53:44.322 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0044937919999999999 2024-02-05T04:53:44.323 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.323 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.323 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:44.323 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.963704+0000", 2024-02-05T04:53:44.323 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.4107999999999999e-05 2024-02-05T04:53:44.323 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.323 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:44.323 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.323 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.323 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.323 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.cs4 52/31 ECSubWrite(tid=86, reqid=osd.11.0:83, at_version=0'0, trim_to=0'0, roll_forward_to=52'11))", 2024-02-05T04:53:44.323 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:43.959278+0000", 2024-02-05T04:53:44.323 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 0.292037451, 2024-02-05T04:53:44.323 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0044474529999999996, 2024-02-05T04:53:44.324 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:44.324 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:53:44.324 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:44.324 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.324 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:44.324 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.959278+0000", 2024-02-05T04:53:44.324 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:44.324 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.324 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.324 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:44.324 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.959278+0000", 2024-02-05T04:53:44.324 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:44.324 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.325 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.325 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:44.325 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.959279+0000", 2024-02-05T04:53:44.325 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.9999999999999997e-07 2024-02-05T04:53:44.325 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.325 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.325 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:44.325 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.959288+0000", 2024-02-05T04:53:44.325 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.5030000000000003e-06 2024-02-05T04:53:44.325 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.325 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.325 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:44.325 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.959299+0000", 2024-02-05T04:53:44.325 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1069e-05 2024-02-05T04:53:44.326 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.326 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.326 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:44.326 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.959307+0000", 2024-02-05T04:53:44.326 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.8539999999999997e-06 2024-02-05T04:53:44.326 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.326 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.326 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:44.326 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.959358+0000", 2024-02-05T04:53:44.326 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.0793e-05 2024-02-05T04:53:44.326 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.326 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.326 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:44.327 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.959367+0000", 2024-02-05T04:53:44.327 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.3700000000000001e-06 2024-02-05T04:53:44.327 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.327 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.327 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:44.327 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.963676+0000", 2024-02-05T04:53:44.327 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0043093519999999998 2024-02-05T04:53:44.327 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.327 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.327 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:44.327 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.963725+0000", 2024-02-05T04:53:44.327 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.8912000000000002e-05 2024-02-05T04:53:44.327 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.328 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:44.328 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.328 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.328 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.328 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.7s1 52/31 ECSubWrite(tid=204, reqid=osd.13.0:201, at_version=0'0, trim_to=0'0, roll_forward_to=52'21))", 2024-02-05T04:53:44.328 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:43.962024+0000", 2024-02-05T04:53:44.328 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 0.28929126700000002, 2024-02-05T04:53:44.328 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0043224350000000003, 2024-02-05T04:53:44.328 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:44.328 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:53:44.328 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:44.328 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.328 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:44.329 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.962024+0000", 2024-02-05T04:53:44.329 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:44.329 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.329 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.329 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:44.329 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.962024+0000", 2024-02-05T04:53:44.329 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:44.329 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.329 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.329 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:44.329 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.962025+0000", 2024-02-05T04:53:44.329 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.7899999999999998e-07 2024-02-05T04:53:44.330 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.330 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.330 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:44.330 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.962044+0000", 2024-02-05T04:53:44.330 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.8890999999999999e-05 2024-02-05T04:53:44.330 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.330 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.330 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:44.330 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.962057+0000", 2024-02-05T04:53:44.330 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.3054000000000001e-05 2024-02-05T04:53:44.330 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.330 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.330 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:44.330 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.962065+0000", 2024-02-05T04:53:44.331 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.1480000000000006e-06 2024-02-05T04:53:44.331 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.331 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.331 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:44.331 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.962120+0000", 2024-02-05T04:53:44.331 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.4710000000000003e-05 2024-02-05T04:53:44.331 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.331 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.331 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:44.331 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.962125+0000", 2024-02-05T04:53:44.331 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.4480000000000002e-06 2024-02-05T04:53:44.331 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.331 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.331 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:44.331 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.966307+0000", 2024-02-05T04:53:44.331 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0041822480000000004 2024-02-05T04:53:44.331 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.332 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.332 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:44.332 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.966347+0000", 2024-02-05T04:53:44.332 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.9257e-05 2024-02-05T04:53:44.332 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.332 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:44.332 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.332 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.332 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.332 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.5s6 52/31 ECSubWrite(tid=42, reqid=osd.15.0:42, at_version=52'32, trim_to=0'0, roll_forward_to=51'30))", 2024-02-05T04:53:44.332 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:43.965838+0000", 2024-02-05T04:53:44.332 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 0.285477759, 2024-02-05T04:53:44.332 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.003198102, 2024-02-05T04:53:44.332 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:44.333 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:53:44.333 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:44.333 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.333 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:44.333 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.965838+0000", 2024-02-05T04:53:44.333 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:44.333 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.333 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.333 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:44.333 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.965838+0000", 2024-02-05T04:53:44.333 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:44.333 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.333 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.333 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:44.333 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.965838+0000", 2024-02-05T04:53:44.333 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.58e-07 2024-02-05T04:53:44.333 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.334 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.334 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:44.334 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.965856+0000", 2024-02-05T04:53:44.334 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.8250999999999999e-05 2024-02-05T04:53:44.334 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.334 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.334 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:44.334 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.965872+0000", 2024-02-05T04:53:44.334 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.5868000000000001e-05 2024-02-05T04:53:44.334 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.334 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.334 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:44.334 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.965881+0000", 2024-02-05T04:53:44.334 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.4230000000000005e-06 2024-02-05T04:53:44.335 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.335 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.335 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:44.335 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.965927+0000", 2024-02-05T04:53:44.335 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.6230000000000003e-05 2024-02-05T04:53:44.335 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.335 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.335 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:44.335 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.965933+0000", 2024-02-05T04:53:44.335 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.3849999999999999e-06 2024-02-05T04:53:44.335 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.335 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.335 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:44.335 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.969009+0000", 2024-02-05T04:53:44.335 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0030754350000000001 2024-02-05T04:53:44.335 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.336 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.336 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:44.336 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.969036+0000", 2024-02-05T04:53:44.336 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.7152e-05 2024-02-05T04:53:44.336 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.336 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:44.336 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.336 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.336 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:44.336 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-02-05T04:53:44.338 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-02-05T04:53:44.338 INFO:tasks.ceph.osd.1.smithi091.stderr:2024-02-05T04:53:44.336+0000 7ff9dc283640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34275) UID: 0 2024-02-05T04:53:44.438 INFO:tasks.ceph.osd.0.smithi062.stderr:2024-02-05T04:53:44.437+0000 7fece8e03640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34688) UID: 0 2024-02-05T04:53:44.485 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-02-05T04:53:44.486 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [], 2024-02-05T04:53:44.486 INFO:teuthology.orchestra.run.smithi150.stdout: "num_ops": 0 2024-02-05T04:53:44.486 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-02-05T04:53:44.496 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-02-05T04:53:44.539 INFO:tasks.ceph.osd.1.smithi091.stderr:2024-02-05T04:53:44.538+0000 7ff9dc283640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34275) UID: 0 2024-02-05T04:53:44.601 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-02-05T04:53:44.601 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [], 2024-02-05T04:53:44.601 INFO:teuthology.orchestra.run.smithi150.stdout: "complaint_time": 30, 2024-02-05T04:53:44.601 INFO:teuthology.orchestra.run.smithi150.stdout: "num_blocked_ops": 0 2024-02-05T04:53:44.601 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-02-05T04:53:44.610 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-02-05T04:53:44.640 INFO:tasks.ceph.osd.12.smithi062.stderr:2024-02-05T04:53:44.639+0000 7f5e5847e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 34695) UID: 0 2024-02-05T04:53:44.718 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-02-05T04:53:44.718 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 20, 2024-02-05T04:53:44.718 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 600, 2024-02-05T04:53:44.718 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [ 2024-02-05T04:53:44.719 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.719 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.ds3 37/31 ECSubWrite(tid=23, reqid=client.4634.0:225, at_version=37'14, trim_to=0'0, roll_forward_to=34'12))", 2024-02-05T04:53:44.719 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:29.146416+0000", 2024-02-05T04:53:44.719 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 15.569800614, 2024-02-05T04:53:44.719 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0040617370000000002, 2024-02-05T04:53:44.719 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:44.719 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:53:44.719 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:44.719 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.719 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:44.719 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.146416+0000", 2024-02-05T04:53:44.719 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:44.719 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.720 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.720 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:44.720 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.146416+0000", 2024-02-05T04:53:44.720 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:44.720 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.720 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.720 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:44.720 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.146416+0000", 2024-02-05T04:53:44.720 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.6899999999999999e-07 2024-02-05T04:53:44.720 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.721 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.721 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:44.721 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.146884+0000", 2024-02-05T04:53:44.721 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00046836600000000001 2024-02-05T04:53:44.721 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.721 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.721 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:44.721 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.146897+0000", 2024-02-05T04:53:44.721 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2353e-05 2024-02-05T04:53:44.721 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.721 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.721 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:44.721 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.146906+0000", 2024-02-05T04:53:44.722 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.4720000000000001e-06 2024-02-05T04:53:44.722 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.722 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.722 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:44.722 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.146964+0000", 2024-02-05T04:53:44.722 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.7955000000000003e-05 2024-02-05T04:53:44.722 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.722 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.722 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:44.722 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.146971+0000", 2024-02-05T04:53:44.722 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.1930000000000004e-06 2024-02-05T04:53:44.722 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.722 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.723 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:44.723 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.150423+0000", 2024-02-05T04:53:44.723 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0034517409999999999 2024-02-05T04:53:44.723 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.723 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.723 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:44.723 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.150478+0000", 2024-02-05T04:53:44.723 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.4487999999999997e-05 2024-02-05T04:53:44.723 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.723 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:44.723 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.723 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.723 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.723 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.1s4 37/31 ECSubWrite(tid=86, reqid=client.4634.0:247, at_version=37'15, trim_to=0'0, roll_forward_to=35'13))", 2024-02-05T04:53:44.724 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:29.193372+0000", 2024-02-05T04:53:44.724 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 15.522844336, 2024-02-05T04:53:44.724 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0032430580000000001, 2024-02-05T04:53:44.724 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:44.724 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:53:44.724 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:44.724 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.724 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:44.724 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.193372+0000", 2024-02-05T04:53:44.724 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:44.724 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.724 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.724 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:44.724 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.193372+0000", 2024-02-05T04:53:44.725 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:44.725 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.725 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.725 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:44.725 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.193372+0000", 2024-02-05T04:53:44.725 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.72e-07 2024-02-05T04:53:44.725 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.725 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.725 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:44.725 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.193863+0000", 2024-02-05T04:53:44.725 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00049020699999999999 2024-02-05T04:53:44.725 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.725 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.725 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:44.726 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.193879+0000", 2024-02-05T04:53:44.726 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.6212e-05 2024-02-05T04:53:44.726 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.726 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.726 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:44.726 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.193890+0000", 2024-02-05T04:53:44.726 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1433999999999999e-05 2024-02-05T04:53:44.726 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.726 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.726 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:44.726 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.193993+0000", 2024-02-05T04:53:44.726 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000102954 2024-02-05T04:53:44.726 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.726 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.726 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:44.727 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.194016+0000", 2024-02-05T04:53:44.727 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.2370000000000001e-05 2024-02-05T04:53:44.727 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.727 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.727 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:44.727 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.196583+0000", 2024-02-05T04:53:44.727 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0025678939999999998 2024-02-05T04:53:44.727 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.727 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.727 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:44.727 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.196615+0000", 2024-02-05T04:53:44.727 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.1714999999999997e-05 2024-02-05T04:53:44.727 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.727 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:44.727 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.728 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.728 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.728 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.ds3 38/31 ECSubWrite(tid=26, reqid=osd.7.0:26, at_version=38'16, trim_to=0'0, roll_forward_to=37'14))", 2024-02-05T04:53:44.728 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:30.134724+0000", 2024-02-05T04:53:44.728 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 14.581491992, 2024-02-05T04:53:44.728 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0034259059999999998, 2024-02-05T04:53:44.729 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:44.729 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:53:44.729 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:44.729 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.729 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:44.729 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.134724+0000", 2024-02-05T04:53:44.730 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:44.730 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.730 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.730 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:44.730 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.134724+0000", 2024-02-05T04:53:44.730 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:44.731 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.731 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.731 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:44.731 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.134725+0000", 2024-02-05T04:53:44.731 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.4499999999999998e-07 2024-02-05T04:53:44.731 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.732 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.732 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:44.732 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.134761+0000", 2024-02-05T04:53:44.732 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.6272000000000001e-05 2024-02-05T04:53:44.732 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.732 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.732 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:44.732 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.135179+0000", 2024-02-05T04:53:44.733 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000417809 2024-02-05T04:53:44.733 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.733 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.733 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:44.733 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.135189+0000", 2024-02-05T04:53:44.733 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0614999999999999e-05 2024-02-05T04:53:44.733 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.733 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.734 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:44.734 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.135259+0000", 2024-02-05T04:53:44.734 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.9238000000000004e-05 2024-02-05T04:53:44.734 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.734 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.734 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:44.734 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.135268+0000", 2024-02-05T04:53:44.734 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.8179999999999999e-06 2024-02-05T04:53:44.734 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.734 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.734 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:44.734 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.138118+0000", 2024-02-05T04:53:44.734 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0028499049999999998 2024-02-05T04:53:44.735 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.735 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.735 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:44.735 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.138150+0000", 2024-02-05T04:53:44.735 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.2004000000000001e-05 2024-02-05T04:53:44.735 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.735 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:44.735 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.735 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.735 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.735 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.1s4 38/31 ECSubWrite(tid=89, reqid=osd.13.0:89, at_version=38'17, trim_to=0'0, roll_forward_to=37'15))", 2024-02-05T04:53:44.735 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:30.139873+0000", 2024-02-05T04:53:44.735 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 14.576343474, 2024-02-05T04:53:44.735 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0039549099999999998, 2024-02-05T04:53:44.735 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:44.736 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:53:44.736 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:44.736 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.736 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:44.736 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.139873+0000", 2024-02-05T04:53:44.736 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:44.736 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.736 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.736 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:44.736 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.139873+0000", 2024-02-05T04:53:44.736 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:44.736 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.737 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.737 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:44.737 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.139873+0000", 2024-02-05T04:53:44.737 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.3099999999999999e-07 2024-02-05T04:53:44.737 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.737 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.737 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:44.737 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.139892+0000", 2024-02-05T04:53:44.737 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.8209000000000001e-05 2024-02-05T04:53:44.737 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.737 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.737 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:44.737 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.139929+0000", 2024-02-05T04:53:44.737 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.7335000000000001e-05 2024-02-05T04:53:44.737 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.738 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.738 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:44.738 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.139940+0000", 2024-02-05T04:53:44.738 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0871e-05 2024-02-05T04:53:44.738 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.738 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.738 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:44.738 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.140006+0000", 2024-02-05T04:53:44.738 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.5809000000000006e-05 2024-02-05T04:53:44.738 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.739 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.739 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:44.739 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.140017+0000", 2024-02-05T04:53:44.739 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1066e-05 2024-02-05T04:53:44.739 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.739 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.739 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:44.739 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.143799+0000", 2024-02-05T04:53:44.739 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0037826579999999999 2024-02-05T04:53:44.739 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.739 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.739 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:44.739 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.143828+0000", 2024-02-05T04:53:44.739 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.8631000000000001e-05 2024-02-05T04:53:44.740 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.740 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:44.740 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.740 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.740 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.740 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.ds3 38/31 ECSubWrite(tid=27, reqid=osd.7.0:26, at_version=0'0, trim_to=0'0, roll_forward_to=38'16))", 2024-02-05T04:53:44.740 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:30.141571+0000", 2024-02-05T04:53:44.740 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 14.574645882, 2024-02-05T04:53:44.740 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.002233016, 2024-02-05T04:53:44.740 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:44.740 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:53:44.740 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:44.741 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.741 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:44.741 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.141571+0000", 2024-02-05T04:53:44.741 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:44.741 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.741 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.741 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:44.741 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.141571+0000", 2024-02-05T04:53:44.741 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:44.741 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.741 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.741 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:44.741 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.141571+0000", 2024-02-05T04:53:44.742 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.3e-07 2024-02-05T04:53:44.742 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.742 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.742 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:44.742 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.141579+0000", 2024-02-05T04:53:44.742 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.4410000000000004e-06 2024-02-05T04:53:44.742 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.742 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.742 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:44.742 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.141588+0000", 2024-02-05T04:53:44.742 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.4290000000000007e-06 2024-02-05T04:53:44.742 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.742 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.743 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:44.743 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.141594+0000", 2024-02-05T04:53:44.743 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.8490000000000002e-06 2024-02-05T04:53:44.743 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.743 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.743 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:44.743 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.141640+0000", 2024-02-05T04:53:44.744 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.5986000000000003e-05 2024-02-05T04:53:44.744 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.744 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.744 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:44.744 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.141646+0000", 2024-02-05T04:53:44.745 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.489e-06 2024-02-05T04:53:44.745 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.745 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.745 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:44.745 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.143762+0000", 2024-02-05T04:53:44.745 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.002115933 2024-02-05T04:53:44.746 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.746 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.746 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:44.746 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.143804+0000", 2024-02-05T04:53:44.746 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.1258999999999999e-05 2024-02-05T04:53:44.746 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.746 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:44.746 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.746 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.746 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.746 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.fs5 38/31 ECSubWrite(tid=52, reqid=client.4634.0:291, at_version=38'18, trim_to=0'0, roll_forward_to=34'16))", 2024-02-05T04:53:44.746 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:30.389344+0000", 2024-02-05T04:53:44.747 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 14.326872668, 2024-02-05T04:53:44.747 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0043331690000000004, 2024-02-05T04:53:44.747 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:44.747 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:53:44.747 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:44.747 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.747 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:44.747 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.389344+0000", 2024-02-05T04:53:44.747 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:44.747 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.747 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.747 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:44.747 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.389344+0000", 2024-02-05T04:53:44.747 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:44.748 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.748 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.748 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:44.748 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.389344+0000", 2024-02-05T04:53:44.748 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.0699999999999997e-07 2024-02-05T04:53:44.748 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.748 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.748 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:44.748 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.389601+0000", 2024-02-05T04:53:44.748 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00025713199999999998 2024-02-05T04:53:44.748 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.748 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.749 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:44.749 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.389611+0000", 2024-02-05T04:53:44.749 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.8130000000000007e-06 2024-02-05T04:53:44.749 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.749 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.749 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:44.749 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.389617+0000", 2024-02-05T04:53:44.749 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.9320000000000001e-06 2024-02-05T04:53:44.749 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.749 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.749 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:44.749 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.389674+0000", 2024-02-05T04:53:44.749 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.6675000000000002e-05 2024-02-05T04:53:44.749 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.750 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.750 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:44.750 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.389685+0000", 2024-02-05T04:53:44.750 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0692e-05 2024-02-05T04:53:44.750 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.750 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.750 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:44.750 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.393645+0000", 2024-02-05T04:53:44.750 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0039600390000000003 2024-02-05T04:53:44.750 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.750 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.750 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:44.750 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.393677+0000", 2024-02-05T04:53:44.750 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.2379000000000003e-05 2024-02-05T04:53:44.751 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.751 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:44.751 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.751 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.751 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.751 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.fs5 38/31 ECSubWrite(tid=53, reqid=client.4634.0:292, at_version=38'19, trim_to=0'0, roll_forward_to=34'16))", 2024-02-05T04:53:44.751 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:30.389931+0000", 2024-02-05T04:53:44.751 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 14.32628553, 2024-02-05T04:53:44.751 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0037694059999999999, 2024-02-05T04:53:44.751 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:44.751 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:53:44.752 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:44.752 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.752 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:44.752 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.389931+0000", 2024-02-05T04:53:44.752 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:44.752 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.752 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.752 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:44.752 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.389931+0000", 2024-02-05T04:53:44.752 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:44.752 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.752 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.752 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:44.753 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.389932+0000", 2024-02-05T04:53:44.753 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.1099999999999995e-07 2024-02-05T04:53:44.753 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.753 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.753 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:44.753 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.389977+0000", 2024-02-05T04:53:44.753 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.5744000000000001e-05 2024-02-05T04:53:44.753 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.753 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.753 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:44.753 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.389988+0000", 2024-02-05T04:53:44.753 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0624e-05 2024-02-05T04:53:44.753 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.754 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.754 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:44.754 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.390000+0000", 2024-02-05T04:53:44.754 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1922999999999999e-05 2024-02-05T04:53:44.754 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.754 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.754 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:44.754 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.390076+0000", 2024-02-05T04:53:44.754 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.5833000000000004e-05 2024-02-05T04:53:44.754 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.754 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.754 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:44.754 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.390081+0000", 2024-02-05T04:53:44.755 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.152e-06 2024-02-05T04:53:44.755 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.755 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.755 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:44.755 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.393680+0000", 2024-02-05T04:53:44.755 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0035987359999999999 2024-02-05T04:53:44.755 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.755 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.755 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:44.755 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.393700+0000", 2024-02-05T04:53:44.755 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.0783000000000001e-05 2024-02-05T04:53:44.755 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.756 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:44.756 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.756 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.756 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.756 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.fs5 38/31 ECSubWrite(tid=54, reqid=client.4634.0:293, at_version=38'20, trim_to=0'0, roll_forward_to=34'16))", 2024-02-05T04:53:44.756 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:30.390527+0000", 2024-02-05T04:53:44.756 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 14.325689092999999, 2024-02-05T04:53:44.756 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0031954259999999999, 2024-02-05T04:53:44.756 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:44.756 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:53:44.756 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:44.756 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.757 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:44.757 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.390527+0000", 2024-02-05T04:53:44.757 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:44.757 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.757 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.757 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:44.757 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.390527+0000", 2024-02-05T04:53:44.757 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:44.757 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.757 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.757 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:44.757 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.390528+0000", 2024-02-05T04:53:44.757 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.3799999999999997e-07 2024-02-05T04:53:44.758 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.758 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.758 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:44.758 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.390542+0000", 2024-02-05T04:53:44.758 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.4182000000000001e-05 2024-02-05T04:53:44.758 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.758 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.758 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:44.758 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.390554+0000", 2024-02-05T04:53:44.758 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2089999999999999e-05 2024-02-05T04:53:44.758 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.759 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.759 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:44.759 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.390563+0000", 2024-02-05T04:53:44.759 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.4300000000000006e-06 2024-02-05T04:53:44.760 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.760 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.760 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:44.760 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.390617+0000", 2024-02-05T04:53:44.760 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.4073999999999998e-05 2024-02-05T04:53:44.760 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.760 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.760 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:44.760 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.390626+0000", 2024-02-05T04:53:44.760 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.6420000000000002e-06 2024-02-05T04:53:44.760 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.760 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.760 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:44.760 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.393705+0000", 2024-02-05T04:53:44.760 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0030783640000000001 2024-02-05T04:53:44.761 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.761 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.761 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:44.761 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.393723+0000", 2024-02-05T04:53:44.761 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.8105999999999999e-05 2024-02-05T04:53:44.761 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.761 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:44.761 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.761 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.761 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.761 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.1s4 42/31 ECSubWrite(tid=135, reqid=client.4634.0:379, at_version=42'27, trim_to=0'0, roll_forward_to=40'25))", 2024-02-05T04:53:44.761 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:34.167832+0000", 2024-02-05T04:53:44.761 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 10.548384113999999, 2024-02-05T04:53:44.761 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0027264910000000002, 2024-02-05T04:53:44.761 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:44.762 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:53:44.762 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:44.762 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.762 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:44.762 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.167832+0000", 2024-02-05T04:53:44.762 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:44.762 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.762 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.762 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:44.762 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.167832+0000", 2024-02-05T04:53:44.762 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:44.762 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.762 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.762 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:44.762 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.167832+0000", 2024-02-05T04:53:44.762 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1000000000000001e-07 2024-02-05T04:53:44.763 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.763 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.763 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:44.763 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.168626+0000", 2024-02-05T04:53:44.763 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00079308799999999995 2024-02-05T04:53:44.763 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.763 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.763 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:44.763 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.168650+0000", 2024-02-05T04:53:44.763 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.4457e-05 2024-02-05T04:53:44.763 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.763 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.763 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:44.763 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.168662+0000", 2024-02-05T04:53:44.763 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2247e-05 2024-02-05T04:53:44.764 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.764 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.764 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:44.764 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.168750+0000", 2024-02-05T04:53:44.764 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.8116999999999997e-05 2024-02-05T04:53:44.764 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.764 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.764 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:44.764 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.168762+0000", 2024-02-05T04:53:44.764 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1664e-05 2024-02-05T04:53:44.764 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.764 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.764 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:44.764 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.170515+0000", 2024-02-05T04:53:44.764 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0017534370000000001 2024-02-05T04:53:44.765 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.765 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.765 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:44.765 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.170559+0000", 2024-02-05T04:53:44.765 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.3371000000000002e-05 2024-02-05T04:53:44.765 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.765 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:44.765 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.765 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.765 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.765 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.1s4 43/31 ECSubWrite(tid=140, reqid=osd.13.0:140, at_version=43'29, trim_to=0'0, roll_forward_to=42'27))", 2024-02-05T04:53:44.765 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:35.182024+0000", 2024-02-05T04:53:44.765 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 9.5341928060000001, 2024-02-05T04:53:44.765 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.002531752, 2024-02-05T04:53:44.765 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:44.766 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:53:44.766 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:44.766 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.766 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:44.766 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.182024+0000", 2024-02-05T04:53:44.766 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:44.766 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.766 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.766 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:44.766 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.182024+0000", 2024-02-05T04:53:44.766 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:44.766 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.766 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.766 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:44.766 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.182024+0000", 2024-02-05T04:53:44.766 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.2e-07 2024-02-05T04:53:44.767 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.767 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.767 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:44.767 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.182046+0000", 2024-02-05T04:53:44.767 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.2005999999999999e-05 2024-02-05T04:53:44.767 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.767 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.767 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:44.767 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.182087+0000", 2024-02-05T04:53:44.767 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.1233999999999999e-05 2024-02-05T04:53:44.767 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.767 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.767 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:44.767 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.182098+0000", 2024-02-05T04:53:44.767 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1026000000000001e-05 2024-02-05T04:53:44.768 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.768 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.768 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:44.768 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.182193+0000", 2024-02-05T04:53:44.768 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.501e-05 2024-02-05T04:53:44.768 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.768 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.768 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:44.768 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.182206+0000", 2024-02-05T04:53:44.768 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2398999999999999e-05 2024-02-05T04:53:44.768 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.768 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.768 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:44.768 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.184505+0000", 2024-02-05T04:53:44.769 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.002299499 2024-02-05T04:53:44.769 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.769 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.769 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:44.769 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.184555+0000", 2024-02-05T04:53:44.769 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.0158000000000003e-05 2024-02-05T04:53:44.769 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.769 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:44.769 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.769 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.769 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.769 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.ds3 43/31 ECSubWrite(tid=44, reqid=client.4634.0:401, at_version=43'24, trim_to=0'0, roll_forward_to=38'22))", 2024-02-05T04:53:44.769 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:35.183531+0000", 2024-02-05T04:53:44.769 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 9.5326854159999996, 2024-02-05T04:53:44.770 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0039565659999999999, 2024-02-05T04:53:44.770 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:44.770 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:53:44.770 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:44.770 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.770 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:44.770 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.183531+0000", 2024-02-05T04:53:44.770 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:44.770 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.770 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.770 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:44.770 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.183531+0000", 2024-02-05T04:53:44.770 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:44.770 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.770 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.771 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:44.771 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.183531+0000", 2024-02-05T04:53:44.771 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.24e-07 2024-02-05T04:53:44.771 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.771 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.771 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:44.771 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.183547+0000", 2024-02-05T04:53:44.771 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.5735000000000001e-05 2024-02-05T04:53:44.771 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.771 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.771 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:44.771 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.183557+0000", 2024-02-05T04:53:44.771 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0043e-05 2024-02-05T04:53:44.771 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.771 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.772 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:44.772 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.183565+0000", 2024-02-05T04:53:44.772 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.3510000000000002e-06 2024-02-05T04:53:44.772 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.772 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.772 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:44.772 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.183627+0000", 2024-02-05T04:53:44.772 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.1655000000000005e-05 2024-02-05T04:53:44.772 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.772 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.772 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:44.772 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.183633+0000", 2024-02-05T04:53:44.772 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.438e-06 2024-02-05T04:53:44.772 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.772 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.773 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:44.773 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.187449+0000", 2024-02-05T04:53:44.773 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0038160249999999998 2024-02-05T04:53:44.773 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.773 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.773 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:44.773 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.187488+0000", 2024-02-05T04:53:44.773 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.8195000000000003e-05 2024-02-05T04:53:44.773 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.773 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:44.773 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.773 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.773 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.773 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.1s4 43/31 ECSubWrite(tid=142, reqid=osd.13.0:142, at_version=43'31, trim_to=0'0, roll_forward_to=42'27))", 2024-02-05T04:53:44.774 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:35.183838+0000", 2024-02-05T04:53:44.774 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 9.5323789150000007, 2024-02-05T04:53:44.774 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0036591430000000001, 2024-02-05T04:53:44.774 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:44.774 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:53:44.774 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:44.774 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.774 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:44.774 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.183838+0000", 2024-02-05T04:53:44.774 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:44.774 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.774 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.774 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:44.774 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.183838+0000", 2024-02-05T04:53:44.774 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:44.774 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.775 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.775 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:44.775 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.183838+0000", 2024-02-05T04:53:44.775 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.8299999999999996e-07 2024-02-05T04:53:44.775 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.775 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.775 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:44.775 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.183879+0000", 2024-02-05T04:53:44.775 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.1020999999999999e-05 2024-02-05T04:53:44.775 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.775 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.775 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:44.775 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.183891+0000", 2024-02-05T04:53:44.775 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1477e-05 2024-02-05T04:53:44.775 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.776 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.776 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:44.776 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.183899+0000", 2024-02-05T04:53:44.776 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.6119999999999999e-06 2024-02-05T04:53:44.776 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.776 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.776 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:44.776 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.183965+0000", 2024-02-05T04:53:44.776 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.5078000000000003e-05 2024-02-05T04:53:44.776 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.776 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.776 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:44.776 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.183975+0000", 2024-02-05T04:53:44.776 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0002e-05 2024-02-05T04:53:44.776 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.777 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.777 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:44.777 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.187464+0000", 2024-02-05T04:53:44.777 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0034895210000000002 2024-02-05T04:53:44.777 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.777 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.777 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:44.777 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.187497+0000", 2024-02-05T04:53:44.777 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.2648999999999999e-05 2024-02-05T04:53:44.777 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.777 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:44.777 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.777 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.777 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.777 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.ds3 43/31 ECSubWrite(tid=45, reqid=client.4634.0:401, at_version=0'0, trim_to=0'0, roll_forward_to=43'24))", 2024-02-05T04:53:44.778 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:35.191307+0000", 2024-02-05T04:53:44.778 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 9.5249094470000006, 2024-02-05T04:53:44.778 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.001733547, 2024-02-05T04:53:44.778 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:44.778 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:53:44.778 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:44.778 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.778 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:44.778 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.191307+0000", 2024-02-05T04:53:44.778 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:44.778 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.778 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.778 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:44.778 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.191307+0000", 2024-02-05T04:53:44.779 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:44.779 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.779 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.779 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:44.779 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.191308+0000", 2024-02-05T04:53:44.779 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.6400000000000001e-07 2024-02-05T04:53:44.779 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.779 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.779 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:44.779 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.191319+0000", 2024-02-05T04:53:44.779 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.11e-05 2024-02-05T04:53:44.779 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.779 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.779 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:44.780 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.191335+0000", 2024-02-05T04:53:44.780 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.6039e-05 2024-02-05T04:53:44.780 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.780 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.780 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:44.780 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.191344+0000", 2024-02-05T04:53:44.780 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.1060000000000006e-06 2024-02-05T04:53:44.780 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.780 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.780 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:44.780 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.191404+0000", 2024-02-05T04:53:44.781 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.0250000000000001e-05 2024-02-05T04:53:44.781 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.781 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.781 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:44.781 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.191416+0000", 2024-02-05T04:53:44.782 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1389999999999999e-05 2024-02-05T04:53:44.782 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.782 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.782 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:44.782 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.193007+0000", 2024-02-05T04:53:44.782 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.001591007 2024-02-05T04:53:44.782 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.782 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.782 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:44.782 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.193041+0000", 2024-02-05T04:53:44.782 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.3890999999999998e-05 2024-02-05T04:53:44.782 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.782 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:44.783 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.783 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.783 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.783 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.fs5 52/31 ECSubWrite(tid=84, reqid=osd.11.0:84, at_version=52'38, trim_to=0'0, roll_forward_to=51'36))", 2024-02-05T04:53:44.783 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:43.953955+0000", 2024-02-05T04:53:44.783 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 0.76226128800000004, 2024-02-05T04:53:44.783 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0028144300000000001, 2024-02-05T04:53:44.783 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:44.783 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:53:44.783 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:44.783 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.783 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:44.783 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.953955+0000", 2024-02-05T04:53:44.784 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:44.784 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.784 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.784 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:44.784 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.953955+0000", 2024-02-05T04:53:44.784 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:44.784 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.784 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.784 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:44.784 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.953956+0000", 2024-02-05T04:53:44.784 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.41e-07 2024-02-05T04:53:44.785 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.785 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.785 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:44.785 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.953988+0000", 2024-02-05T04:53:44.785 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.2632000000000002e-05 2024-02-05T04:53:44.785 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.785 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.785 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:44.785 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.954010+0000", 2024-02-05T04:53:44.785 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.2266999999999999e-05 2024-02-05T04:53:44.785 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.786 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.786 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:44.786 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.954020+0000", 2024-02-05T04:53:44.786 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.8709999999999992e-06 2024-02-05T04:53:44.786 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.786 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.786 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:44.786 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.954089+0000", 2024-02-05T04:53:44.786 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.8577000000000004e-05 2024-02-05T04:53:44.786 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.786 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.786 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:44.786 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.954099+0000", 2024-02-05T04:53:44.786 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0054000000000001e-05 2024-02-05T04:53:44.787 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.787 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.787 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:44.787 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.956737+0000", 2024-02-05T04:53:44.787 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0026379039999999999 2024-02-05T04:53:44.787 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.787 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.787 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:44.787 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.956770+0000", 2024-02-05T04:53:44.787 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.2784000000000003e-05 2024-02-05T04:53:44.787 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.787 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:44.787 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.787 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.787 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.788 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.fs5 52/31 ECSubWrite(tid=85, reqid=osd.11.0:85, at_version=52'40, trim_to=0'0, roll_forward_to=51'36))", 2024-02-05T04:53:44.788 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:43.954787+0000", 2024-02-05T04:53:44.788 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 0.76142925900000002, 2024-02-05T04:53:44.788 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0039530299999999997, 2024-02-05T04:53:44.788 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:44.788 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:53:44.788 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:44.788 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.788 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:44.788 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.954787+0000", 2024-02-05T04:53:44.788 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:44.788 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.788 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.788 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:44.789 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.954787+0000", 2024-02-05T04:53:44.789 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:44.789 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.789 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.789 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:44.789 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.954788+0000", 2024-02-05T04:53:44.789 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.0500000000000004e-07 2024-02-05T04:53:44.789 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.789 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.789 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:44.789 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.954806+0000", 2024-02-05T04:53:44.789 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.7960000000000001e-05 2024-02-05T04:53:44.789 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.789 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.790 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:44.790 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.954817+0000", 2024-02-05T04:53:44.790 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1567e-05 2024-02-05T04:53:44.790 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.790 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.790 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:44.790 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.954824+0000", 2024-02-05T04:53:44.790 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.0210000000000002e-06 2024-02-05T04:53:44.790 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.790 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.790 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:44.790 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.955014+0000", 2024-02-05T04:53:44.791 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000189698 2024-02-05T04:53:44.791 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.791 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.791 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:44.791 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.955023+0000", 2024-02-05T04:53:44.791 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.7069999999999998e-06 2024-02-05T04:53:44.791 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.791 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.791 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:44.791 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.958708+0000", 2024-02-05T04:53:44.791 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00368505 2024-02-05T04:53:44.791 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.791 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.791 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:44.792 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.958740+0000", 2024-02-05T04:53:44.792 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.2521999999999998e-05 2024-02-05T04:53:44.792 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.792 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:44.792 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.792 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.792 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.792 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.9s6 52/31 ECSubWrite(tid=44, reqid=osd.14.0:44, at_version=52'19, trim_to=0'0, roll_forward_to=51'17))", 2024-02-05T04:53:44.792 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:43.957769+0000", 2024-02-05T04:53:44.792 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 0.75844753799999998, 2024-02-05T04:53:44.792 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0032554459999999999, 2024-02-05T04:53:44.792 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:44.793 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:53:44.793 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:44.793 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.793 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:44.793 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.957769+0000", 2024-02-05T04:53:44.793 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:44.793 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.793 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.793 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:44.793 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.957769+0000", 2024-02-05T04:53:44.793 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:44.793 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.793 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.793 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:44.793 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.957769+0000", 2024-02-05T04:53:44.794 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.1799999999999999e-07 2024-02-05T04:53:44.794 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.794 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.794 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:44.794 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.957786+0000", 2024-02-05T04:53:44.794 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.7323e-05 2024-02-05T04:53:44.794 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.794 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.794 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:44.794 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.957794+0000", 2024-02-05T04:53:44.794 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.9840000000000001e-06 2024-02-05T04:53:44.794 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.794 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.795 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:44.795 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.957801+0000", 2024-02-05T04:53:44.795 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.883e-06 2024-02-05T04:53:44.795 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.795 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.795 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:44.796 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.957862+0000", 2024-02-05T04:53:44.796 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.0885999999999999e-05 2024-02-05T04:53:44.796 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.796 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.796 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:44.796 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.957873+0000", 2024-02-05T04:53:44.796 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0781999999999999e-05 2024-02-05T04:53:44.796 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.796 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.796 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:44.796 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.960987+0000", 2024-02-05T04:53:44.797 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0031136079999999999 2024-02-05T04:53:44.797 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.797 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.797 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:44.797 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.961024+0000", 2024-02-05T04:53:44.797 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.7762000000000002e-05 2024-02-05T04:53:44.797 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.797 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:44.797 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.797 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.797 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.797 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.cs3 52/31 ECSubWrite(tid=86, reqid=osd.11.0:83, at_version=0'0, trim_to=0'0, roll_forward_to=52'11))", 2024-02-05T04:53:44.797 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:43.959278+0000", 2024-02-05T04:53:44.798 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 0.75693828299999999, 2024-02-05T04:53:44.798 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0043494279999999998, 2024-02-05T04:53:44.798 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:44.798 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:53:44.798 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:44.798 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.798 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:44.798 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.959278+0000", 2024-02-05T04:53:44.798 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:44.799 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.799 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.799 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:44.799 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.959278+0000", 2024-02-05T04:53:44.799 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:44.799 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.799 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.799 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:44.799 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.959279+0000", 2024-02-05T04:53:44.799 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.2699999999999999e-07 2024-02-05T04:53:44.800 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.800 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.800 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:44.800 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.959292+0000", 2024-02-05T04:53:44.800 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2754e-05 2024-02-05T04:53:44.800 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.801 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.801 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:44.801 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.959315+0000", 2024-02-05T04:53:44.801 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.3524999999999999e-05 2024-02-05T04:53:44.801 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.801 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.801 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:44.802 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.959324+0000", 2024-02-05T04:53:44.802 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.7830000000000004e-06 2024-02-05T04:53:44.802 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.802 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.802 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:44.802 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.959387+0000", 2024-02-05T04:53:44.803 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.347e-05 2024-02-05T04:53:44.803 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.803 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.803 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:44.803 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.959400+0000", 2024-02-05T04:53:44.803 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2289999999999999e-05 2024-02-05T04:53:44.804 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.804 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.804 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:44.804 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.963595+0000", 2024-02-05T04:53:44.804 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0041957119999999999 2024-02-05T04:53:44.804 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.804 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.805 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:44.805 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.963628+0000", 2024-02-05T04:53:44.805 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.2267000000000001e-05 2024-02-05T04:53:44.805 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.805 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:44.805 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.806 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.806 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.806 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.1s4 52/31 ECSubWrite(tid=203, reqid=osd.13.0:203, at_version=52'45, trim_to=0'0, roll_forward_to=51'43))", 2024-02-05T04:53:44.806 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:43.961350+0000", 2024-02-05T04:53:44.806 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 0.75486616299999998, 2024-02-05T04:53:44.806 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0051650439999999997, 2024-02-05T04:53:44.807 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:44.807 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:53:44.807 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:44.807 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.807 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:44.807 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.961350+0000", 2024-02-05T04:53:44.807 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:44.808 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.808 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.808 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:44.808 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.961350+0000", 2024-02-05T04:53:44.808 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:44.808 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.809 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.809 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:44.809 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.961351+0000", 2024-02-05T04:53:44.809 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.1699999999999999e-07 2024-02-05T04:53:44.809 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.809 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.810 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:44.810 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.961370+0000", 2024-02-05T04:53:44.810 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.8933e-05 2024-02-05T04:53:44.810 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.810 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.810 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:44.810 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.961379+0000", 2024-02-05T04:53:44.811 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.4660000000000005e-06 2024-02-05T04:53:44.811 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.811 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.811 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:44.811 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.961387+0000", 2024-02-05T04:53:44.811 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.6240000000000002e-06 2024-02-05T04:53:44.812 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.812 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.812 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:44.812 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.961457+0000", 2024-02-05T04:53:44.812 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.0288000000000002e-05 2024-02-05T04:53:44.812 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.813 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.813 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:44.813 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.961469+0000", 2024-02-05T04:53:44.813 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1945e-05 2024-02-05T04:53:44.813 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.813 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.814 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:44.814 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.966405+0000", 2024-02-05T04:53:44.814 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0049363649999999999 2024-02-05T04:53:44.814 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.814 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.814 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:44.815 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.966515+0000", 2024-02-05T04:53:44.815 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00011010600000000001 2024-02-05T04:53:44.815 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.815 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:44.815 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.815 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.816 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.816 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.fs5 52/31 ECSubWrite(tid=87, reqid=osd.11.0:85, at_version=0'0, trim_to=0'0, roll_forward_to=52'40))", 2024-02-05T04:53:44.816 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:43.961866+0000", 2024-02-05T04:53:44.816 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 0.75435038300000001, 2024-02-05T04:53:44.816 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0045966669999999996, 2024-02-05T04:53:44.817 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:44.817 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:53:44.817 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:44.817 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.817 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:44.817 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.961866+0000", 2024-02-05T04:53:44.818 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:44.818 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.818 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.818 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:44.818 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.961866+0000", 2024-02-05T04:53:44.818 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:44.818 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.818 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.819 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:44.819 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.961867+0000", 2024-02-05T04:53:44.819 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.9200000000000001e-07 2024-02-05T04:53:44.819 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.819 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.819 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:44.819 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.961876+0000", 2024-02-05T04:53:44.819 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.5140000000000007e-06 2024-02-05T04:53:44.819 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.819 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.819 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:44.819 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.961889+0000", 2024-02-05T04:53:44.819 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2737e-05 2024-02-05T04:53:44.820 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.820 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.820 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:44.820 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.961897+0000", 2024-02-05T04:53:44.820 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.3420000000000008e-06 2024-02-05T04:53:44.820 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.820 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.820 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:44.820 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.961957+0000", 2024-02-05T04:53:44.820 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.0117000000000001e-05 2024-02-05T04:53:44.820 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.820 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.821 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:44.821 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.961967+0000", 2024-02-05T04:53:44.821 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.7170000000000003e-06 2024-02-05T04:53:44.821 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.821 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.821 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:44.821 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.966427+0000", 2024-02-05T04:53:44.821 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0044603359999999996 2024-02-05T04:53:44.822 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.822 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.822 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:44.822 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.966463+0000", 2024-02-05T04:53:44.822 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.5311999999999997e-05 2024-02-05T04:53:44.822 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.822 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:44.822 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.822 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.822 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.822 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.9s6 52/31 ECSubWrite(tid=45, reqid=osd.14.0:44, at_version=0'0, trim_to=0'0, roll_forward_to=52'19))", 2024-02-05T04:53:44.822 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:43.962485+0000", 2024-02-05T04:53:44.822 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 0.75373105900000004, 2024-02-05T04:53:44.822 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0040679100000000001, 2024-02-05T04:53:44.823 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:44.823 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:53:44.823 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:44.823 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.823 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:44.823 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.962485+0000", 2024-02-05T04:53:44.823 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:44.823 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.823 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.823 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:44.823 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.962485+0000", 2024-02-05T04:53:44.823 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:44.823 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.823 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.823 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:44.824 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.962486+0000", 2024-02-05T04:53:44.824 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.5099999999999999e-07 2024-02-05T04:53:44.824 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.824 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.824 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:44.824 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.962496+0000", 2024-02-05T04:53:44.824 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.9280000000000005e-06 2024-02-05T04:53:44.824 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.824 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.824 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:44.824 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.962511+0000", 2024-02-05T04:53:44.824 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.4667999999999999e-05 2024-02-05T04:53:44.825 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.825 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.825 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:44.825 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.962520+0000", 2024-02-05T04:53:44.825 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.9469999999999992e-06 2024-02-05T04:53:44.825 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.825 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.825 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:44.825 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.962587+0000", 2024-02-05T04:53:44.825 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.7460999999999997e-05 2024-02-05T04:53:44.825 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.825 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.825 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:44.825 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.962598+0000", 2024-02-05T04:53:44.825 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0417999999999999e-05 2024-02-05T04:53:44.825 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.826 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.826 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:44.826 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.966521+0000", 2024-02-05T04:53:44.826 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0039237739999999997 2024-02-05T04:53:44.826 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:44.826 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:44.826 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:44.826 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.966553+0000", 2024-02-05T04:53:44.826 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.1962999999999999e-05 2024-02-05T04:53:44.826 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.826 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:44.826 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.826 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:44.826 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:44.826 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-02-05T04:53:44.828 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-02-05T04:53:44.829 INFO:tasks.ceph.osd.1.smithi091.stderr:2024-02-05T04:53:44.827+0000 7ff9dc283640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34275) UID: 0 2024-02-05T04:53:44.929 INFO:tasks.ceph.osd.1.smithi091.stderr:2024-02-05T04:53:44.927+0000 7ff9dc283640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34275) UID: 0 2024-02-05T04:53:44.935 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-02-05T04:53:44.935 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [], 2024-02-05T04:53:44.935 INFO:teuthology.orchestra.run.smithi150.stdout: "num_ops": 0 2024-02-05T04:53:44.935 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-02-05T04:53:44.943 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-02-05T04:53:44.950 INFO:teuthology.orchestra.run.smithi062.stderr:noscrub is set 2024-02-05T04:53:44.953 INFO:tasks.rados.rados.0.smithi196.stdout:185: done (3 left) 2024-02-05T04:53:44.953 INFO:tasks.rados.rados.0.smithi196.stdout:187: done (2 left) 2024-02-05T04:53:44.953 INFO:tasks.rados.rados.0.smithi196.stdout:190: done (1 left) 2024-02-05T04:53:44.953 INFO:tasks.rados.rados.0.smithi196.stdout:191: done (0 left) 2024-02-05T04:53:44.954 INFO:tasks.rados.rados.0.smithi196.stdout:append oid 6 current snap is 12 2024-02-05T04:53:44.954 INFO:tasks.rados.rados.0.smithi196.stdout:192: seq_num 100 ranges {1277952=524288,1802240=753664,2555904=409600} 2024-02-05T04:53:44.959 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-02-05T04:53:44.959 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T04:53:44.959 INFO:tasks.thrashosds.thrasher:Removing osd 8, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] 2024-02-05T04:53:44.959 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 8 2024-02-05T04:53:44.963 INFO:tasks.rados.rados.0.smithi196.stdout:192: writing smithi19639362-6 from 1277952 to 1802240 tid 1 2024-02-05T04:53:44.967 INFO:tasks.rados.rados.0.smithi196.stdout:192: writing smithi19639362-6 from 1802240 to 2555904 tid 2 2024-02-05T04:53:44.969 INFO:tasks.rados.rados.0.smithi196.stdout:192: writing smithi19639362-6 from 2555904 to 2965504 tid 3 2024-02-05T04:53:44.969 INFO:tasks.rados.rados.0.smithi196.stdout:193: rmattr oid 49 current snap is 12 2024-02-05T04:53:44.969 INFO:tasks.rados.rados.0.smithi196.stdout:193: done (1 left) 2024-02-05T04:53:44.970 INFO:tasks.rados.rados.0.smithi196.stdout:194: snap_create 2024-02-05T04:53:44.973 INFO:tasks.rados.rados.0.smithi196.stdout:192: finishing write tid 1 to smithi19639362-6 2024-02-05T04:53:44.978 INFO:tasks.rados.rados.0.smithi196.stdout:192: finishing write tid 2 to smithi19639362-6 2024-02-05T04:53:44.978 INFO:tasks.rados.rados.0.smithi196.stdout:192: finishing write tid 3 to smithi19639362-6 2024-02-05T04:53:44.979 INFO:tasks.rados.rados.0.smithi196.stdout:192: finishing write tid 4 to smithi19639362-6 2024-02-05T04:53:44.980 INFO:tasks.rados.rados.0.smithi196.stdout:192: finishing write tid 5 to smithi19639362-6 2024-02-05T04:53:44.980 INFO:tasks.rados.rados.0.smithi196.stdout:192: oid 6 updating version 0 to 57 2024-02-05T04:53:44.980 INFO:tasks.rados.rados.0.smithi196.stdout:192: oid 6 version 57 is already newer than 56 2024-02-05T04:53:44.980 INFO:tasks.rados.rados.0.smithi196.stdout:192: oid 6 version 57 is already newer than 54 2024-02-05T04:53:44.980 INFO:tasks.rados.rados.0.smithi196.stdout:192: oid 6 version 57 is already newer than 55 2024-02-05T04:53:44.981 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 6 v 57 (ObjNum 100 snap 12 seq_num 100) dirty exists 2024-02-05T04:53:44.981 INFO:tasks.rados.rados.0.smithi196.stdout:192: left oid 6 (ObjNum 100 snap 12 seq_num 100) 2024-02-05T04:53:45.029 INFO:tasks.ceph.osd.4.smithi062.stderr:2024-02-05T04:53:45.028+0000 7f708c308640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34689) UID: 0 2024-02-05T04:53:45.049 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-02-05T04:53:45.049 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [], 2024-02-05T04:53:45.049 INFO:teuthology.orchestra.run.smithi150.stdout: "complaint_time": 30, 2024-02-05T04:53:45.049 INFO:teuthology.orchestra.run.smithi150.stdout: "num_blocked_ops": 0 2024-02-05T04:53:45.049 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-02-05T04:53:45.057 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-02-05T04:53:45.130 INFO:tasks.ceph.osd.3.smithi196.stderr:2024-02-05T04:53:45.129+0000 7fc8817c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34605) UID: 0 2024-02-05T04:53:45.167 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-02-05T04:53:45.167 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 20, 2024-02-05T04:53:45.167 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 600, 2024-02-05T04:53:45.167 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [ 2024-02-05T04:53:45.167 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.167 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.8s4 37/31 ECSubWrite(tid=28, reqid=osd.4.0:25, at_version=0'0, trim_to=0'0, roll_forward_to=37'10))", 2024-02-05T04:53:45.167 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:29.132223+0000", 2024-02-05T04:53:45.167 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 16.032871106000002, 2024-02-05T04:53:45.167 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0037413870000000001, 2024-02-05T04:53:45.167 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:45.167 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:53:45.167 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:45.167 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.168 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:45.168 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.132223+0000", 2024-02-05T04:53:45.168 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:45.168 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.168 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.168 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:45.168 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.132223+0000", 2024-02-05T04:53:45.168 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:45.168 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.168 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.168 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:45.168 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.132224+0000", 2024-02-05T04:53:45.168 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.7000000000000003e-07 2024-02-05T04:53:45.168 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.169 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.169 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:45.169 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.132237+0000", 2024-02-05T04:53:45.169 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.3158e-05 2024-02-05T04:53:45.169 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.169 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.169 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:45.169 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.132251+0000", 2024-02-05T04:53:45.169 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.3952e-05 2024-02-05T04:53:45.169 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.169 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.169 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:45.169 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.132261+0000", 2024-02-05T04:53:45.170 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0545e-05 2024-02-05T04:53:45.170 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.170 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.170 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:45.170 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.132325+0000", 2024-02-05T04:53:45.170 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.3474999999999998e-05 2024-02-05T04:53:45.170 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.170 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.170 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:45.170 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.132333+0000", 2024-02-05T04:53:45.170 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.4999999999999999e-06 2024-02-05T04:53:45.170 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.170 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.170 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:45.171 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.135927+0000", 2024-02-05T04:53:45.171 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0035936240000000001 2024-02-05T04:53:45.171 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.171 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.171 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:45.171 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.135964+0000", 2024-02-05T04:53:45.171 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.7262999999999999e-05 2024-02-05T04:53:45.171 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:45.171 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:45.171 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:45.171 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.171 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.171 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.ds2 37/31 ECSubWrite(tid=23, reqid=client.4634.0:225, at_version=37'14, trim_to=0'0, roll_forward_to=34'12))", 2024-02-05T04:53:45.171 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:29.146452+0000", 2024-02-05T04:53:45.171 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 16.01864226, 2024-02-05T04:53:45.172 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0039709539999999996, 2024-02-05T04:53:45.172 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:45.172 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:53:45.172 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:45.172 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.172 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:45.172 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.146452+0000", 2024-02-05T04:53:45.172 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:45.172 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.172 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.172 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:45.172 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.146452+0000", 2024-02-05T04:53:45.172 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:45.173 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.173 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.173 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:45.173 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.146452+0000", 2024-02-05T04:53:45.173 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.6899999999999998e-07 2024-02-05T04:53:45.173 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.173 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.173 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:45.173 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.146940+0000", 2024-02-05T04:53:45.173 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000487746 2024-02-05T04:53:45.173 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.173 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.173 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:45.174 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.146954+0000", 2024-02-05T04:53:45.174 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.4525999999999999e-05 2024-02-05T04:53:45.174 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.174 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.174 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:45.174 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.146962+0000", 2024-02-05T04:53:45.174 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.0900000000000005e-06 2024-02-05T04:53:45.174 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.174 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.174 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:45.174 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.147053+0000", 2024-02-05T04:53:45.174 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.0932000000000003e-05 2024-02-05T04:53:45.174 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.175 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.175 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:45.175 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.147066+0000", 2024-02-05T04:53:45.175 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.3149999999999999e-05 2024-02-05T04:53:45.175 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.175 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.175 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:45.175 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.150383+0000", 2024-02-05T04:53:45.175 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0033165960000000002 2024-02-05T04:53:45.175 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.175 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.175 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:45.175 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.150422+0000", 2024-02-05T04:53:45.176 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.9444999999999999e-05 2024-02-05T04:53:45.176 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:45.176 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:45.176 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:45.176 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.176 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.176 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.1s6 37/31 ECSubWrite(tid=86, reqid=client.4634.0:247, at_version=37'15, trim_to=0'0, roll_forward_to=35'13))", 2024-02-05T04:53:45.176 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:29.192907+0000", 2024-02-05T04:53:45.176 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 15.972187185999999, 2024-02-05T04:53:45.176 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0035093020000000002, 2024-02-05T04:53:45.176 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:45.176 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:53:45.176 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:45.176 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.176 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:45.177 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.192907+0000", 2024-02-05T04:53:45.177 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:45.177 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.177 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.177 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:45.177 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.192907+0000", 2024-02-05T04:53:45.177 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:45.177 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.177 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.177 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:45.177 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.192907+0000", 2024-02-05T04:53:45.177 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.1799999999999995e-07 2024-02-05T04:53:45.177 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.177 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.177 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:45.178 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.193519+0000", 2024-02-05T04:53:45.178 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00061205299999999999 2024-02-05T04:53:45.178 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.178 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.178 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:45.178 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.193536+0000", 2024-02-05T04:53:45.178 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.6257999999999999e-05 2024-02-05T04:53:45.178 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.178 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.178 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:45.178 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.193543+0000", 2024-02-05T04:53:45.178 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.5610000000000001e-06 2024-02-05T04:53:45.178 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.178 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.178 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:45.179 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.193618+0000", 2024-02-05T04:53:45.179 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.4785999999999999e-05 2024-02-05T04:53:45.179 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.179 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.179 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:45.179 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.193628+0000", 2024-02-05T04:53:45.179 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0392e-05 2024-02-05T04:53:45.179 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.179 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.179 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:45.179 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.196365+0000", 2024-02-05T04:53:45.179 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0027367149999999998 2024-02-05T04:53:45.179 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.179 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.179 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:45.180 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.196416+0000", 2024-02-05T04:53:45.180 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.0918999999999998e-05 2024-02-05T04:53:45.180 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:45.180 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:45.180 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:45.180 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.180 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.180 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.ds2 38/31 ECSubWrite(tid=26, reqid=osd.7.0:26, at_version=38'16, trim_to=0'0, roll_forward_to=37'14))", 2024-02-05T04:53:45.180 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:30.134818+0000", 2024-02-05T04:53:45.180 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 15.030275937000001, 2024-02-05T04:53:45.180 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0031531430000000002, 2024-02-05T04:53:45.180 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:45.180 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:53:45.180 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:45.181 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.181 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:45.181 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.134818+0000", 2024-02-05T04:53:45.181 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:45.181 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.181 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.181 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:45.181 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.134818+0000", 2024-02-05T04:53:45.181 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:45.181 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.181 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.181 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:45.181 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.134818+0000", 2024-02-05T04:53:45.181 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.3300000000000001e-07 2024-02-05T04:53:45.182 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.182 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.182 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:45.182 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.134839+0000", 2024-02-05T04:53:45.182 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.1280999999999998e-05 2024-02-05T04:53:45.182 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.182 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.182 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:45.182 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.134931+0000", 2024-02-05T04:53:45.182 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.1266000000000003e-05 2024-02-05T04:53:45.182 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.182 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.182 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:45.182 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.134943+0000", 2024-02-05T04:53:45.183 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1994000000000001e-05 2024-02-05T04:53:45.183 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.183 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.183 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:45.183 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.135021+0000", 2024-02-05T04:53:45.183 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.8563e-05 2024-02-05T04:53:45.183 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.183 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.183 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:45.183 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.135031+0000", 2024-02-05T04:53:45.183 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0309000000000001e-05 2024-02-05T04:53:45.183 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.183 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.184 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:45.184 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.137914+0000", 2024-02-05T04:53:45.184 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0028829279999999999 2024-02-05T04:53:45.184 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.184 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.184 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:45.184 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.137971+0000", 2024-02-05T04:53:45.184 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.6569e-05 2024-02-05T04:53:45.184 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:45.184 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:45.184 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:45.184 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.184 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.184 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.1s6 38/31 ECSubWrite(tid=89, reqid=osd.13.0:89, at_version=38'17, trim_to=0'0, roll_forward_to=37'15))", 2024-02-05T04:53:45.185 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:30.140056+0000", 2024-02-05T04:53:45.185 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 15.025037767000001, 2024-02-05T04:53:45.185 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0047998199999999998, 2024-02-05T04:53:45.185 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:45.185 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:53:45.185 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:45.185 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.185 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:45.185 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.140056+0000", 2024-02-05T04:53:45.185 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:45.185 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.185 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.186 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:45.186 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.140056+0000", 2024-02-05T04:53:45.186 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:45.186 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.186 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.186 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:45.186 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.140056+0000", 2024-02-05T04:53:45.186 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.16e-07 2024-02-05T04:53:45.186 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.186 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.186 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:45.186 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.140075+0000", 2024-02-05T04:53:45.186 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.9177999999999998e-05 2024-02-05T04:53:45.187 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.187 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.187 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:45.187 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.140086+0000", 2024-02-05T04:53:45.187 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0815000000000001e-05 2024-02-05T04:53:45.187 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.187 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.187 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:45.187 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.140095+0000", 2024-02-05T04:53:45.187 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.3010000000000007e-06 2024-02-05T04:53:45.188 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.188 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.188 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:45.188 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.140173+0000", 2024-02-05T04:53:45.188 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.8189000000000006e-05 2024-02-05T04:53:45.188 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.188 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.188 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:45.188 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.140184+0000", 2024-02-05T04:53:45.188 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1041e-05 2024-02-05T04:53:45.188 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.189 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.189 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:45.189 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.144822+0000", 2024-02-05T04:53:45.189 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.004637899 2024-02-05T04:53:45.189 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.189 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.189 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:45.189 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.144856+0000", 2024-02-05T04:53:45.189 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.4180999999999997e-05 2024-02-05T04:53:45.189 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:45.189 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:45.189 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:45.190 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.190 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.190 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.ds2 38/31 ECSubWrite(tid=27, reqid=osd.7.0:26, at_version=0'0, trim_to=0'0, roll_forward_to=38'16))", 2024-02-05T04:53:45.190 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:30.141585+0000", 2024-02-05T04:53:45.190 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 15.023508941999999, 2024-02-05T04:53:45.190 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0029951919999999998, 2024-02-05T04:53:45.190 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:45.190 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:53:45.190 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:45.190 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.190 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:45.190 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.141585+0000", 2024-02-05T04:53:45.190 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:45.190 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.190 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.191 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:45.191 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.141585+0000", 2024-02-05T04:53:45.191 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:45.191 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.191 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.191 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:45.191 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.141586+0000", 2024-02-05T04:53:45.191 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.2800000000000005e-07 2024-02-05T04:53:45.191 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.191 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.191 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:45.191 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.141593+0000", 2024-02-05T04:53:45.191 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.3799999999999996e-06 2024-02-05T04:53:45.191 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.192 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.192 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:45.192 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.141604+0000", 2024-02-05T04:53:45.192 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0419e-05 2024-02-05T04:53:45.192 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.192 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.192 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:45.192 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.141611+0000", 2024-02-05T04:53:45.192 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.3069999999999997e-06 2024-02-05T04:53:45.192 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.192 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.192 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:45.192 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.141662+0000", 2024-02-05T04:53:45.193 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.1020000000000003e-05 2024-02-05T04:53:45.193 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.193 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.193 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:45.193 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.141672+0000", 2024-02-05T04:53:45.193 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0531e-05 2024-02-05T04:53:45.193 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.193 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.193 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:45.193 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.144546+0000", 2024-02-05T04:53:45.193 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0028737350000000001 2024-02-05T04:53:45.193 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.194 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.194 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:45.194 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.144580+0000", 2024-02-05T04:53:45.194 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.3871999999999997e-05 2024-02-05T04:53:45.194 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:45.194 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:45.194 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:45.194 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.194 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.194 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.1s6 38/31 ECSubWrite(tid=91, reqid=osd.13.0:89, at_version=0'0, trim_to=0'0, roll_forward_to=38'17))", 2024-02-05T04:53:45.194 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:30.148719+0000", 2024-02-05T04:53:45.194 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 15.016375200000001, 2024-02-05T04:53:45.194 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.003480098, 2024-02-05T04:53:45.194 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:45.194 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:53:45.194 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:45.195 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.195 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:45.195 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.148719+0000", 2024-02-05T04:53:45.195 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:45.195 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.195 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.195 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:45.195 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.148719+0000", 2024-02-05T04:53:45.195 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:45.195 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.195 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.195 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:45.195 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.148719+0000", 2024-02-05T04:53:45.195 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.8999999999999995e-07 2024-02-05T04:53:45.196 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.196 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.196 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:45.196 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.148729+0000", 2024-02-05T04:53:45.196 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.3999999999999998e-06 2024-02-05T04:53:45.196 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.196 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.196 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:45.196 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.148759+0000", 2024-02-05T04:53:45.196 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.0090000000000002e-05 2024-02-05T04:53:45.196 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.196 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.196 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:45.196 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.148785+0000", 2024-02-05T04:53:45.196 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.6424999999999999e-05 2024-02-05T04:53:45.197 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.197 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.197 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:45.197 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.148860+0000", 2024-02-05T04:53:45.197 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.4311000000000001e-05 2024-02-05T04:53:45.197 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.197 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.197 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:45.197 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.148869+0000", 2024-02-05T04:53:45.197 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.7680000000000003e-06 2024-02-05T04:53:45.197 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.197 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.197 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:45.197 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.152163+0000", 2024-02-05T04:53:45.197 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0032932999999999999 2024-02-05T04:53:45.198 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.198 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.198 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:45.198 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.152199+0000", 2024-02-05T04:53:45.198 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.6013999999999998e-05 2024-02-05T04:53:45.198 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:45.198 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:45.198 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:45.198 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.198 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.198 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.fs2 38/31 ECSubWrite(tid=51, reqid=client.4634.0:290, at_version=38'17, trim_to=0'0, roll_forward_to=34'16))", 2024-02-05T04:53:45.198 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:30.387989+0000", 2024-02-05T04:53:45.198 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 14.777104327, 2024-02-05T04:53:45.198 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0034901139999999999, 2024-02-05T04:53:45.198 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:45.198 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:53:45.198 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:45.199 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.199 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:45.199 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.387989+0000", 2024-02-05T04:53:45.199 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:45.199 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.199 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.199 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:45.199 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.387989+0000", 2024-02-05T04:53:45.199 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:45.199 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.199 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.199 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:45.199 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.387990+0000", 2024-02-05T04:53:45.199 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.0699999999999997e-07 2024-02-05T04:53:45.199 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.200 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.200 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:45.200 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.388368+0000", 2024-02-05T04:53:45.200 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00037788900000000001 2024-02-05T04:53:45.200 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.200 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.200 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:45.200 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.388382+0000", 2024-02-05T04:53:45.200 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.3974999999999999e-05 2024-02-05T04:53:45.200 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.200 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.200 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:45.200 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.388391+0000", 2024-02-05T04:53:45.201 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.3309999999999993e-06 2024-02-05T04:53:45.201 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.201 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.201 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:45.201 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.388451+0000", 2024-02-05T04:53:45.201 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.9488999999999999e-05 2024-02-05T04:53:45.201 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.201 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.201 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:45.201 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.388462+0000", 2024-02-05T04:53:45.201 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1031e-05 2024-02-05T04:53:45.201 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.201 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.201 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:45.201 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.391457+0000", 2024-02-05T04:53:45.201 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0029955699999999999 2024-02-05T04:53:45.202 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.202 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.202 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:45.202 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.391480+0000", 2024-02-05T04:53:45.202 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.2322000000000001e-05 2024-02-05T04:53:45.202 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:45.202 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:45.202 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:45.202 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.202 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.202 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.fs2 38/31 ECSubWrite(tid=52, reqid=client.4634.0:291, at_version=38'18, trim_to=0'0, roll_forward_to=34'16))", 2024-02-05T04:53:45.202 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:30.389407+0000", 2024-02-05T04:53:45.202 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 14.775686961, 2024-02-05T04:53:45.202 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0043544739999999997, 2024-02-05T04:53:45.202 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:45.202 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:53:45.203 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:45.203 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.203 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:45.203 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.389407+0000", 2024-02-05T04:53:45.203 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:45.203 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.203 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.203 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:45.203 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.389407+0000", 2024-02-05T04:53:45.203 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:45.203 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.203 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.203 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:45.203 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.389407+0000", 2024-02-05T04:53:45.203 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.3600000000000004e-07 2024-02-05T04:53:45.204 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.204 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.204 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:45.204 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.389665+0000", 2024-02-05T04:53:45.204 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00025736999999999998 2024-02-05T04:53:45.204 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.204 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.204 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:45.204 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.389677+0000", 2024-02-05T04:53:45.204 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1837e-05 2024-02-05T04:53:45.204 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.204 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.204 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:45.204 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.389684+0000", 2024-02-05T04:53:45.204 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.9319999999999996e-06 2024-02-05T04:53:45.205 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.205 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.205 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:45.205 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.389732+0000", 2024-02-05T04:53:45.205 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.7336999999999998e-05 2024-02-05T04:53:45.205 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.205 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.205 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:45.205 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.389737+0000", 2024-02-05T04:53:45.205 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.3210000000000004e-06 2024-02-05T04:53:45.205 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.205 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.205 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:45.205 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.393736+0000", 2024-02-05T04:53:45.205 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0039987139999999996 2024-02-05T04:53:45.206 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.206 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.206 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:45.206 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.393761+0000", 2024-02-05T04:53:45.206 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.5426999999999999e-05 2024-02-05T04:53:45.206 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:45.206 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:45.206 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:45.206 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.206 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.206 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.fs2 38/31 ECSubWrite(tid=53, reqid=client.4634.0:292, at_version=38'19, trim_to=0'0, roll_forward_to=34'16))", 2024-02-05T04:53:45.206 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:30.389969+0000", 2024-02-05T04:53:45.207 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 14.775124811, 2024-02-05T04:53:45.207 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0041055889999999998, 2024-02-05T04:53:45.207 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:45.207 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:53:45.207 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:45.207 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.207 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:45.207 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.389969+0000", 2024-02-05T04:53:45.207 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:45.207 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.207 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.207 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:45.207 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.389969+0000", 2024-02-05T04:53:45.207 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:45.207 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.207 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.208 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:45.208 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.389969+0000", 2024-02-05T04:53:45.208 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.9399999999999995e-07 2024-02-05T04:53:45.208 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.208 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.208 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:45.208 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.390019+0000", 2024-02-05T04:53:45.208 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.9675000000000001e-05 2024-02-05T04:53:45.208 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.208 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.208 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:45.208 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.390030+0000", 2024-02-05T04:53:45.208 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1327e-05 2024-02-05T04:53:45.209 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.209 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.209 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:45.209 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.390037+0000", 2024-02-05T04:53:45.209 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.002e-06 2024-02-05T04:53:45.209 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.209 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.209 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:45.209 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.391196+0000", 2024-02-05T04:53:45.209 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.001158807 2024-02-05T04:53:45.209 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.209 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.209 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:45.209 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.391204+0000", 2024-02-05T04:53:45.209 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.2099999999999993e-06 2024-02-05T04:53:45.210 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.210 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.210 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:45.210 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.394050+0000", 2024-02-05T04:53:45.210 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0028459290000000001 2024-02-05T04:53:45.210 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.210 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.210 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:45.210 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.394075+0000", 2024-02-05T04:53:45.210 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.4145e-05 2024-02-05T04:53:45.210 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:45.210 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:45.210 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:45.210 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.211 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.211 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.fs2 38/31 ECSubWrite(tid=54, reqid=client.4634.0:293, at_version=38'20, trim_to=0'0, roll_forward_to=34'16))", 2024-02-05T04:53:45.211 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:30.390508+0000", 2024-02-05T04:53:45.211 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 14.774585761999999, 2024-02-05T04:53:45.211 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0035808960000000001, 2024-02-05T04:53:45.211 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:45.211 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:53:45.211 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:45.211 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.211 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:45.211 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.390508+0000", 2024-02-05T04:53:45.211 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:45.211 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.211 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.211 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:45.211 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.390508+0000", 2024-02-05T04:53:45.211 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:45.212 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.212 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.212 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:45.212 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.390509+0000", 2024-02-05T04:53:45.212 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.0699999999999997e-07 2024-02-05T04:53:45.212 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.212 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.212 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:45.212 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.390521+0000", 2024-02-05T04:53:45.212 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.288e-05 2024-02-05T04:53:45.212 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.212 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.212 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:45.213 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.390538+0000", 2024-02-05T04:53:45.213 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.6782999999999998e-05 2024-02-05T04:53:45.213 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.213 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.213 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:45.213 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.390547+0000", 2024-02-05T04:53:45.213 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.8349999999999993e-06 2024-02-05T04:53:45.213 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.213 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.213 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:45.213 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.391508+0000", 2024-02-05T04:53:45.213 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00096128100000000003 2024-02-05T04:53:45.213 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.214 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.214 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:45.214 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.391514+0000", 2024-02-05T04:53:45.214 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.6609999999999997e-06 2024-02-05T04:53:45.214 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.214 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.214 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:45.214 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.394077+0000", 2024-02-05T04:53:45.214 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0025628690000000002 2024-02-05T04:53:45.214 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.214 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.214 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:45.214 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:30.394089+0000", 2024-02-05T04:53:45.214 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.208e-05 2024-02-05T04:53:45.215 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:45.215 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:45.215 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:45.215 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.215 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.215 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.1s6 42/31 ECSubWrite(tid=135, reqid=client.4634.0:379, at_version=42'27, trim_to=0'0, roll_forward_to=40'25))", 2024-02-05T04:53:45.215 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:34.167718+0000", 2024-02-05T04:53:45.215 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 10.99737562, 2024-02-05T04:53:45.215 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.002722435, 2024-02-05T04:53:45.215 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:45.215 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:53:45.215 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:45.215 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.215 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:45.216 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.167718+0000", 2024-02-05T04:53:45.216 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:45.216 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.216 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.216 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:45.216 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.167718+0000", 2024-02-05T04:53:45.216 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:45.216 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.216 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.216 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:45.216 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.167718+0000", 2024-02-05T04:53:45.216 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.1799999999999999e-07 2024-02-05T04:53:45.216 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.217 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.217 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:45.217 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.168545+0000", 2024-02-05T04:53:45.217 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00082666700000000001 2024-02-05T04:53:45.217 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.217 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.217 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:45.217 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.168572+0000", 2024-02-05T04:53:45.217 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.7059e-05 2024-02-05T04:53:45.217 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.217 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.217 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:45.217 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.168589+0000", 2024-02-05T04:53:45.217 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.6764000000000001e-05 2024-02-05T04:53:45.218 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.218 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.218 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:45.218 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.168671+0000", 2024-02-05T04:53:45.218 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.2257999999999997e-05 2024-02-05T04:53:45.218 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.218 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.218 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:45.218 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.168684+0000", 2024-02-05T04:53:45.218 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2873e-05 2024-02-05T04:53:45.218 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.218 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.218 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:45.218 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.170374+0000", 2024-02-05T04:53:45.219 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.001690141 2024-02-05T04:53:45.219 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.219 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.219 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:45.219 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.170441+0000", 2024-02-05T04:53:45.219 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.6455e-05 2024-02-05T04:53:45.219 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:45.219 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:45.219 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:45.219 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.219 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.219 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.8s4 42/31 ECSubWrite(tid=60, reqid=client.4634.0:390, at_version=42'17, trim_to=0'0, roll_forward_to=40'15))", 2024-02-05T04:53:45.219 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:34.180833+0000", 2024-02-05T04:53:45.219 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 10.984260376, 2024-02-05T04:53:45.220 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0026498310000000001, 2024-02-05T04:53:45.220 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:45.220 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:53:45.220 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:45.220 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.220 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:45.220 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.180833+0000", 2024-02-05T04:53:45.220 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:45.220 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.220 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.220 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:45.220 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.180833+0000", 2024-02-05T04:53:45.220 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:45.220 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.220 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.221 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:45.221 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.180836+0000", 2024-02-05T04:53:45.221 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.0479999999999999e-06 2024-02-05T04:53:45.221 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.221 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.221 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:45.221 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.181748+0000", 2024-02-05T04:53:45.221 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00091126499999999997 2024-02-05T04:53:45.221 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.221 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.221 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:45.221 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.181765+0000", 2024-02-05T04:53:45.221 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.7601e-05 2024-02-05T04:53:45.221 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.221 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.221 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:45.222 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.181774+0000", 2024-02-05T04:53:45.222 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.0690000000000008e-06 2024-02-05T04:53:45.222 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.222 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.222 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:45.222 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.181850+0000", 2024-02-05T04:53:45.222 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.5444999999999993e-05 2024-02-05T04:53:45.222 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.222 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.222 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:45.222 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.181860+0000", 2024-02-05T04:53:45.222 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0122000000000001e-05 2024-02-05T04:53:45.222 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.222 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.223 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:45.223 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.183454+0000", 2024-02-05T04:53:45.223 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0015938409999999999 2024-02-05T04:53:45.223 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.223 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.223 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:45.223 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.183483+0000", 2024-02-05T04:53:45.223 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.9439999999999999e-05 2024-02-05T04:53:45.223 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:45.223 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:45.223 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:45.223 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.223 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.223 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.8s4 43/31 ECSubWrite(tid=62, reqid=osd.4.0:62, at_version=43'19, trim_to=0'0, roll_forward_to=42'17))", 2024-02-05T04:53:45.223 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:35.172391+0000", 2024-02-05T04:53:45.224 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 9.9927024000000007, 2024-02-05T04:53:45.224 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0036785099999999999, 2024-02-05T04:53:45.224 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:45.224 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:53:45.224 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:45.224 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.224 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:45.224 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.172391+0000", 2024-02-05T04:53:45.224 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:45.224 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.224 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.224 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:45.224 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.172391+0000", 2024-02-05T04:53:45.224 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:45.224 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.225 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.225 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:45.225 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.172392+0000", 2024-02-05T04:53:45.225 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.0100000000000001e-07 2024-02-05T04:53:45.225 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.225 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.225 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:45.225 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.172413+0000", 2024-02-05T04:53:45.225 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.1863999999999999e-05 2024-02-05T04:53:45.225 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.225 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.225 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:45.226 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.172424+0000", 2024-02-05T04:53:45.226 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0825999999999999e-05 2024-02-05T04:53:45.226 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.226 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.226 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:45.226 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.172433+0000", 2024-02-05T04:53:45.226 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.0759999999999992e-06 2024-02-05T04:53:45.226 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.226 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.226 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:45.226 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.172533+0000", 2024-02-05T04:53:45.227 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000100102 2024-02-05T04:53:45.227 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.227 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.227 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:45.227 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.172546+0000", 2024-02-05T04:53:45.227 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2952999999999999e-05 2024-02-05T04:53:45.227 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.227 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.227 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:45.227 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.176020+0000", 2024-02-05T04:53:45.227 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0034737549999999998 2024-02-05T04:53:45.227 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.227 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.227 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:45.228 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.176070+0000", 2024-02-05T04:53:45.228 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.9733e-05 2024-02-05T04:53:45.228 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:45.228 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:45.228 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:45.228 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.228 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.228 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.1s6 43/31 ECSubWrite(tid=140, reqid=osd.13.0:140, at_version=43'29, trim_to=0'0, roll_forward_to=42'27))", 2024-02-05T04:53:45.228 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:35.182104+0000", 2024-02-05T04:53:45.228 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 9.9829900089999999, 2024-02-05T04:53:45.228 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0056319439999999998, 2024-02-05T04:53:45.228 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:45.229 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:53:45.229 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:45.229 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.229 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:45.229 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.182104+0000", 2024-02-05T04:53:45.229 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:45.229 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.229 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.229 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:45.229 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.182104+0000", 2024-02-05T04:53:45.229 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:45.229 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.229 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.229 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:45.229 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.182104+0000", 2024-02-05T04:53:45.230 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.3699999999999999e-07 2024-02-05T04:53:45.230 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.230 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.230 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:45.230 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.182127+0000", 2024-02-05T04:53:45.230 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.2994999999999999e-05 2024-02-05T04:53:45.230 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.230 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.230 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:45.230 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.182140+0000", 2024-02-05T04:53:45.230 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2965999999999999e-05 2024-02-05T04:53:45.230 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.230 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.231 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:45.231 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.182150+0000", 2024-02-05T04:53:45.231 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0423e-05 2024-02-05T04:53:45.231 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.231 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.231 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:45.231 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.182216+0000", 2024-02-05T04:53:45.231 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.5209999999999994e-05 2024-02-05T04:53:45.231 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.231 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.231 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:45.231 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.182225+0000", 2024-02-05T04:53:45.232 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.4250000000000004e-06 2024-02-05T04:53:45.232 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.232 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.232 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:45.232 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.187667+0000", 2024-02-05T04:53:45.232 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0054421160000000003 2024-02-05T04:53:45.232 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.232 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.232 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:45.232 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.187736+0000", 2024-02-05T04:53:45.232 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.8571999999999993e-05 2024-02-05T04:53:45.232 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:45.232 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:45.233 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:45.233 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.233 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.233 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.ds2 43/31 ECSubWrite(tid=44, reqid=client.4634.0:401, at_version=43'24, trim_to=0'0, roll_forward_to=38'22))", 2024-02-05T04:53:45.233 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:35.183613+0000", 2024-02-05T04:53:45.233 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 9.981480715, 2024-02-05T04:53:45.233 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0072371980000000002, 2024-02-05T04:53:45.233 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:45.233 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:53:45.233 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:45.233 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.233 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:45.233 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.183613+0000", 2024-02-05T04:53:45.234 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:45.234 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.234 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.234 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:45.234 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.183613+0000", 2024-02-05T04:53:45.234 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:45.234 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.234 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.234 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:45.234 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.183613+0000", 2024-02-05T04:53:45.234 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.5600000000000002e-07 2024-02-05T04:53:45.234 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.234 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.234 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:45.234 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.183636+0000", 2024-02-05T04:53:45.235 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.2248000000000001e-05 2024-02-05T04:53:45.235 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.235 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.235 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:45.235 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.183648+0000", 2024-02-05T04:53:45.235 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2666e-05 2024-02-05T04:53:45.235 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.235 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.235 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:45.235 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.183659+0000", 2024-02-05T04:53:45.235 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1131999999999999e-05 2024-02-05T04:53:45.235 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.235 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.235 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:45.235 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.183713+0000", 2024-02-05T04:53:45.236 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.3400999999999999e-05 2024-02-05T04:53:45.236 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.236 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.236 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:45.236 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.183721+0000", 2024-02-05T04:53:45.236 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.1999999999999994e-06 2024-02-05T04:53:45.236 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.236 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.236 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:45.236 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.190813+0000", 2024-02-05T04:53:45.236 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0070917660000000002 2024-02-05T04:53:45.236 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.236 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.237 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:45.237 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.190850+0000", 2024-02-05T04:53:45.237 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.7528999999999999e-05 2024-02-05T04:53:45.237 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:45.237 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:45.237 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:45.237 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.237 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.237 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.1s6 43/31 ECSubWrite(tid=142, reqid=osd.13.0:142, at_version=43'31, trim_to=0'0, roll_forward_to=42'27))", 2024-02-05T04:53:45.237 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:35.183877+0000", 2024-02-05T04:53:45.237 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 9.9812165820000001, 2024-02-05T04:53:45.237 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.006942312, 2024-02-05T04:53:45.237 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:45.237 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:53:45.238 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:45.238 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.238 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:45.238 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.183877+0000", 2024-02-05T04:53:45.238 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:45.238 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.238 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.238 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:45.238 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.183877+0000", 2024-02-05T04:53:45.238 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:45.238 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.238 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.238 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:45.238 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.183878+0000", 2024-02-05T04:53:45.238 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.5000000000000002e-07 2024-02-05T04:53:45.239 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.239 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.239 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:45.239 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.183901+0000", 2024-02-05T04:53:45.239 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.2895999999999999e-05 2024-02-05T04:53:45.239 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.239 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.239 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:45.239 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.183917+0000", 2024-02-05T04:53:45.239 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.6345e-05 2024-02-05T04:53:45.239 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.239 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.239 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:45.239 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.183930+0000", 2024-02-05T04:53:45.240 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.3308e-05 2024-02-05T04:53:45.240 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.240 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.240 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:45.240 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.183993+0000", 2024-02-05T04:53:45.240 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.2979999999999997e-05 2024-02-05T04:53:45.240 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.240 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.240 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:45.240 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.184008+0000", 2024-02-05T04:53:45.240 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.4861000000000001e-05 2024-02-05T04:53:45.240 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.240 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.240 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:45.241 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.190786+0000", 2024-02-05T04:53:45.241 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0067771979999999999 2024-02-05T04:53:45.241 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.241 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.241 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:45.241 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.190820+0000", 2024-02-05T04:53:45.241 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.3973999999999997e-05 2024-02-05T04:53:45.241 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:45.241 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:45.241 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:45.241 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.241 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.241 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.fs2 52/31 ECSubWrite(tid=84, reqid=osd.11.0:84, at_version=52'38, trim_to=0'0, roll_forward_to=51'36))", 2024-02-05T04:53:45.241 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:43.954455+0000", 2024-02-05T04:53:45.242 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 1.210639271, 2024-02-05T04:53:45.242 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.002661531, 2024-02-05T04:53:45.242 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:45.242 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:53:45.242 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:45.242 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.242 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:45.242 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.954455+0000", 2024-02-05T04:53:45.242 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:45.242 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.242 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.242 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:45.242 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.954455+0000", 2024-02-05T04:53:45.242 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:45.243 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.243 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.243 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:45.243 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.954455+0000", 2024-02-05T04:53:45.243 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.4200000000000001e-07 2024-02-05T04:53:45.243 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.243 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.243 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:45.243 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.954478+0000", 2024-02-05T04:53:45.243 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.2869000000000001e-05 2024-02-05T04:53:45.243 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.243 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.243 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:45.243 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.954488+0000", 2024-02-05T04:53:45.243 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0363000000000001e-05 2024-02-05T04:53:45.244 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.244 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.244 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:45.244 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.954496+0000", 2024-02-05T04:53:45.244 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.1680000000000004e-06 2024-02-05T04:53:45.244 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.244 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.244 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:45.244 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.954600+0000", 2024-02-05T04:53:45.244 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000103378 2024-02-05T04:53:45.244 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.244 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.244 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:45.244 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.954612+0000", 2024-02-05T04:53:45.244 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2043e-05 2024-02-05T04:53:45.245 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.245 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.245 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:45.245 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.957047+0000", 2024-02-05T04:53:45.245 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.002435272 2024-02-05T04:53:45.245 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.245 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.245 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:45.245 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.957116+0000", 2024-02-05T04:53:45.245 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.8996000000000001e-05 2024-02-05T04:53:45.245 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:45.245 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:45.245 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:45.245 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.245 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.246 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.fs2 52/31 ECSubWrite(tid=85, reqid=osd.11.0:85, at_version=52'40, trim_to=0'0, roll_forward_to=51'36))", 2024-02-05T04:53:45.246 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:43.954828+0000", 2024-02-05T04:53:45.246 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 1.2102656869999999, 2024-02-05T04:53:45.246 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0026379929999999999, 2024-02-05T04:53:45.246 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:45.246 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:53:45.246 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:45.246 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.246 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:45.246 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.954828+0000", 2024-02-05T04:53:45.246 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:45.246 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.246 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.246 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:45.247 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.954828+0000", 2024-02-05T04:53:45.247 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:45.247 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.247 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.247 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:45.247 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.954829+0000", 2024-02-05T04:53:45.247 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.6400000000000002e-07 2024-02-05T04:53:45.247 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.247 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.247 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:45.247 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.954846+0000", 2024-02-05T04:53:45.247 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.7805999999999999e-05 2024-02-05T04:53:45.247 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.247 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.247 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:45.248 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.954857+0000", 2024-02-05T04:53:45.248 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.007e-05 2024-02-05T04:53:45.248 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.248 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.248 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:45.248 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.954864+0000", 2024-02-05T04:53:45.248 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.0029999999999997e-06 2024-02-05T04:53:45.248 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.248 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.248 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:45.248 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.955096+0000", 2024-02-05T04:53:45.248 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00023214700000000001 2024-02-05T04:53:45.248 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.248 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.248 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:45.248 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.955105+0000", 2024-02-05T04:53:45.249 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.5699999999999999e-06 2024-02-05T04:53:45.249 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.249 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.249 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:45.249 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.957437+0000", 2024-02-05T04:53:45.249 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0023314859999999998 2024-02-05T04:53:45.249 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.249 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.249 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:45.249 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.957466+0000", 2024-02-05T04:53:45.249 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.9346999999999999e-05 2024-02-05T04:53:45.249 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:45.249 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:45.249 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:45.249 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.250 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.250 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.fs2 52/31 ECSubWrite(tid=87, reqid=osd.11.0:85, at_version=0'0, trim_to=0'0, roll_forward_to=52'40))", 2024-02-05T04:53:45.250 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:43.961881+0000", 2024-02-05T04:53:45.250 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 1.2032123189999999, 2024-02-05T04:53:45.250 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0069676850000000004, 2024-02-05T04:53:45.250 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:45.250 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:53:45.250 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:45.250 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.250 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:45.250 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.961881+0000", 2024-02-05T04:53:45.250 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:45.250 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.250 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.251 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:45.251 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.961881+0000", 2024-02-05T04:53:45.251 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:45.251 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.251 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.251 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:45.251 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.961882+0000", 2024-02-05T04:53:45.251 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.8599999999999998e-07 2024-02-05T04:53:45.251 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.251 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.251 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:45.251 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.961893+0000", 2024-02-05T04:53:45.251 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1019000000000001e-05 2024-02-05T04:53:45.251 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.252 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.252 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:45.252 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.961903+0000", 2024-02-05T04:53:45.252 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.8840000000000006e-06 2024-02-05T04:53:45.252 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.252 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.252 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:45.252 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.961910+0000", 2024-02-05T04:53:45.252 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.1579999999999999e-06 2024-02-05T04:53:45.252 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.252 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.252 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:45.252 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.961981+0000", 2024-02-05T04:53:45.253 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.1058999999999998e-05 2024-02-05T04:53:45.253 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.253 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.253 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:45.253 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.961990+0000", 2024-02-05T04:53:45.253 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.9979999999999992e-06 2024-02-05T04:53:45.253 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.253 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.253 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:45.253 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.968815+0000", 2024-02-05T04:53:45.253 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0068249920000000002 2024-02-05T04:53:45.253 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:45.253 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:45.253 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:45.253 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.968849+0000", 2024-02-05T04:53:45.254 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.3989000000000003e-05 2024-02-05T04:53:45.254 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:45.254 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:45.254 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:45.254 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:45.254 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:45.254 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-02-05T04:53:45.256 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-02-05T04:53:45.256 INFO:tasks.ceph.osd.10.smithi150.stderr:2024-02-05T04:53:45.254+0000 7f3075a4d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 34018) UID: 0 2024-02-05T04:53:45.271 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-02-05T04:53:45.270+0000 7fe5ff27b640 -1 mon.a@0(leader).osd e53 definitely_dead 0 2024-02-05T04:53:45.356 INFO:tasks.ceph.osd.15.smithi196.stderr:2024-02-05T04:53:45.355+0000 7f1facc45640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 34611) UID: 0 2024-02-05T04:53:45.373 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T04:53:45.373 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-02-05T04:53:45.374 INFO:teuthology.orchestra.run.smithi196.stdout: "num_ops": 0 2024-02-05T04:53:45.374 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T04:53:45.383 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-02-05T04:53:45.457 INFO:tasks.ceph.osd.8.smithi062.stderr:2024-02-05T04:53:45.456+0000 7f9385bf2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 34693) UID: 0 2024-02-05T04:53:45.530 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T04:53:45.530 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-02-05T04:53:45.530 INFO:teuthology.orchestra.run.smithi196.stdout: "complaint_time": 30, 2024-02-05T04:53:45.530 INFO:teuthology.orchestra.run.smithi196.stdout: "num_blocked_ops": 0 2024-02-05T04:53:45.530 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T04:53:45.539 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-02-05T04:53:45.558 INFO:tasks.ceph.osd.13.smithi091.stderr:2024-02-05T04:53:45.557+0000 7f526fd4d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 34281) UID: 0 2024-02-05T04:53:45.645 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T04:53:45.645 INFO:teuthology.orchestra.run.smithi196.stdout: "size": 20, 2024-02-05T04:53:45.646 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 600, 2024-02-05T04:53:45.646 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-02-05T04:53:45.646 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.646 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.3s6 37/31 ECSubWrite(tid=27, reqid=osd.4.0:26, at_version=0'0, trim_to=0'0, roll_forward_to=37'10))", 2024-02-05T04:53:45.646 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:29.131110+0000", 2024-02-05T04:53:45.646 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 16.512708528000001, 2024-02-05T04:53:45.646 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002311582, 2024-02-05T04:53:45.646 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:45.646 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:45.646 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:45.646 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.646 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:45.646 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.131110+0000", 2024-02-05T04:53:45.646 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:45.647 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.647 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.647 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:45.647 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.131110+0000", 2024-02-05T04:53:45.647 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:45.647 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.647 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.647 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:45.647 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.131111+0000", 2024-02-05T04:53:45.647 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6099999999999999e-07 2024-02-05T04:53:45.647 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.647 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.647 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:45.647 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.131285+0000", 2024-02-05T04:53:45.648 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00017457799999999999 2024-02-05T04:53:45.648 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.648 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.648 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:45.648 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.131298+0000", 2024-02-05T04:53:45.648 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2500999999999999e-05 2024-02-05T04:53:45.648 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.648 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.648 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:45.648 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.131307+0000", 2024-02-05T04:53:45.648 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1109999999999997e-06 2024-02-05T04:53:45.648 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.648 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.648 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:45.649 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.131396+0000", 2024-02-05T04:53:45.649 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.9377999999999997e-05 2024-02-05T04:53:45.649 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.649 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.650 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:45.650 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.131405+0000", 2024-02-05T04:53:45.650 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4090000000000003e-06 2024-02-05T04:53:45.650 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.650 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.650 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:45.650 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.133378+0000", 2024-02-05T04:53:45.650 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0019734090000000002 2024-02-05T04:53:45.650 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.650 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.650 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:45.650 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.133421+0000", 2024-02-05T04:53:45.650 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3235000000000002e-05 2024-02-05T04:53:45.650 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:45.651 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:45.651 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:45.651 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.651 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.651 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.3s6 37/31 ECSubWrite(tid=29, reqid=client.4634.0:218, at_version=37'12, trim_to=0'0, roll_forward_to=37'10))", 2024-02-05T04:53:45.651 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:29.134419+0000", 2024-02-05T04:53:45.651 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 16.509399206000001, 2024-02-05T04:53:45.651 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0029388539999999999, 2024-02-05T04:53:45.651 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:45.651 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:45.651 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:45.651 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.651 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:45.651 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.134419+0000", 2024-02-05T04:53:45.651 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:45.652 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.652 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.652 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:45.652 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.134419+0000", 2024-02-05T04:53:45.652 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:45.652 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.652 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.652 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:45.652 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.134420+0000", 2024-02-05T04:53:45.652 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2900000000000003e-07 2024-02-05T04:53:45.652 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.652 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.653 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:45.653 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135211+0000", 2024-02-05T04:53:45.653 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00079167699999999996 2024-02-05T04:53:45.653 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.653 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.653 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:45.653 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135234+0000", 2024-02-05T04:53:45.653 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2869000000000001e-05 2024-02-05T04:53:45.653 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.653 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.654 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:45.654 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135287+0000", 2024-02-05T04:53:45.654 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2948999999999997e-05 2024-02-05T04:53:45.654 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.654 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.654 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:45.654 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135370+0000", 2024-02-05T04:53:45.654 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.2416000000000005e-05 2024-02-05T04:53:45.654 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.654 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.654 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:45.654 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135380+0000", 2024-02-05T04:53:45.654 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.9250000000000007e-06 2024-02-05T04:53:45.654 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.654 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.655 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:45.655 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.137316+0000", 2024-02-05T04:53:45.655 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0019360930000000001 2024-02-05T04:53:45.655 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.655 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.655 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:45.655 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.137358+0000", 2024-02-05T04:53:45.655 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2196000000000001e-05 2024-02-05T04:53:45.655 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:45.655 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:45.655 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:45.655 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.655 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.655 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.3s6 37/31 ECSubWrite(tid=30, reqid=client.4634.0:219, at_version=37'13, trim_to=0'0, roll_forward_to=37'10))", 2024-02-05T04:53:45.655 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:29.137475+0000", 2024-02-05T04:53:45.655 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 16.506343111, 2024-02-05T04:53:45.655 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0022173890000000002, 2024-02-05T04:53:45.656 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:45.656 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:45.656 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:45.656 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.656 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:45.656 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.137475+0000", 2024-02-05T04:53:45.656 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:45.656 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.656 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.656 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:45.656 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.137475+0000", 2024-02-05T04:53:45.656 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:45.656 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.656 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.657 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:45.657 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.137476+0000", 2024-02-05T04:53:45.657 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0729999999999999e-06 2024-02-05T04:53:45.657 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.657 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.657 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:45.657 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.137869+0000", 2024-02-05T04:53:45.657 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00039313999999999999 2024-02-05T04:53:45.657 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.657 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.657 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:45.657 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.137912+0000", 2024-02-05T04:53:45.657 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2379999999999997e-05 2024-02-05T04:53:45.657 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.657 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.658 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:45.658 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.137925+0000", 2024-02-05T04:53:45.658 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2948999999999999e-05 2024-02-05T04:53:45.658 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.658 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.658 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:45.658 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.137998+0000", 2024-02-05T04:53:45.658 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2995000000000002e-05 2024-02-05T04:53:45.658 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.658 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.658 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:45.658 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.138008+0000", 2024-02-05T04:53:45.658 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0104e-05 2024-02-05T04:53:45.658 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.658 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.658 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:45.659 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.139653+0000", 2024-02-05T04:53:45.659 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0016451009999999999 2024-02-05T04:53:45.659 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.659 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.659 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:45.659 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.139693+0000", 2024-02-05T04:53:45.659 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9647000000000002e-05 2024-02-05T04:53:45.659 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:45.659 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:45.659 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:45.659 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.659 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.659 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.3s6 37/31 ECSubWrite(tid=31, reqid=client.4634.0:220, at_version=37'14, trim_to=0'0, roll_forward_to=37'10))", 2024-02-05T04:53:45.660 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:29.138137+0000", 2024-02-05T04:53:45.660 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 16.505681213999999, 2024-02-05T04:53:45.660 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0041763130000000001, 2024-02-05T04:53:45.660 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:45.660 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:45.660 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:45.660 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.660 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:45.660 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.138137+0000", 2024-02-05T04:53:45.660 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:45.661 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.661 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.661 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:45.661 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.138137+0000", 2024-02-05T04:53:45.661 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:45.661 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.661 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.661 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:45.661 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.138138+0000", 2024-02-05T04:53:45.661 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.37e-07 2024-02-05T04:53:45.661 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.661 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.661 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:45.661 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.138156+0000", 2024-02-05T04:53:45.662 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8448999999999999e-05 2024-02-05T04:53:45.662 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.662 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.662 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:45.662 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.138167+0000", 2024-02-05T04:53:45.662 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0930999999999999e-05 2024-02-05T04:53:45.662 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.662 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.662 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:45.662 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.138176+0000", 2024-02-05T04:53:45.662 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1600000000000004e-06 2024-02-05T04:53:45.662 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.662 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.662 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:45.662 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.138835+0000", 2024-02-05T04:53:45.663 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00065867499999999995 2024-02-05T04:53:45.663 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.663 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.663 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:45.663 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.138845+0000", 2024-02-05T04:53:45.663 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8109999999999997e-06 2024-02-05T04:53:45.663 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.663 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.663 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:45.663 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.142268+0000", 2024-02-05T04:53:45.663 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.003423059 2024-02-05T04:53:45.663 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.663 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.663 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:45.664 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.142313+0000", 2024-02-05T04:53:45.664 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5491000000000002e-05 2024-02-05T04:53:45.664 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:45.664 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:45.664 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:45.664 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.664 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.664 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.2s2 37/31 ECSubWrite(tid=22, reqid=client.4634.0:254, at_version=37'15, trim_to=0'0, roll_forward_to=37'10))", 2024-02-05T04:53:45.664 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:29.208534+0000", 2024-02-05T04:53:45.664 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 16.435284076999999, 2024-02-05T04:53:45.664 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0022982129999999999, 2024-02-05T04:53:45.664 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:45.664 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:45.664 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:45.664 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.665 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:45.665 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.208534+0000", 2024-02-05T04:53:45.665 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:45.665 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.665 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.665 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:45.665 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.208534+0000", 2024-02-05T04:53:45.665 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:45.665 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.665 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.665 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:45.665 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.208535+0000", 2024-02-05T04:53:45.665 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6599999999999996e-07 2024-02-05T04:53:45.666 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.666 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.666 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:45.666 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.208746+0000", 2024-02-05T04:53:45.666 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000211418 2024-02-05T04:53:45.666 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.666 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.666 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:45.666 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.208757+0000", 2024-02-05T04:53:45.666 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0878e-05 2024-02-05T04:53:45.666 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.666 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.666 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:45.666 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.208763+0000", 2024-02-05T04:53:45.666 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.75e-06 2024-02-05T04:53:45.667 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.667 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.667 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:45.667 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.208815+0000", 2024-02-05T04:53:45.667 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2169999999999997e-05 2024-02-05T04:53:45.667 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.667 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.667 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:45.667 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.208825+0000", 2024-02-05T04:53:45.667 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8120000000000002e-06 2024-02-05T04:53:45.667 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.667 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.667 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:45.667 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.210797+0000", 2024-02-05T04:53:45.667 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0019724500000000002 2024-02-05T04:53:45.668 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.668 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.668 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:45.668 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.210832+0000", 2024-02-05T04:53:45.668 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5169000000000002e-05 2024-02-05T04:53:45.668 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:45.668 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:45.668 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:45.668 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.668 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.668 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.2s2 37/31 ECSubWrite(tid=24, reqid=client.4634.0:257, at_version=37'18, trim_to=0'0, roll_forward_to=37'12))", 2024-02-05T04:53:45.668 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:29.211387+0000", 2024-02-05T04:53:45.668 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 16.432431000000001, 2024-02-05T04:53:45.669 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0043575100000000002, 2024-02-05T04:53:45.669 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:45.669 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:45.669 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:45.669 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.669 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:45.669 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.211387+0000", 2024-02-05T04:53:45.669 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:45.669 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.669 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.669 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:45.669 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.211387+0000", 2024-02-05T04:53:45.669 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:45.669 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.669 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.670 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:45.670 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.211388+0000", 2024-02-05T04:53:45.670 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2200000000000004e-07 2024-02-05T04:53:45.670 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.670 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.670 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:45.670 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.211406+0000", 2024-02-05T04:53:45.670 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7796999999999999e-05 2024-02-05T04:53:45.670 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.671 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.671 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:45.671 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.211416+0000", 2024-02-05T04:53:45.671 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0234000000000001e-05 2024-02-05T04:53:45.671 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.671 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.671 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:45.671 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.211421+0000", 2024-02-05T04:53:45.671 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.609e-06 2024-02-05T04:53:45.671 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.671 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.671 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:45.671 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.211467+0000", 2024-02-05T04:53:45.671 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5271000000000001e-05 2024-02-05T04:53:45.672 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.672 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.672 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:45.672 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.211473+0000", 2024-02-05T04:53:45.672 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5549999999999999e-06 2024-02-05T04:53:45.672 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.672 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.672 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:45.672 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.215717+0000", 2024-02-05T04:53:45.672 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.004243592 2024-02-05T04:53:45.672 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.672 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.672 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:45.672 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.215745+0000", 2024-02-05T04:53:45.673 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.7929999999999999e-05 2024-02-05T04:53:45.673 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:45.673 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:45.673 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:45.673 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.673 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.673 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.2s2 37/31 ECSubWrite(tid=26, reqid=client.4634.0:257, at_version=0'0, trim_to=0'0, roll_forward_to=37'18))", 2024-02-05T04:53:45.673 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:29.216477+0000", 2024-02-05T04:53:45.673 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 16.427341318, 2024-02-05T04:53:45.673 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0023123649999999998, 2024-02-05T04:53:45.673 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:45.673 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:45.673 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:45.673 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.674 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:45.674 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.216477+0000", 2024-02-05T04:53:45.674 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:45.674 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.674 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.674 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:45.674 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.216477+0000", 2024-02-05T04:53:45.674 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:45.674 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.674 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.674 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:45.674 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.216478+0000", 2024-02-05T04:53:45.674 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6499999999999999e-07 2024-02-05T04:53:45.674 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.675 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.675 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:45.675 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.216485+0000", 2024-02-05T04:53:45.675 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9529999999999999e-06 2024-02-05T04:53:45.675 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.675 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.675 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:45.675 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.216495+0000", 2024-02-05T04:53:45.675 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.9620000000000005e-06 2024-02-05T04:53:45.675 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.675 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.675 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:45.675 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.216504+0000", 2024-02-05T04:53:45.675 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.5639999999999997e-06 2024-02-05T04:53:45.676 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.676 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.676 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:45.676 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.216562+0000", 2024-02-05T04:53:45.676 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7837999999999997e-05 2024-02-05T04:53:45.676 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.676 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.676 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:45.676 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.216570+0000", 2024-02-05T04:53:45.676 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.1510000000000004e-06 2024-02-05T04:53:45.676 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.676 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.676 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:45.676 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.218758+0000", 2024-02-05T04:53:45.677 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0021876690000000002 2024-02-05T04:53:45.677 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.677 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.677 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:45.677 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.218789+0000", 2024-02-05T04:53:45.677 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.1662999999999998e-05 2024-02-05T04:53:45.677 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:45.677 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:45.677 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:45.677 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.677 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.677 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.2s2 38/31 ECSubWrite(tid=27, reqid=osd.12.0:27, at_version=38'20, trim_to=0'0, roll_forward_to=37'18))", 2024-02-05T04:53:45.677 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:30.143602+0000", 2024-02-05T04:53:45.677 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 15.500216336999999, 2024-02-05T04:53:45.677 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0021805349999999999, 2024-02-05T04:53:45.678 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:45.679 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:45.679 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:45.679 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.679 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:45.679 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.143602+0000", 2024-02-05T04:53:45.679 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:45.679 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.680 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.680 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:45.680 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.143602+0000", 2024-02-05T04:53:45.680 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:45.680 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.680 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.680 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:45.680 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.143602+0000", 2024-02-05T04:53:45.680 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2399999999999999e-07 2024-02-05T04:53:45.680 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.680 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.680 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:45.680 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.143626+0000", 2024-02-05T04:53:45.681 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3947e-05 2024-02-05T04:53:45.681 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.681 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.681 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:45.681 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.143640+0000", 2024-02-05T04:53:45.681 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3679999999999999e-05 2024-02-05T04:53:45.681 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.681 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.681 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:45.681 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.143648+0000", 2024-02-05T04:53:45.681 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0919999999999998e-06 2024-02-05T04:53:45.681 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.681 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.681 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:45.681 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.143722+0000", 2024-02-05T04:53:45.682 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4177999999999994e-05 2024-02-05T04:53:45.682 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.682 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.682 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:45.682 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.143733+0000", 2024-02-05T04:53:45.682 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0596e-05 2024-02-05T04:53:45.682 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.682 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.682 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:45.682 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.145745+0000", 2024-02-05T04:53:45.682 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0020118879999999999 2024-02-05T04:53:45.682 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.682 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.682 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:45.683 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.145782+0000", 2024-02-05T04:53:45.683 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7929999999999998e-05 2024-02-05T04:53:45.683 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:45.683 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:45.683 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:45.683 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.683 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.683 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.2s2 38/31 ECSubWrite(tid=28, reqid=osd.12.0:28, at_version=38'22, trim_to=0'0, roll_forward_to=37'18))", 2024-02-05T04:53:45.683 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:30.144310+0000", 2024-02-05T04:53:45.683 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 15.499507871, 2024-02-05T04:53:45.683 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0042662150000000003, 2024-02-05T04:53:45.683 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:45.683 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:45.683 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:45.683 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.684 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:45.684 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.144310+0000", 2024-02-05T04:53:45.684 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:45.684 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.684 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.684 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:45.684 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.144310+0000", 2024-02-05T04:53:45.684 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:45.684 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.684 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.684 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:45.684 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.144311+0000", 2024-02-05T04:53:45.684 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4499999999999996e-07 2024-02-05T04:53:45.685 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.685 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.685 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:45.685 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.144332+0000", 2024-02-05T04:53:45.685 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0954999999999998e-05 2024-02-05T04:53:45.685 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.685 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.685 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:45.685 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.144345+0000", 2024-02-05T04:53:45.685 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.259e-05 2024-02-05T04:53:45.685 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.685 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.685 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:45.685 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.144354+0000", 2024-02-05T04:53:45.685 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.7110000000000001e-06 2024-02-05T04:53:45.685 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.686 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.686 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:45.686 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.144438+0000", 2024-02-05T04:53:45.686 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4466000000000001e-05 2024-02-05T04:53:45.686 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.686 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.686 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:45.686 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.144448+0000", 2024-02-05T04:53:45.686 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0419e-05 2024-02-05T04:53:45.686 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.686 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.686 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:45.686 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.148538+0000", 2024-02-05T04:53:45.686 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0040892769999999997 2024-02-05T04:53:45.686 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.687 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.687 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:45.687 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.148577+0000", 2024-02-05T04:53:45.687 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8952000000000002e-05 2024-02-05T04:53:45.687 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:45.687 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:45.687 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:45.687 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.687 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.687 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.2s2 38/31 ECSubWrite(tid=29, reqid=osd.12.0:28, at_version=0'0, trim_to=0'0, roll_forward_to=38'22))", 2024-02-05T04:53:45.687 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:30.154256+0000", 2024-02-05T04:53:45.687 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 15.489562472999999, 2024-02-05T04:53:45.687 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002332673, 2024-02-05T04:53:45.687 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:45.687 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:45.688 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:45.688 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.688 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:45.688 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.154256+0000", 2024-02-05T04:53:45.688 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:45.688 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.688 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.688 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:45.688 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.154256+0000", 2024-02-05T04:53:45.688 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:45.688 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.688 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.688 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:45.689 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.154257+0000", 2024-02-05T04:53:45.689 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.062e-06 2024-02-05T04:53:45.689 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.689 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.689 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:45.689 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.154269+0000", 2024-02-05T04:53:45.689 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2147000000000001e-05 2024-02-05T04:53:45.689 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.689 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.689 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:45.689 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.154293+0000", 2024-02-05T04:53:45.689 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.4304999999999999e-05 2024-02-05T04:53:45.689 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.689 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.690 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:45.690 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.154303+0000", 2024-02-05T04:53:45.690 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5759999999999995e-06 2024-02-05T04:53:45.690 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.690 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.690 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:45.690 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.154372+0000", 2024-02-05T04:53:45.690 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9572000000000004e-05 2024-02-05T04:53:45.690 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.690 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.690 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:45.690 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.154382+0000", 2024-02-05T04:53:45.690 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.679e-06 2024-02-05T04:53:45.690 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.691 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.691 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:45.691 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.156518+0000", 2024-02-05T04:53:45.691 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002136097 2024-02-05T04:53:45.691 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.691 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.691 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:45.691 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.156588+0000", 2024-02-05T04:53:45.691 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0234999999999994e-05 2024-02-05T04:53:45.691 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:45.691 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:45.691 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:45.691 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.691 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.691 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.2s2 38/31 ECSubWrite(tid=30, reqid=osd.12.0:30, at_version=38'24, trim_to=0'0, roll_forward_to=38'22))", 2024-02-05T04:53:45.692 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:30.155268+0000", 2024-02-05T04:53:45.692 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 15.488550451, 2024-02-05T04:53:45.692 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0043229849999999997, 2024-02-05T04:53:45.692 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:45.692 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:45.692 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:45.692 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.692 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:45.692 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.155268+0000", 2024-02-05T04:53:45.692 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:45.692 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.692 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.692 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:45.692 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.155268+0000", 2024-02-05T04:53:45.693 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:45.693 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.693 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.693 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:45.693 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.155269+0000", 2024-02-05T04:53:45.693 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.33e-07 2024-02-05T04:53:45.693 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.693 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.693 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:45.693 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.155291+0000", 2024-02-05T04:53:45.693 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2708e-05 2024-02-05T04:53:45.693 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.693 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.693 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:45.693 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.155308+0000", 2024-02-05T04:53:45.694 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6996000000000001e-05 2024-02-05T04:53:45.694 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.694 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.694 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:45.694 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.155318+0000", 2024-02-05T04:53:45.694 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8849999999999994e-06 2024-02-05T04:53:45.694 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.694 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.694 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:45.694 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.155388+0000", 2024-02-05T04:53:45.694 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9704999999999998e-05 2024-02-05T04:53:45.694 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.694 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.694 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:45.694 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.155399+0000", 2024-02-05T04:53:45.695 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0473e-05 2024-02-05T04:53:45.695 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.695 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.695 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:45.695 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.159532+0000", 2024-02-05T04:53:45.695 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0041339180000000003 2024-02-05T04:53:45.695 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.695 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.695 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:45.695 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.159591+0000", 2024-02-05T04:53:45.695 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8366999999999999e-05 2024-02-05T04:53:45.695 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:45.695 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:45.695 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:45.695 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.696 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.696 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.2s2 42/31 ECSubWrite(tid=46, reqid=client.4634.0:381, at_version=0'0, trim_to=0'0, roll_forward_to=42'35))", 2024-02-05T04:53:45.696 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:34.162004+0000", 2024-02-05T04:53:45.696 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 11.481813821999999, 2024-02-05T04:53:45.696 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0025338520000000001, 2024-02-05T04:53:45.696 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:45.696 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:45.696 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:45.696 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.696 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:45.696 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.162004+0000", 2024-02-05T04:53:45.696 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:45.696 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.696 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.697 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:45.697 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.162004+0000", 2024-02-05T04:53:45.697 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:45.697 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.697 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.697 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:45.697 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.162005+0000", 2024-02-05T04:53:45.697 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.78e-07 2024-02-05T04:53:45.697 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.697 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.697 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:45.697 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.162025+0000", 2024-02-05T04:53:45.697 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9670999999999999e-05 2024-02-05T04:53:45.697 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.697 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.698 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:45.698 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.162051+0000", 2024-02-05T04:53:45.698 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5803e-05 2024-02-05T04:53:45.698 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.698 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.698 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:45.698 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.162062+0000", 2024-02-05T04:53:45.698 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0977e-05 2024-02-05T04:53:45.698 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.698 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.698 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:45.698 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.162151+0000", 2024-02-05T04:53:45.698 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.9441e-05 2024-02-05T04:53:45.698 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.698 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.699 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:45.699 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.162160+0000", 2024-02-05T04:53:45.699 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.6649999999999992e-06 2024-02-05T04:53:45.699 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.699 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.699 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:45.699 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.164512+0000", 2024-02-05T04:53:45.699 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0023523009999999998 2024-02-05T04:53:45.699 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.699 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.699 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:45.699 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.164538+0000", 2024-02-05T04:53:45.699 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.6115999999999999e-05 2024-02-05T04:53:45.699 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:45.700 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:45.700 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:45.700 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.700 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.700 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "pg_update_log_missing(3.2s2 epoch 43/31 rep_tid 47 entries 43'36 (0'0) error 3:4d05e2f1:::smithi19639362-34:head by client.4634.0:397 0.000000 -2 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 39'33)", 2024-02-05T04:53:45.700 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:35.163812+0000", 2024-02-05T04:53:45.700 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 10.480006682999999, 2024-02-05T04:53:45.700 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0028187389999999998, 2024-02-05T04:53:45.700 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:45.700 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:45.700 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:45.700 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.700 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:45.700 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.163812+0000", 2024-02-05T04:53:45.700 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:45.701 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.701 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.701 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:45.701 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.163812+0000", 2024-02-05T04:53:45.701 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:45.701 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.701 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.701 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:45.701 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.163812+0000", 2024-02-05T04:53:45.701 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4100000000000001e-07 2024-02-05T04:53:45.701 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.701 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.701 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:45.701 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.163833+0000", 2024-02-05T04:53:45.701 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1659e-05 2024-02-05T04:53:45.702 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.702 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.702 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:45.702 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.163861+0000", 2024-02-05T04:53:45.702 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.7149999999999999e-05 2024-02-05T04:53:45.702 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.702 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.702 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:45.702 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.163878+0000", 2024-02-05T04:53:45.702 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7371999999999999e-05 2024-02-05T04:53:45.702 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.702 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.702 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:45.702 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.163982+0000", 2024-02-05T04:53:45.702 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010451299999999999 2024-02-05T04:53:45.703 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.703 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.703 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:45.703 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.166630+0000", 2024-02-05T04:53:45.703 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002647904 2024-02-05T04:53:45.703 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:45.703 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:45.703 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:45.703 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.703 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.703 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.2s2 43/31 ECSubWrite(tid=48, reqid=osd.12.0:48, at_version=43'38, trim_to=0'0, roll_forward_to=42'35))", 2024-02-05T04:53:45.703 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:35.164524+0000", 2024-02-05T04:53:45.703 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 10.479294201, 2024-02-05T04:53:45.703 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0049387320000000004, 2024-02-05T04:53:45.704 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:45.704 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:45.704 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:45.704 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.704 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:45.704 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.164524+0000", 2024-02-05T04:53:45.704 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:45.704 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.704 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.704 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:45.704 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.164524+0000", 2024-02-05T04:53:45.704 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:45.704 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.704 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.705 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:45.705 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.164525+0000", 2024-02-05T04:53:45.705 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.2200000000000003e-07 2024-02-05T04:53:45.705 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.705 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.705 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:45.705 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.164556+0000", 2024-02-05T04:53:45.705 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.1050000000000003e-05 2024-02-05T04:53:45.705 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.705 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.705 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:45.705 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.164572+0000", 2024-02-05T04:53:45.705 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6387000000000001e-05 2024-02-05T04:53:45.705 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.705 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.706 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:45.706 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.164582+0000", 2024-02-05T04:53:45.706 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2050000000000008e-06 2024-02-05T04:53:45.706 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.706 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.706 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:45.706 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.164651+0000", 2024-02-05T04:53:45.706 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9219000000000003e-05 2024-02-05T04:53:45.706 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.706 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.706 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:45.706 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.164661+0000", 2024-02-05T04:53:45.706 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0210999999999999e-05 2024-02-05T04:53:45.706 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.706 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.707 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:45.707 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.169414+0000", 2024-02-05T04:53:45.707 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0047528470000000001 2024-02-05T04:53:45.707 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.707 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.707 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:45.707 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.169463+0000", 2024-02-05T04:53:45.707 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8990999999999999e-05 2024-02-05T04:53:45.707 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:45.707 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:45.707 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:45.707 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.707 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.707 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.2s2 43/31 ECSubWrite(tid=49, reqid=osd.12.0:48, at_version=0'0, trim_to=0'0, roll_forward_to=43'38))", 2024-02-05T04:53:45.707 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:35.170040+0000", 2024-02-05T04:53:45.707 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 10.473778698, 2024-02-05T04:53:45.708 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.004631652, 2024-02-05T04:53:45.708 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:45.708 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:45.708 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:45.708 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.708 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:45.708 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.170040+0000", 2024-02-05T04:53:45.708 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:45.708 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.708 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.708 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:45.708 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.170040+0000", 2024-02-05T04:53:45.708 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:45.708 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.709 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.709 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:45.709 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.170040+0000", 2024-02-05T04:53:45.709 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.0599999999999999e-07 2024-02-05T04:53:45.709 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.709 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.709 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:45.709 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.170053+0000", 2024-02-05T04:53:45.709 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2179e-05 2024-02-05T04:53:45.709 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.709 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.709 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:45.709 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.170066+0000", 2024-02-05T04:53:45.709 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2875e-05 2024-02-05T04:53:45.710 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.710 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.710 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:45.710 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.170074+0000", 2024-02-05T04:53:45.710 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.7760000000000003e-06 2024-02-05T04:53:45.710 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.710 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.710 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:45.710 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.170139+0000", 2024-02-05T04:53:45.710 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5086000000000007e-05 2024-02-05T04:53:45.710 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.710 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.710 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:45.710 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.170149+0000", 2024-02-05T04:53:45.710 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0045e-05 2024-02-05T04:53:45.711 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.711 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.711 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:45.711 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.174632+0000", 2024-02-05T04:53:45.711 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0044826570000000001 2024-02-05T04:53:45.711 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.711 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.711 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:45.711 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.174671+0000", 2024-02-05T04:53:45.711 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9128000000000002e-05 2024-02-05T04:53:45.711 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:45.711 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:45.711 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:45.711 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.712 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.712 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.ds1 43/31 ECSubWrite(tid=44, reqid=client.4634.0:401, at_version=43'24, trim_to=0'0, roll_forward_to=38'22))", 2024-02-05T04:53:45.712 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:35.183587+0000", 2024-02-05T04:53:45.712 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 10.460231199000001, 2024-02-05T04:53:45.712 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0048306419999999996, 2024-02-05T04:53:45.712 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:45.712 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:45.712 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:45.712 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.712 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:45.712 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.183587+0000", 2024-02-05T04:53:45.712 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:45.712 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.712 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.712 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:45.713 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.183587+0000", 2024-02-05T04:53:45.713 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:45.713 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.713 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.713 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:45.713 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.183588+0000", 2024-02-05T04:53:45.713 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5999999999999999e-07 2024-02-05T04:53:45.713 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.713 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.713 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:45.713 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.183610+0000", 2024-02-05T04:53:45.713 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2606e-05 2024-02-05T04:53:45.713 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.713 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.713 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:45.714 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.183623+0000", 2024-02-05T04:53:45.714 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2461e-05 2024-02-05T04:53:45.714 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.714 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.714 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:45.714 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.183631+0000", 2024-02-05T04:53:45.714 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8249999999999994e-06 2024-02-05T04:53:45.714 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.714 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.714 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:45.714 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.183703+0000", 2024-02-05T04:53:45.714 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1551999999999995e-05 2024-02-05T04:53:45.714 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.714 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.714 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:45.715 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.183713+0000", 2024-02-05T04:53:45.715 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0472000000000001e-05 2024-02-05T04:53:45.715 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.715 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.715 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:45.715 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.188379+0000", 2024-02-05T04:53:45.715 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0046652819999999998 2024-02-05T04:53:45.715 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.715 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.715 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:45.715 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.188418+0000", 2024-02-05T04:53:45.715 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8983999999999998e-05 2024-02-05T04:53:45.715 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:45.715 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:45.715 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:45.716 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.716 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.716 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.3s6 47/31 ECSubWrite(tid=73, reqid=client.4634.0:419, at_version=47'33, trim_to=0'0, roll_forward_to=44'31))", 2024-02-05T04:53:45.716 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:38.944689+0000", 2024-02-05T04:53:45.716 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 6.6991296919999996, 2024-02-05T04:53:45.716 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.004725796, 2024-02-05T04:53:45.716 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:45.716 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:45.716 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:45.716 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.716 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:45.716 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:38.944689+0000", 2024-02-05T04:53:45.716 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:45.716 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.717 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.717 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:45.717 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:38.944689+0000", 2024-02-05T04:53:45.717 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:45.717 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.717 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.717 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:45.717 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:38.944689+0000", 2024-02-05T04:53:45.717 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5699999999999999e-07 2024-02-05T04:53:45.717 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.717 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.717 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:45.717 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:38.944911+0000", 2024-02-05T04:53:45.717 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00022261199999999999 2024-02-05T04:53:45.718 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.718 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.718 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:45.718 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:38.944925+0000", 2024-02-05T04:53:45.718 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4098e-05 2024-02-05T04:53:45.718 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.718 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.718 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:45.718 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:38.944939+0000", 2024-02-05T04:53:45.718 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3424000000000001e-05 2024-02-05T04:53:45.718 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.718 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.718 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:45.718 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:38.945110+0000", 2024-02-05T04:53:45.718 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00017102 2024-02-05T04:53:45.719 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.719 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.719 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:45.719 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:38.945119+0000", 2024-02-05T04:53:45.719 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2220000000000008e-06 2024-02-05T04:53:45.719 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.719 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.719 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:45.719 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:38.949379+0000", 2024-02-05T04:53:45.719 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0042596719999999999 2024-02-05T04:53:45.719 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.719 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.719 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:45.720 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:38.949414+0000", 2024-02-05T04:53:45.720 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5590999999999999e-05 2024-02-05T04:53:45.720 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:45.720 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:45.720 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:45.720 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.720 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.720 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.3s6 47/31 ECSubWrite(tid=74, reqid=client.4634.0:419, at_version=0'0, trim_to=0'0, roll_forward_to=47'33))", 2024-02-05T04:53:45.720 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:38.950018+0000", 2024-02-05T04:53:45.720 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 6.6938005619999998, 2024-02-05T04:53:45.720 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0021228449999999999, 2024-02-05T04:53:45.720 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:45.720 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:45.720 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:45.720 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.721 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:45.721 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:38.950018+0000", 2024-02-05T04:53:45.721 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:45.721 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.721 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.721 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:45.721 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:38.950018+0000", 2024-02-05T04:53:45.721 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:45.721 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.721 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.721 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:45.721 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:38.950018+0000", 2024-02-05T04:53:45.721 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8299999999999996e-07 2024-02-05T04:53:45.721 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.721 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.722 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:45.722 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:38.950034+0000", 2024-02-05T04:53:45.722 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5885000000000001e-05 2024-02-05T04:53:45.722 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.722 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.722 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:45.722 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:38.950045+0000", 2024-02-05T04:53:45.722 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0621999999999999e-05 2024-02-05T04:53:45.722 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.722 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.722 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:45.722 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:38.950052+0000", 2024-02-05T04:53:45.722 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.832e-06 2024-02-05T04:53:45.722 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.722 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.722 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:45.723 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:38.950109+0000", 2024-02-05T04:53:45.723 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7655000000000003e-05 2024-02-05T04:53:45.723 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.723 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.723 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:45.723 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:38.950120+0000", 2024-02-05T04:53:45.723 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0178e-05 2024-02-05T04:53:45.723 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.723 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.723 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:45.723 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:38.952100+0000", 2024-02-05T04:53:45.723 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0019804340000000001 2024-02-05T04:53:45.723 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.723 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.723 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:45.723 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:38.952141+0000", 2024-02-05T04:53:45.724 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0556000000000003e-05 2024-02-05T04:53:45.724 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:45.724 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:45.724 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:45.724 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.724 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.724 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.9s3 52/31 ECSubWrite(tid=44, reqid=osd.14.0:44, at_version=52'19, trim_to=0'0, roll_forward_to=51'17))", 2024-02-05T04:53:45.724 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:43.957868+0000", 2024-02-05T04:53:45.724 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 1.685950179, 2024-02-05T04:53:45.724 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0035345699999999999, 2024-02-05T04:53:45.724 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:45.724 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:45.724 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:45.724 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.724 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:45.725 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:43.957868+0000", 2024-02-05T04:53:45.725 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:45.725 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.725 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.725 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:45.725 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:43.957868+0000", 2024-02-05T04:53:45.725 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:45.725 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.725 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.725 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:45.725 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:43.957868+0000", 2024-02-05T04:53:45.725 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.7399999999999999e-07 2024-02-05T04:53:45.725 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.725 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.726 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:45.726 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:43.957889+0000", 2024-02-05T04:53:45.726 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0489999999999999e-05 2024-02-05T04:53:45.726 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.726 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.726 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:45.726 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:43.957900+0000", 2024-02-05T04:53:45.726 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1114999999999999e-05 2024-02-05T04:53:45.726 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.726 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.726 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:45.726 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:43.957908+0000", 2024-02-05T04:53:45.726 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9009999999999994e-06 2024-02-05T04:53:45.726 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.726 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.726 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:45.727 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:43.957982+0000", 2024-02-05T04:53:45.727 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4196999999999995e-05 2024-02-05T04:53:45.727 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.727 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.727 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:45.727 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:43.957995+0000", 2024-02-05T04:53:45.727 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2435e-05 2024-02-05T04:53:45.727 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.727 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.727 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:45.727 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:43.961350+0000", 2024-02-05T04:53:45.727 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.003355846 2024-02-05T04:53:45.727 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.727 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.727 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:45.727 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:43.961403+0000", 2024-02-05T04:53:45.727 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2312000000000003e-05 2024-02-05T04:53:45.728 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:45.728 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:45.728 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:45.728 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.728 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.728 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.9s3 52/31 ECSubWrite(tid=45, reqid=osd.14.0:44, at_version=0'0, trim_to=0'0, roll_forward_to=52'19))", 2024-02-05T04:53:45.728 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:43.962590+0000", 2024-02-05T04:53:45.728 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 1.6812278790000001, 2024-02-05T04:53:45.728 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002481839, 2024-02-05T04:53:45.728 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:45.728 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:45.728 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:45.728 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.728 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:45.729 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:43.962590+0000", 2024-02-05T04:53:45.729 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:45.729 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.729 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.729 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:45.729 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:43.962590+0000", 2024-02-05T04:53:45.729 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:45.729 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.729 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.729 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:45.729 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:43.962591+0000", 2024-02-05T04:53:45.729 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.2200000000000003e-07 2024-02-05T04:53:45.729 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.729 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.729 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:45.730 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:43.962602+0000", 2024-02-05T04:53:45.730 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.128e-05 2024-02-05T04:53:45.730 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.730 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.730 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:45.730 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:43.962615+0000", 2024-02-05T04:53:45.730 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2762000000000001e-05 2024-02-05T04:53:45.730 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.730 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.730 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:45.730 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:43.962622+0000", 2024-02-05T04:53:45.730 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7379999999999997e-06 2024-02-05T04:53:45.730 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.730 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.731 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:45.731 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:43.962678+0000", 2024-02-05T04:53:45.731 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6267999999999997e-05 2024-02-05T04:53:45.731 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.731 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.731 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:45.731 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:43.962687+0000", 2024-02-05T04:53:45.731 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3080000000000008e-06 2024-02-05T04:53:45.731 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.731 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.731 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:45.731 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:43.965039+0000", 2024-02-05T04:53:45.731 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0023525740000000001 2024-02-05T04:53:45.731 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:45.731 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:45.732 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:45.732 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:43.965072+0000", 2024-02-05T04:53:45.732 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3086999999999997e-05 2024-02-05T04:53:45.732 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:45.732 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:45.732 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:45.732 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:45.732 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:45.732 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T04:53:45.734 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-02-05T04:53:45.734 INFO:tasks.ceph.osd.13.smithi091.stderr:2024-02-05T04:53:45.732+0000 7f526fd4d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 34281) UID: 0 2024-02-05T04:53:45.828 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T04:53:45.834 INFO:tasks.ceph.osd.6.smithi150.stderr:2024-02-05T04:53:45.833+0000 7f6bea9f0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34015) UID: 0 2024-02-05T04:53:45.839 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T04:53:45.839 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-02-05T04:53:45.840 INFO:teuthology.orchestra.run.smithi196.stdout: "num_ops": 0 2024-02-05T04:53:45.840 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T04:53:45.849 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-02-05T04:53:45.935 INFO:tasks.ceph.osd.7.smithi196.stderr:2024-02-05T04:53:45.934+0000 7fb281419640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34606) UID: 0 2024-02-05T04:53:45.952 INFO:teuthology.orchestra.run.smithi062.stderr:marked out osd.8. 2024-02-05T04:53:45.956 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T04:53:45.956 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-02-05T04:53:45.956 INFO:teuthology.orchestra.run.smithi196.stdout: "complaint_time": 30, 2024-02-05T04:53:45.956 INFO:teuthology.orchestra.run.smithi196.stdout: "num_blocked_ops": 0 2024-02-05T04:53:45.956 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T04:53:45.957 INFO:tasks.rados.rados.0.smithi196.stdout:192: done (1 left) 2024-02-05T04:53:45.957 INFO:tasks.rados.rados.0.smithi196.stdout:194: done (0 left) 2024-02-05T04:53:45.957 INFO:tasks.rados.rados.0.smithi196.stdout:195: rollback oid 3 current snap is 13 2024-02-05T04:53:45.957 INFO:tasks.rados.rados.0.smithi196.stdout:rollback oid 3 to 9 2024-02-05T04:53:45.957 INFO:tasks.rados.rados.0.smithi196.stdout:196: delete oid 41 current snap is 13 2024-02-05T04:53:45.966 INFO:tasks.rados.rados.0.smithi196.stdout:196: done (1 left) 2024-02-05T04:53:45.967 INFO:tasks.rados.rados.0.smithi196.stdout:197: snap_create 2024-02-05T04:53:45.968 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-02-05T04:53:46.053 INFO:tasks.ceph.osd.11.smithi196.stderr:2024-02-05T04:53:46.052+0000 7f85a8d86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 34609) UID: 0 2024-02-05T04:53:46.116 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T04:53:46.116 INFO:teuthology.orchestra.run.smithi196.stdout: "size": 20, 2024-02-05T04:53:46.116 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 600, 2024-02-05T04:53:46.116 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-02-05T04:53:46.116 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.116 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:225 3.ds0 3:b4e8e82c:::smithi19639362-25:head [copy-from ver 6] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:53:46.116 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:29.135146+0000", 2024-02-05T04:53:46.116 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 16.978762609, 2024-02-05T04:53:46.116 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.016925534999999998, 2024-02-05T04:53:46.116 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:46.116 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:46.116 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:53:46.117 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:53:46.117 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:46.117 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 225 2024-02-05T04:53:46.117 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.117 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:46.117 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.117 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:46.117 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135146+0000", 2024-02-05T04:53:46.117 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:46.117 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.117 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.117 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:46.117 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135146+0000", 2024-02-05T04:53:46.117 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:46.118 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.118 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.118 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:46.118 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135148+0000", 2024-02-05T04:53:46.118 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5170000000000002e-06 2024-02-05T04:53:46.118 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.118 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.118 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:46.118 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135158+0000", 2024-02-05T04:53:46.118 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.3819999999999993e-06 2024-02-05T04:53:46.118 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.118 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.119 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:46.119 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135177+0000", 2024-02-05T04:53:46.119 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9513000000000001e-05 2024-02-05T04:53:46.119 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.119 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.119 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:46.119 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135187+0000", 2024-02-05T04:53:46.119 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0054000000000001e-05 2024-02-05T04:53:46.119 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.119 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.119 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:46.119 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135292+0000", 2024-02-05T04:53:46.119 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000104651 2024-02-05T04:53:46.119 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.119 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.119 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:53:46.120 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135405+0000", 2024-02-05T04:53:46.120 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011277099999999999 2024-02-05T04:53:46.120 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.120 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.120 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:46.120 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.146446+0000", 2024-02-05T04:53:46.120 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.011041535 2024-02-05T04:53:46.120 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.120 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.120 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:46.120 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.148233+0000", 2024-02-05T04:53:46.120 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001787089 2024-02-05T04:53:46.120 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.120 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.121 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T04:53:46.121 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.151960+0000", 2024-02-05T04:53:46.121 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0037264720000000002 2024-02-05T04:53:46.121 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.121 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.121 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:46.121 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.152071+0000", 2024-02-05T04:53:46.121 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000111551 2024-02-05T04:53:46.121 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:46.121 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:46.121 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:46.121 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.121 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.121 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:226 3.ds0 3:b4e8e82c:::smithi19639362-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:53:46.121 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:29.135222+0000", 2024-02-05T04:53:46.122 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 16.978686089, 2024-02-05T04:53:46.122 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.012833986, 2024-02-05T04:53:46.122 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:46.122 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-02-05T04:53:46.122 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:53:46.122 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:53:46.122 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:46.122 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 226 2024-02-05T04:53:46.122 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.122 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:46.122 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.122 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:46.122 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135222+0000", 2024-02-05T04:53:46.122 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:46.122 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.122 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.123 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:46.123 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135222+0000", 2024-02-05T04:53:46.123 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:46.123 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.123 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.123 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:46.123 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135223+0000", 2024-02-05T04:53:46.123 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4099999999999998e-07 2024-02-05T04:53:46.123 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.123 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.123 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:46.123 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135227+0000", 2024-02-05T04:53:46.123 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8399999999999997e-06 2024-02-05T04:53:46.123 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.123 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.124 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:46.124 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135237+0000", 2024-02-05T04:53:46.124 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8290000000000003e-06 2024-02-05T04:53:46.124 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.124 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.124 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:46.124 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135244+0000", 2024-02-05T04:53:46.124 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4989999999999997e-06 2024-02-05T04:53:46.124 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.124 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.124 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:46.124 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135624+0000", 2024-02-05T04:53:46.124 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00037992100000000002 2024-02-05T04:53:46.125 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.125 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.125 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for blocked object", 2024-02-05T04:53:46.125 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135672+0000", 2024-02-05T04:53:46.125 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7861999999999997e-05 2024-02-05T04:53:46.125 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.125 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.125 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:46.125 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.147862+0000", 2024-02-05T04:53:46.125 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.012189847 2024-02-05T04:53:46.125 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.125 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.125 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:53:46.125 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.147947+0000", 2024-02-05T04:53:46.125 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4862000000000002e-05 2024-02-05T04:53:46.125 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.126 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.126 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:46.126 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.148056+0000", 2024-02-05T04:53:46.126 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000109585 2024-02-05T04:53:46.126 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:46.126 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:46.126 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:46.126 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.126 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.126 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.3s1 37/31 ECSubWrite(tid=33, reqid=client.4634.0:220, at_version=0'0, trim_to=0'0, roll_forward_to=37'14))", 2024-02-05T04:53:46.126 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:29.144152+0000", 2024-02-05T04:53:46.126 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 16.969756561000001, 2024-02-05T04:53:46.126 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0036900819999999999, 2024-02-05T04:53:46.126 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:46.126 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:46.127 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:46.127 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.127 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:46.127 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.144152+0000", 2024-02-05T04:53:46.127 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:46.127 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.127 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.127 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:46.127 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.144152+0000", 2024-02-05T04:53:46.127 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:46.127 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.127 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.127 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:46.127 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.144153+0000", 2024-02-05T04:53:46.127 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0279999999999999e-06 2024-02-05T04:53:46.127 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.128 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.128 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:46.128 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.144165+0000", 2024-02-05T04:53:46.128 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2584e-05 2024-02-05T04:53:46.128 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.128 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.128 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:46.128 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.144184+0000", 2024-02-05T04:53:46.128 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8389e-05 2024-02-05T04:53:46.128 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.128 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.128 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:46.128 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.144197+0000", 2024-02-05T04:53:46.128 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3291e-05 2024-02-05T04:53:46.128 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.129 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.129 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:46.129 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.144268+0000", 2024-02-05T04:53:46.129 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0489999999999998e-05 2024-02-05T04:53:46.129 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.129 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.129 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:46.129 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.144276+0000", 2024-02-05T04:53:46.129 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.5359999999999992e-06 2024-02-05T04:53:46.129 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.129 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.129 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:46.129 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.147731+0000", 2024-02-05T04:53:46.129 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.003455175 2024-02-05T04:53:46.130 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.130 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.130 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:46.130 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.147842+0000", 2024-02-05T04:53:46.130 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011058899999999999 2024-02-05T04:53:46.130 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:46.130 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:46.130 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:46.130 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.130 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.130 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.3s1 38/31 ECSubWrite(tid=35, reqid=osd.4.0:35, at_version=38'16, trim_to=0'0, roll_forward_to=37'14))", 2024-02-05T04:53:46.130 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:30.131623+0000", 2024-02-05T04:53:46.130 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 15.982285149999999, 2024-02-05T04:53:46.130 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0037957590000000001, 2024-02-05T04:53:46.130 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:46.131 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:46.131 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:46.131 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.131 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:46.131 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.131623+0000", 2024-02-05T04:53:46.131 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:46.131 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.131 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.131 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:46.131 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.131623+0000", 2024-02-05T04:53:46.131 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:46.131 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.131 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.131 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:46.131 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.131624+0000", 2024-02-05T04:53:46.132 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2999999999999999e-07 2024-02-05T04:53:46.132 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.132 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.132 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:46.132 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.131647+0000", 2024-02-05T04:53:46.132 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2982000000000001e-05 2024-02-05T04:53:46.132 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.132 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.132 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:46.132 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.131659+0000", 2024-02-05T04:53:46.132 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2371999999999999e-05 2024-02-05T04:53:46.132 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.132 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.132 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:46.133 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.131672+0000", 2024-02-05T04:53:46.133 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3108000000000001e-05 2024-02-05T04:53:46.133 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.133 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.133 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:46.133 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.131756+0000", 2024-02-05T04:53:46.133 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4287000000000001e-05 2024-02-05T04:53:46.133 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.133 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.133 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:46.133 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.131767+0000", 2024-02-05T04:53:46.133 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1113e-05 2024-02-05T04:53:46.134 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.134 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.134 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:46.134 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.135381+0000", 2024-02-05T04:53:46.134 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00361372 2024-02-05T04:53:46.134 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.134 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.134 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:46.134 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.135419+0000", 2024-02-05T04:53:46.134 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7947000000000001e-05 2024-02-05T04:53:46.134 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:46.134 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:46.134 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:46.134 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.134 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.135 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.6s5 38/31 ECSubWrite(tid=90, reqid=osd.13.0:90, at_version=38'11, trim_to=0'0, roll_forward_to=37'8))", 2024-02-05T04:53:46.135 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:30.142135+0000", 2024-02-05T04:53:46.135 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 15.971772977000001, 2024-02-05T04:53:46.135 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0038445620000000002, 2024-02-05T04:53:46.135 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:46.135 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:46.135 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:46.135 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.135 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:46.135 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.142135+0000", 2024-02-05T04:53:46.135 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:46.135 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.135 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.135 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:46.135 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.142135+0000", 2024-02-05T04:53:46.136 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:46.136 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.136 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.136 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:46.136 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.142136+0000", 2024-02-05T04:53:46.136 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0400000000000005e-07 2024-02-05T04:53:46.136 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.136 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.136 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:46.136 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.142162+0000", 2024-02-05T04:53:46.136 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.6200999999999999e-05 2024-02-05T04:53:46.136 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.136 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.136 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:46.137 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.142176+0000", 2024-02-05T04:53:46.137 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3451e-05 2024-02-05T04:53:46.137 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.137 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.137 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:46.137 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.142190+0000", 2024-02-05T04:53:46.137 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4049000000000001e-05 2024-02-05T04:53:46.137 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.137 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.137 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:46.137 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.142273+0000", 2024-02-05T04:53:46.137 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3490000000000002e-05 2024-02-05T04:53:46.137 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.137 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.138 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:46.138 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.142285+0000", 2024-02-05T04:53:46.138 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1398e-05 2024-02-05T04:53:46.138 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.138 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.138 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:46.138 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.145940+0000", 2024-02-05T04:53:46.138 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.003655635 2024-02-05T04:53:46.138 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.138 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.139 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:46.139 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.145980+0000", 2024-02-05T04:53:46.139 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9533999999999998e-05 2024-02-05T04:53:46.139 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:46.139 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:46.139 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:46.139 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.139 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.139 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:263 3.ds0 3:b7ad4782:::smithi19639362-40:head [append 0~737280] snapc 3=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-02-05T04:53:46.139 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:30.172056+0000", 2024-02-05T04:53:46.139 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 15.941852484, 2024-02-05T04:53:46.140 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0047829179999999997, 2024-02-05T04:53:46.140 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:46.140 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:46.140 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:53:46.140 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:53:46.140 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:46.140 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 263 2024-02-05T04:53:46.140 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.140 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:46.140 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.140 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:46.140 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.172056+0000", 2024-02-05T04:53:46.140 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:46.141 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.141 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.141 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:46.141 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.172056+0000", 2024-02-05T04:53:46.141 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:46.141 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.141 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.141 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:46.141 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.172058+0000", 2024-02-05T04:53:46.141 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0660000000000002e-06 2024-02-05T04:53:46.141 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.141 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.141 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:46.141 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.172410+0000", 2024-02-05T04:53:46.141 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00035195700000000002 2024-02-05T04:53:46.142 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.142 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.142 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:46.142 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.172441+0000", 2024-02-05T04:53:46.142 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.0676999999999998e-05 2024-02-05T04:53:46.142 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.142 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.142 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:46.142 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.172447+0000", 2024-02-05T04:53:46.142 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4250000000000003e-06 2024-02-05T04:53:46.142 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.142 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.142 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:46.142 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.172535+0000", 2024-02-05T04:53:46.142 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.7855000000000005e-05 2024-02-05T04:53:46.143 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.143 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.143 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:53:46.143 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.172613+0000", 2024-02-05T04:53:46.143 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8356999999999995e-05 2024-02-05T04:53:46.143 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.143 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.143 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:46.143 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.173406+0000", 2024-02-05T04:53:46.143 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00079290199999999995 2024-02-05T04:53:46.143 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.143 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.144 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:46.144 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.174300+0000", 2024-02-05T04:53:46.144 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00089424000000000003 2024-02-05T04:53:46.144 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.144 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.144 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T04:53:46.144 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.176774+0000", 2024-02-05T04:53:46.144 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0024735030000000002 2024-02-05T04:53:46.144 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.144 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.144 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:46.144 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.176839+0000", 2024-02-05T04:53:46.144 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4936000000000003e-05 2024-02-05T04:53:46.145 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:46.145 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:46.145 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:46.145 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.145 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.145 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:266 3.ds0 3:b7ad4782:::smithi19639362-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-02-05T04:53:46.145 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:30.174843+0000", 2024-02-05T04:53:46.145 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 15.939064965, 2024-02-05T04:53:46.145 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.004642803, 2024-02-05T04:53:46.145 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:46.145 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-02-05T04:53:46.145 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:53:46.145 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:53:46.145 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:46.145 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 266 2024-02-05T04:53:46.145 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.146 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:46.146 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.146 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:46.146 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.174843+0000", 2024-02-05T04:53:46.146 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:46.146 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.146 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.146 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:46.146 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.174843+0000", 2024-02-05T04:53:46.146 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:46.146 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.146 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.146 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:46.146 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.174844+0000", 2024-02-05T04:53:46.147 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1900000000000003e-07 2024-02-05T04:53:46.147 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.147 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.147 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:46.147 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.174846+0000", 2024-02-05T04:53:46.147 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.486e-06 2024-02-05T04:53:46.147 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.147 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.147 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:46.147 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.174854+0000", 2024-02-05T04:53:46.147 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9230000000000002e-06 2024-02-05T04:53:46.147 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.147 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.147 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:46.147 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.174860+0000", 2024-02-05T04:53:46.148 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7559999999999996e-06 2024-02-05T04:53:46.148 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.148 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.148 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:46.148 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.176912+0000", 2024-02-05T04:53:46.148 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002051977 2024-02-05T04:53:46.148 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.148 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.148 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for rw locks", 2024-02-05T04:53:46.148 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.176993+0000", 2024-02-05T04:53:46.148 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.1355999999999998e-05 2024-02-05T04:53:46.148 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.148 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.149 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:46.149 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.178630+0000", 2024-02-05T04:53:46.149 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0016364210000000001 2024-02-05T04:53:46.149 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.149 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.149 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:53:46.149 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.178680+0000", 2024-02-05T04:53:46.149 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0340999999999998e-05 2024-02-05T04:53:46.149 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.149 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.149 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:46.149 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.179486+0000", 2024-02-05T04:53:46.149 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00080602400000000002 2024-02-05T04:53:46.149 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:46.149 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:46.150 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:46.150 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.150 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.150 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:274 3.ds0 3:be3a9027:::smithi19639362-44:head [append 0~770048] snapc 3=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-02-05T04:53:46.150 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:30.190943+0000", 2024-02-05T04:53:46.150 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 15.922965308, 2024-02-05T04:53:46.150 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0053404719999999998, 2024-02-05T04:53:46.150 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:46.150 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:46.150 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:53:46.150 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:53:46.150 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:46.150 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 274 2024-02-05T04:53:46.150 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.150 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:46.150 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.151 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:46.151 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.190943+0000", 2024-02-05T04:53:46.151 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:46.151 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.151 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.151 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:46.151 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.190943+0000", 2024-02-05T04:53:46.151 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:46.151 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.151 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.151 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:46.151 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.190945+0000", 2024-02-05T04:53:46.151 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8530000000000001e-06 2024-02-05T04:53:46.151 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.151 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.152 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:46.152 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.191153+0000", 2024-02-05T04:53:46.152 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000207584 2024-02-05T04:53:46.152 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.152 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.152 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:46.152 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.191169+0000", 2024-02-05T04:53:46.152 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6568000000000001e-05 2024-02-05T04:53:46.152 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.152 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.152 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:46.152 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.191176+0000", 2024-02-05T04:53:46.152 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.3429999999999998e-06 2024-02-05T04:53:46.152 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.152 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.153 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:46.153 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.191233+0000", 2024-02-05T04:53:46.153 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6952e-05 2024-02-05T04:53:46.153 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.153 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.153 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:53:46.153 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.191301+0000", 2024-02-05T04:53:46.153 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7972999999999994e-05 2024-02-05T04:53:46.153 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.153 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.153 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:46.153 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.192078+0000", 2024-02-05T04:53:46.153 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00077698899999999996 2024-02-05T04:53:46.153 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.154 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.154 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:46.154 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.192971+0000", 2024-02-05T04:53:46.154 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000892971 2024-02-05T04:53:46.154 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.154 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.154 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T04:53:46.154 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.196200+0000", 2024-02-05T04:53:46.154 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0032284810000000001 2024-02-05T04:53:46.154 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.154 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.154 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:46.154 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.196284+0000", 2024-02-05T04:53:46.155 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3757999999999993e-05 2024-02-05T04:53:46.155 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:46.155 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:46.155 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:46.155 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.155 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.155 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:277 3.ds0 3:be3a9027:::smithi19639362-44:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-02-05T04:53:46.155 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:30.195060+0000", 2024-02-05T04:53:46.155 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 15.918847953, 2024-02-05T04:53:46.155 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0044373099999999999, 2024-02-05T04:53:46.155 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:46.155 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-02-05T04:53:46.155 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:53:46.155 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:53:46.156 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:46.156 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 277 2024-02-05T04:53:46.156 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.156 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:46.156 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.156 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:46.156 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.195060+0000", 2024-02-05T04:53:46.156 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:46.156 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.156 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.156 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:46.156 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.195060+0000", 2024-02-05T04:53:46.157 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:46.157 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.157 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.157 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:46.157 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.195061+0000", 2024-02-05T04:53:46.157 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9400000000000005e-07 2024-02-05T04:53:46.157 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.157 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.157 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:46.157 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.195063+0000", 2024-02-05T04:53:46.157 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2639999999999998e-06 2024-02-05T04:53:46.157 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.157 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.157 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:46.157 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.195070+0000", 2024-02-05T04:53:46.158 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0910000000000004e-06 2024-02-05T04:53:46.158 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.158 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.158 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:46.158 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.195075+0000", 2024-02-05T04:53:46.158 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3000000000000003e-06 2024-02-05T04:53:46.158 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.158 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.158 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:46.158 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.196855+0000", 2024-02-05T04:53:46.158 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001780694 2024-02-05T04:53:46.158 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.158 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.158 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for rw locks", 2024-02-05T04:53:46.159 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.196953+0000", 2024-02-05T04:53:46.159 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.7237000000000004e-05 2024-02-05T04:53:46.159 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.159 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.159 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:46.159 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.198325+0000", 2024-02-05T04:53:46.159 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0013727240000000001 2024-02-05T04:53:46.159 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.159 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.159 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:53:46.159 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.198422+0000", 2024-02-05T04:53:46.159 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6675999999999994e-05 2024-02-05T04:53:46.159 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.159 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.159 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:46.160 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.199498+0000", 2024-02-05T04:53:46.160 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00107573 2024-02-05T04:53:46.160 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:46.160 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:46.160 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:46.160 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.160 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.161 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:387 3.ds0 3:b4e8e82c:::smithi19639362-25:head [read 0~638976,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-02-05T04:53:46.161 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:34.171793+0000", 2024-02-05T04:53:46.161 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 11.94211511, 2024-02-05T04:53:46.161 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0052279479999999996, 2024-02-05T04:53:46.161 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:46.161 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-02-05T04:53:46.162 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:53:46.162 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:53:46.162 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:46.162 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 387 2024-02-05T04:53:46.162 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.162 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:46.163 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.163 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:46.163 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.171793+0000", 2024-02-05T04:53:46.163 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:46.163 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.163 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.164 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:46.164 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.171793+0000", 2024-02-05T04:53:46.164 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:46.164 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.164 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.164 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:46.164 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.171795+0000", 2024-02-05T04:53:46.164 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0729999999999999e-06 2024-02-05T04:53:46.165 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.165 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.165 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:46.165 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.171805+0000", 2024-02-05T04:53:46.165 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1339999999999993e-06 2024-02-05T04:53:46.165 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.165 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.165 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:46.165 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.171827+0000", 2024-02-05T04:53:46.165 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2954000000000001e-05 2024-02-05T04:53:46.165 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.165 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.165 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:46.165 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.171839+0000", 2024-02-05T04:53:46.165 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1646e-05 2024-02-05T04:53:46.166 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.166 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.166 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:46.166 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.171919+0000", 2024-02-05T04:53:46.166 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0232e-05 2024-02-05T04:53:46.166 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.166 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.166 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op must wait for map", 2024-02-05T04:53:46.166 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.171927+0000", 2024-02-05T04:53:46.166 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8369999999999997e-06 2024-02-05T04:53:46.166 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.166 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.166 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:46.166 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.174842+0000", 2024-02-05T04:53:46.166 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0029150309999999998 2024-02-05T04:53:46.166 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.167 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.167 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:53:46.167 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.174940+0000", 2024-02-05T04:53:46.167 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8147999999999996e-05 2024-02-05T04:53:46.167 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.167 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.167 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:46.167 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.177021+0000", 2024-02-05T04:53:46.167 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0020808929999999999 2024-02-05T04:53:46.167 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:46.167 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:46.167 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:46.167 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.167 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.167 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:388 3.ds0 3:b4e8e82c:::smithi19639362-25:head [read 0~638976,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-02-05T04:53:46.167 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:34.171871+0000", 2024-02-05T04:53:46.168 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 11.942037664000001, 2024-02-05T04:53:46.168 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0058757660000000001, 2024-02-05T04:53:46.168 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:46.168 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-02-05T04:53:46.168 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:53:46.168 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:53:46.168 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:46.168 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 388 2024-02-05T04:53:46.168 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.168 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:46.168 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.168 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:46.168 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.171871+0000", 2024-02-05T04:53:46.168 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:46.168 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.169 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.169 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:46.169 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.171871+0000", 2024-02-05T04:53:46.169 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:46.169 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.169 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.169 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:46.169 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.171871+0000", 2024-02-05T04:53:46.169 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8899999999999999e-07 2024-02-05T04:53:46.169 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.169 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.169 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:46.169 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.171875+0000", 2024-02-05T04:53:46.169 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.777e-06 2024-02-05T04:53:46.169 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.170 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.170 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:46.170 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.171884+0000", 2024-02-05T04:53:46.170 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8559999999999996e-06 2024-02-05T04:53:46.170 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.170 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.170 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:46.170 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.171892+0000", 2024-02-05T04:53:46.170 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6000000000000001e-06 2024-02-05T04:53:46.170 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.170 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.170 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:46.170 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.171986+0000", 2024-02-05T04:53:46.171 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.4010000000000003e-05 2024-02-05T04:53:46.171 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.171 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.171 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting_for_map not empty", 2024-02-05T04:53:46.171 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.171990+0000", 2024-02-05T04:53:46.171 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2719999999999999e-06 2024-02-05T04:53:46.171 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.171 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.171 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:46.171 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.175603+0000", 2024-02-05T04:53:46.171 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0036129080000000002 2024-02-05T04:53:46.171 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.171 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.171 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:53:46.171 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.175679+0000", 2024-02-05T04:53:46.172 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6552999999999997e-05 2024-02-05T04:53:46.172 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.172 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.172 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:46.172 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.177747+0000", 2024-02-05T04:53:46.172 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0020672009999999998 2024-02-05T04:53:46.172 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:46.172 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:46.172 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:46.172 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.172 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.172 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:389 3.ds0 3:b4e8e82c:::smithi19639362-25:head [read 0~638976,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-02-05T04:53:46.172 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:34.171905+0000", 2024-02-05T04:53:46.173 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 11.942003188999999, 2024-02-05T04:53:46.173 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0072757830000000001, 2024-02-05T04:53:46.173 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:46.173 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-02-05T04:53:46.173 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:53:46.173 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:53:46.173 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:46.173 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 389 2024-02-05T04:53:46.173 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.173 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:46.173 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.173 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:46.173 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.171905+0000", 2024-02-05T04:53:46.173 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:46.173 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.174 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.174 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:46.174 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.171905+0000", 2024-02-05T04:53:46.174 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:46.174 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.174 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.174 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:46.174 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.171906+0000", 2024-02-05T04:53:46.174 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2499999999999995e-07 2024-02-05T04:53:46.174 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.174 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.174 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:46.174 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.171910+0000", 2024-02-05T04:53:46.174 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4490000000000001e-06 2024-02-05T04:53:46.175 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.175 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.175 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:46.175 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.171918+0000", 2024-02-05T04:53:46.175 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.2449999999999998e-06 2024-02-05T04:53:46.175 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.175 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.175 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:46.175 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.171924+0000", 2024-02-05T04:53:46.175 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6350000000000003e-06 2024-02-05T04:53:46.175 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.175 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.175 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:46.176 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.172022+0000", 2024-02-05T04:53:46.176 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8004000000000006e-05 2024-02-05T04:53:46.176 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.176 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.176 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting_for_map not empty", 2024-02-05T04:53:46.176 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.172026+0000", 2024-02-05T04:53:46.176 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8500000000000004e-06 2024-02-05T04:53:46.176 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.176 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.176 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:46.176 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.177805+0000", 2024-02-05T04:53:46.176 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0057789419999999996 2024-02-05T04:53:46.176 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.176 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.177 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:53:46.177 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.177900+0000", 2024-02-05T04:53:46.177 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5529e-05 2024-02-05T04:53:46.177 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.177 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.177 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:46.177 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.179181+0000", 2024-02-05T04:53:46.177 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0012806040000000001 2024-02-05T04:53:46.177 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:46.177 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:46.177 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:46.177 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.177 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.177 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.bs5 43/31 ECSubWrite(tid=137, reqid=osd.13.0:137, at_version=43'28, trim_to=0'0, roll_forward_to=41'26))", 2024-02-05T04:53:46.177 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:35.171816+0000", 2024-02-05T04:53:46.178 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 10.94209219, 2024-02-05T04:53:46.178 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0057279920000000003, 2024-02-05T04:53:46.178 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:46.178 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:46.178 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:46.178 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.178 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:46.178 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.171816+0000", 2024-02-05T04:53:46.178 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:46.178 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.178 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.178 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:46.178 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.171816+0000", 2024-02-05T04:53:46.178 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:46.178 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.179 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.179 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:46.179 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.171817+0000", 2024-02-05T04:53:46.179 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.89e-07 2024-02-05T04:53:46.179 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.179 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.179 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:46.179 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.171845+0000", 2024-02-05T04:53:46.179 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.7804e-05 2024-02-05T04:53:46.179 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.179 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.179 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:46.179 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.171855+0000", 2024-02-05T04:53:46.179 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0227999999999999e-05 2024-02-05T04:53:46.179 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.179 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.180 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:46.180 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.171865+0000", 2024-02-05T04:53:46.180 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.026e-05 2024-02-05T04:53:46.180 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.180 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.180 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:46.180 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.171958+0000", 2024-02-05T04:53:46.180 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2785e-05 2024-02-05T04:53:46.180 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.180 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.180 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:46.180 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.171969+0000", 2024-02-05T04:53:46.180 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0998e-05 2024-02-05T04:53:46.180 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.180 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.181 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:46.181 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.177502+0000", 2024-02-05T04:53:46.181 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0055336270000000002 2024-02-05T04:53:46.181 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.181 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.181 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:46.181 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.177544+0000", 2024-02-05T04:53:46.181 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1801000000000002e-05 2024-02-05T04:53:46.181 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:46.181 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:46.181 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:46.181 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.181 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.181 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.6s5 43/31 ECSubWrite(tid=138, reqid=osd.13.0:138, at_version=43'24, trim_to=0'0, roll_forward_to=40'22))", 2024-02-05T04:53:46.181 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:35.172477+0000", 2024-02-05T04:53:46.181 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 10.941431486000001, 2024-02-05T04:53:46.182 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0079411910000000002, 2024-02-05T04:53:46.182 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:46.182 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:46.182 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:46.182 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.182 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:46.182 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.172477+0000", 2024-02-05T04:53:46.182 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:46.182 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.182 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.182 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:46.182 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.172477+0000", 2024-02-05T04:53:46.182 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:46.182 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.182 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.183 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:46.183 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.172478+0000", 2024-02-05T04:53:46.183 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9299999999999997e-07 2024-02-05T04:53:46.183 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.183 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.183 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:46.183 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.172502+0000", 2024-02-05T04:53:46.183 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.4071e-05 2024-02-05T04:53:46.183 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.183 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.183 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:46.183 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.172525+0000", 2024-02-05T04:53:46.183 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3666e-05 2024-02-05T04:53:46.183 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.183 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.184 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:46.184 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.172534+0000", 2024-02-05T04:53:46.184 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.7749999999999998e-06 2024-02-05T04:53:46.184 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.184 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.184 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:46.184 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.172612+0000", 2024-02-05T04:53:46.184 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7486999999999999e-05 2024-02-05T04:53:46.184 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.184 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.184 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:46.184 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.172622+0000", 2024-02-05T04:53:46.184 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0040999999999999e-05 2024-02-05T04:53:46.184 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.184 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.185 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:46.185 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.180380+0000", 2024-02-05T04:53:46.185 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0077582739999999999 2024-02-05T04:53:46.185 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.185 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.185 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:46.185 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.180418+0000", 2024-02-05T04:53:46.185 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8183999999999999e-05 2024-02-05T04:53:46.185 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:46.186 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:46.186 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:46.186 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.186 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.186 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.3s1 43/31 ECSubWrite(tid=63, reqid=osd.4.0:63, at_version=43'27, trim_to=0'0, roll_forward_to=40'25))", 2024-02-05T04:53:46.186 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:35.173843+0000", 2024-02-05T04:53:46.186 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 10.940065542999999, 2024-02-05T04:53:46.186 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.006542473, 2024-02-05T04:53:46.186 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:46.186 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:46.186 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:46.186 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.186 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:46.186 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.173843+0000", 2024-02-05T04:53:46.187 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:46.187 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.187 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.187 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:46.187 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.173843+0000", 2024-02-05T04:53:46.187 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:46.187 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.187 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.187 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:46.187 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.173843+0000", 2024-02-05T04:53:46.187 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.18e-07 2024-02-05T04:53:46.187 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.187 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.187 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:46.187 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.173862+0000", 2024-02-05T04:53:46.187 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8770000000000002e-05 2024-02-05T04:53:46.188 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.188 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.188 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:46.188 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.173883+0000", 2024-02-05T04:53:46.188 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1072000000000001e-05 2024-02-05T04:53:46.188 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.188 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.188 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:46.188 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.173890+0000", 2024-02-05T04:53:46.188 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9569999999999996e-06 2024-02-05T04:53:46.188 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.188 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.188 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:46.188 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.173952+0000", 2024-02-05T04:53:46.188 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.2553999999999998e-05 2024-02-05T04:53:46.189 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.189 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.189 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:46.189 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.173960+0000", 2024-02-05T04:53:46.189 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7600000000000002e-06 2024-02-05T04:53:46.189 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.189 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.189 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:46.189 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.180350+0000", 2024-02-05T04:53:46.189 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0063900290000000002 2024-02-05T04:53:46.189 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.189 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.189 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:46.189 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.180385+0000", 2024-02-05T04:53:46.190 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5213000000000003e-05 2024-02-05T04:53:46.190 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:46.190 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:46.190 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:46.190 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.190 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.190 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.3s1 43/31 ECSubWrite(tid=65, reqid=osd.4.0:63, at_version=0'0, trim_to=0'0, roll_forward_to=43'27))", 2024-02-05T04:53:46.190 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:35.182145+0000", 2024-02-05T04:53:46.190 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 10.931763637, 2024-02-05T04:53:46.190 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0036892840000000001, 2024-02-05T04:53:46.190 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:46.190 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:46.190 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:46.191 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.191 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:46.191 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.182145+0000", 2024-02-05T04:53:46.191 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:46.191 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.191 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.191 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:46.191 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.182145+0000", 2024-02-05T04:53:46.191 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:46.191 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.191 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.191 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:46.192 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.182146+0000", 2024-02-05T04:53:46.192 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.136e-06 2024-02-05T04:53:46.192 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.192 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.192 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:46.192 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.182159+0000", 2024-02-05T04:53:46.192 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.277e-05 2024-02-05T04:53:46.192 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.192 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.192 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:46.192 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.182177+0000", 2024-02-05T04:53:46.192 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8102e-05 2024-02-05T04:53:46.192 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.192 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.193 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:46.193 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.182190+0000", 2024-02-05T04:53:46.193 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3543000000000001e-05 2024-02-05T04:53:46.193 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.193 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.193 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:46.193 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.182257+0000", 2024-02-05T04:53:46.193 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7032000000000005e-05 2024-02-05T04:53:46.193 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.193 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.193 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:46.193 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.182267+0000", 2024-02-05T04:53:46.193 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2089999999999994e-06 2024-02-05T04:53:46.193 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.193 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.194 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:46.194 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.185799+0000", 2024-02-05T04:53:46.194 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.003532332 2024-02-05T04:53:46.194 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.194 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.194 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:46.194 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.185834+0000", 2024-02-05T04:53:46.194 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5160000000000002e-05 2024-02-05T04:53:46.194 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:46.194 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:46.194 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:46.194 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.194 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.194 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:401 3.ds0 3:b9c88094:::smithi19639362-50:head [delete] snapc 7=[7,6,4] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e43)", 2024-02-05T04:53:46.194 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:35.182518+0000", 2024-02-05T04:53:46.195 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 10.931390346000001, 2024-02-05T04:53:46.195 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0087143140000000008, 2024-02-05T04:53:46.195 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:46.195 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:46.195 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:53:46.195 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:53:46.195 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:46.195 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 401 2024-02-05T04:53:46.195 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.195 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:46.195 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.195 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:46.195 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.182518+0000", 2024-02-05T04:53:46.195 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:46.196 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.196 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.196 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:46.196 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.182518+0000", 2024-02-05T04:53:46.196 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:46.196 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.196 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.196 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:46.196 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.182519+0000", 2024-02-05T04:53:46.196 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.313e-06 2024-02-05T04:53:46.196 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.196 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.197 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:46.197 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.182524+0000", 2024-02-05T04:53:46.197 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9150000000000004e-06 2024-02-05T04:53:46.197 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.197 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.197 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:46.197 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.182544+0000", 2024-02-05T04:53:46.197 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9327999999999999e-05 2024-02-05T04:53:46.197 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.197 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.197 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:46.197 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.182572+0000", 2024-02-05T04:53:46.197 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8180000000000001e-05 2024-02-05T04:53:46.197 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.197 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.197 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:46.198 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.182643+0000", 2024-02-05T04:53:46.198 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1598000000000001e-05 2024-02-05T04:53:46.198 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.198 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.198 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:53:46.198 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.182744+0000", 2024-02-05T04:53:46.198 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000100881 2024-02-05T04:53:46.198 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.198 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.198 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:46.198 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.183600+0000", 2024-02-05T04:53:46.198 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00085556200000000003 2024-02-05T04:53:46.199 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.199 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.199 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:46.199 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.188714+0000", 2024-02-05T04:53:46.199 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.005113828 2024-02-05T04:53:46.199 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.199 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.199 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T04:53:46.199 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.191180+0000", 2024-02-05T04:53:46.199 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0024662130000000001 2024-02-05T04:53:46.199 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.199 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.199 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:46.199 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.191232+0000", 2024-02-05T04:53:46.199 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2496e-05 2024-02-05T04:53:46.200 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:46.200 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:46.200 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:46.200 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.200 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.200 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.6s5 43/31 ECSubWrite(tid=141, reqid=osd.13.0:138, at_version=0'0, trim_to=0'0, roll_forward_to=43'24))", 2024-02-05T04:53:46.200 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:35.183142+0000", 2024-02-05T04:53:46.200 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 10.930766668, 2024-02-05T04:53:46.200 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0055687339999999997, 2024-02-05T04:53:46.200 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:46.200 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:46.200 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:46.200 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.200 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:46.200 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.183142+0000", 2024-02-05T04:53:46.201 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:46.201 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.201 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.201 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:46.201 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.183142+0000", 2024-02-05T04:53:46.201 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:46.201 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.201 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.201 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:46.201 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.183142+0000", 2024-02-05T04:53:46.201 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3799999999999998e-07 2024-02-05T04:53:46.202 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.202 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.202 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:46.202 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.183153+0000", 2024-02-05T04:53:46.202 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0924000000000001e-05 2024-02-05T04:53:46.202 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.202 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.203 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:46.203 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.183171+0000", 2024-02-05T04:53:46.203 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8199e-05 2024-02-05T04:53:46.203 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.203 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.203 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:46.203 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.183185+0000", 2024-02-05T04:53:46.204 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3594999999999999e-05 2024-02-05T04:53:46.204 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.204 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.204 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:46.204 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.183244+0000", 2024-02-05T04:53:46.204 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8714000000000001e-05 2024-02-05T04:53:46.204 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.205 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.205 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:46.205 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.183253+0000", 2024-02-05T04:53:46.205 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5340000000000005e-06 2024-02-05T04:53:46.205 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.205 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.205 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:46.205 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.188665+0000", 2024-02-05T04:53:46.206 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0054115350000000003 2024-02-05T04:53:46.206 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.206 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.206 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:46.206 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.188711+0000", 2024-02-05T04:53:46.206 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5794999999999998e-05 2024-02-05T04:53:46.206 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:46.207 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:46.207 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:46.207 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.207 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.207 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.3s1 47/31 ECSubWrite(tid=73, reqid=client.4634.0:419, at_version=47'33, trim_to=0'0, roll_forward_to=44'31))", 2024-02-05T04:53:46.207 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:38.944771+0000", 2024-02-05T04:53:46.207 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 7.1691376880000002, 2024-02-05T04:53:46.208 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0045872619999999999, 2024-02-05T04:53:46.208 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:46.208 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:46.208 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:46.208 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.208 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:46.208 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:38.944771+0000", 2024-02-05T04:53:46.209 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:46.209 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.209 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.209 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:46.209 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:38.944771+0000", 2024-02-05T04:53:46.209 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:46.209 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.210 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.210 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:46.210 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:38.944771+0000", 2024-02-05T04:53:46.210 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.9999999999999999e-07 2024-02-05T04:53:46.210 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.210 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.210 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:46.211 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:38.945002+0000", 2024-02-05T04:53:46.211 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00023091499999999999 2024-02-05T04:53:46.211 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.211 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.211 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:46.211 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:38.945013+0000", 2024-02-05T04:53:46.211 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0886999999999999e-05 2024-02-05T04:53:46.212 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.212 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.212 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:46.212 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:38.945022+0000", 2024-02-05T04:53:46.212 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.884e-06 2024-02-05T04:53:46.212 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.212 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.213 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:46.213 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:38.945107+0000", 2024-02-05T04:53:46.213 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.5714e-05 2024-02-05T04:53:46.213 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.213 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.213 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:46.213 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:38.945117+0000", 2024-02-05T04:53:46.214 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.9709999999999999e-06 2024-02-05T04:53:46.214 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.214 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.214 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:46.214 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:38.949321+0000", 2024-02-05T04:53:46.214 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0042032090000000003 2024-02-05T04:53:46.214 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.215 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.215 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:46.215 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:38.949358+0000", 2024-02-05T04:53:46.215 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7382000000000002e-05 2024-02-05T04:53:46.215 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:46.215 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:46.216 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:46.216 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.216 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.216 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.5s4 52/31 ECSubWrite(tid=42, reqid=osd.15.0:42, at_version=52'32, trim_to=0'0, roll_forward_to=51'30))", 2024-02-05T04:53:46.216 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:43.966072+0000", 2024-02-05T04:53:46.216 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 2.1478360689999998, 2024-02-05T04:53:46.216 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0059367960000000003, 2024-02-05T04:53:46.217 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:46.217 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:46.217 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:46.217 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.217 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:46.217 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:43.966072+0000", 2024-02-05T04:53:46.217 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:46.217 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.217 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.218 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:46.218 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:43.966072+0000", 2024-02-05T04:53:46.218 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:46.218 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.218 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.218 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:46.218 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:43.966073+0000", 2024-02-05T04:53:46.219 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7199999999999999e-07 2024-02-05T04:53:46.219 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.219 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.219 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:46.219 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:43.966102+0000", 2024-02-05T04:53:46.219 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.9249000000000001e-05 2024-02-05T04:53:46.219 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.220 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.220 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:46.220 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:43.966113+0000", 2024-02-05T04:53:46.220 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0925e-05 2024-02-05T04:53:46.220 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.220 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.220 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:46.220 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:43.966122+0000", 2024-02-05T04:53:46.221 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1649999999999995e-06 2024-02-05T04:53:46.221 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.221 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.221 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:46.221 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:43.966195+0000", 2024-02-05T04:53:46.221 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.3046999999999994e-05 2024-02-05T04:53:46.221 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.222 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.222 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:46.222 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:43.966205+0000", 2024-02-05T04:53:46.222 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0179e-05 2024-02-05T04:53:46.222 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.222 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.222 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:46.222 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:43.971929+0000", 2024-02-05T04:53:46.223 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0057237190000000004 2024-02-05T04:53:46.223 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:46.223 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:46.223 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:46.223 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:43.972009+0000", 2024-02-05T04:53:46.223 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0039999999999999e-05 2024-02-05T04:53:46.223 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:46.224 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:46.224 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:46.224 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:46.224 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:46.224 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T04:53:46.226 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0: {} 2024-02-05T04:53:46.227 INFO:tasks.ceph.osd.5.smithi091.stderr:2024-02-05T04:53:46.224+0000 7f63983b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34277) UID: 0 2024-02-05T04:53:46.227 INFO:teuthology.orchestra.run.smithi062.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T04:53:46.228 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1: {} 2024-02-05T04:53:46.228 INFO:teuthology.orchestra.run.smithi062.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T04:53:46.228 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-02-05T04:53:46.229 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2: {} 2024-02-05T04:53:46.229 INFO:teuthology.orchestra.run.smithi062.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T04:53:46.229 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3: {} 2024-02-05T04:53:46.230 INFO:teuthology.orchestra.run.smithi062.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T04:53:46.230 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4: {} 2024-02-05T04:53:46.230 INFO:teuthology.orchestra.run.smithi062.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T04:53:46.248 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5: {} 2024-02-05T04:53:46.249 INFO:teuthology.orchestra.run.smithi062.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T04:53:46.266 INFO:teuthology.orchestra.run.smithi062.stdout:osd.6: {} 2024-02-05T04:53:46.267 INFO:teuthology.orchestra.run.smithi062.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T04:53:46.284 INFO:teuthology.orchestra.run.smithi062.stdout:osd.7: {} 2024-02-05T04:53:46.285 INFO:teuthology.orchestra.run.smithi062.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T04:53:46.301 INFO:teuthology.orchestra.run.smithi062.stdout:osd.8: {} 2024-02-05T04:53:46.302 INFO:teuthology.orchestra.run.smithi062.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-05T04:53:46.320 INFO:teuthology.orchestra.run.smithi062.stdout:osd.9: {} 2024-02-05T04:53:46.321 INFO:teuthology.orchestra.run.smithi062.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-05T04:53:46.326 INFO:tasks.ceph.osd.8.smithi062.stderr:2024-02-05T04:53:46.325+0000 7f9385bf2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 34693) UID: 0 2024-02-05T04:53:46.338 INFO:teuthology.orchestra.run.smithi062.stdout:osd.10: {} 2024-02-05T04:53:46.338 INFO:teuthology.orchestra.run.smithi062.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-05T04:53:46.355 INFO:teuthology.orchestra.run.smithi062.stdout:osd.11: {} 2024-02-05T04:53:46.356 INFO:teuthology.orchestra.run.smithi062.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-05T04:53:46.372 INFO:teuthology.orchestra.run.smithi062.stdout:osd.12: {} 2024-02-05T04:53:46.372 INFO:teuthology.orchestra.run.smithi062.stderr:osd.12: osd_enable_op_tracker = '' 2024-02-05T04:53:46.375 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:46.375 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:46.385 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-02-05T04:53:46.390 INFO:teuthology.orchestra.run.smithi062.stdout:osd.13: {} 2024-02-05T04:53:46.390 INFO:teuthology.orchestra.run.smithi062.stderr:osd.13: osd_enable_op_tracker = '' 2024-02-05T04:53:46.409 INFO:teuthology.orchestra.run.smithi062.stdout:osd.14: {} 2024-02-05T04:53:46.409 INFO:teuthology.orchestra.run.smithi062.stderr:osd.14: osd_enable_op_tracker = '' 2024-02-05T04:53:46.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-02-05T04:53:46.408+0000 7f241aa0c640 0 -- 172.21.15.62:0/4043325344 >> [v2:172.21.15.196:6826/3187639683,v1:172.21.15.196:6827/3187639683] conn(0x7f2414221df0 msgr2=0x7f24142241e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-02-05T04:53:46.427 INFO:tasks.ceph.osd.14.smithi150.stderr:2024-02-05T04:53:46.426+0000 7f5f81083640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 34020) UID: 0 2024-02-05T04:53:46.490 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:46.490 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:46.500 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-02-05T04:53:46.528 INFO:tasks.ceph.osd.8.smithi062.stderr:2024-02-05T04:53:46.527+0000 7f9385bf2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 34693) UID: 0 2024-02-05T04:53:46.606 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:46.606 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:46.615 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-02-05T04:53:46.628 INFO:teuthology.orchestra.run.smithi062.stdout:osd.15: {} 2024-02-05T04:53:46.628 INFO:teuthology.orchestra.run.smithi062.stderr:osd.15: osd_enable_op_tracker = '' 2024-02-05T04:53:46.659 INFO:tasks.ceph.osd.7.smithi196.stderr:2024-02-05T04:53:46.658+0000 7fb281419640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34606) UID: 0 2024-02-05T04:53:46.721 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:46.721 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:46.729 INFO:tasks.ceph.osd.11.smithi196.stderr:2024-02-05T04:53:46.728+0000 7f85a8d86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 34609) UID: 0 2024-02-05T04:53:46.730 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-02-05T04:53:46.830 INFO:tasks.ceph.osd.5.smithi091.stderr:2024-02-05T04:53:46.829+0000 7f63983b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34277) UID: 0 2024-02-05T04:53:46.876 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:46.876 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:46.885 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-02-05T04:53:46.931 INFO:tasks.ceph.osd.7.smithi196.stderr:2024-02-05T04:53:46.929+0000 7fb281419640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34606) UID: 0 2024-02-05T04:53:46.960 INFO:tasks.rados.rados.0.smithi196.stdout:197: done (1 left) 2024-02-05T04:53:46.960 INFO:tasks.rados.rados.0.smithi196.stdout:198: delete oid 37 current snap is 14 2024-02-05T04:53:46.967 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-02-05T04:53:46.995 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:46.995 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:47.004 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-02-05T04:53:47.045 INFO:tasks.ceph.osd.15.smithi196.stderr:2024-02-05T04:53:47.044+0000 7f1facc45640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 34611) UID: 0 2024-02-05T04:53:47.119 INFO:teuthology.orchestra.run.smithi062.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:47.119 INFO:teuthology.orchestra.run.smithi062.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:47.128 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-02-05T04:53:47.172 INFO:tasks.ceph.osd.0.smithi062.stderr:2024-02-05T04:53:47.171+0000 7fece8e03640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34688) UID: 0 2024-02-05T04:53:47.232 INFO:tasks.ceph.osd.6.smithi150.stderr:2024-02-05T04:53:47.231+0000 7f6bea9f0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34015) UID: 0 2024-02-05T04:53:47.235 INFO:teuthology.orchestra.run.smithi062.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:47.235 INFO:teuthology.orchestra.run.smithi062.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:47.244 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-02-05T04:53:47.333 INFO:tasks.ceph.osd.15.smithi196.stderr:2024-02-05T04:53:47.332+0000 7f1facc45640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 34611) UID: 0 2024-02-05T04:53:47.392 INFO:teuthology.orchestra.run.smithi062.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:47.392 INFO:teuthology.orchestra.run.smithi062.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:47.401 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-02-05T04:53:47.434 INFO:tasks.ceph.osd.3.smithi196.stderr:2024-02-05T04:53:47.433+0000 7fc8817c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34605) UID: 0 2024-02-05T04:53:47.507 INFO:teuthology.orchestra.run.smithi062.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:47.507 INFO:teuthology.orchestra.run.smithi062.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:47.516 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-02-05T04:53:47.535 INFO:tasks.ceph.osd.15.smithi196.stderr:2024-02-05T04:53:47.535+0000 7f1facc45640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 34611) UID: 0 2024-02-05T04:53:47.623 INFO:teuthology.orchestra.run.smithi062.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:47.623 INFO:teuthology.orchestra.run.smithi062.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:47.631 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-02-05T04:53:47.675 INFO:tasks.ceph.osd.12.smithi062.stderr:2024-02-05T04:53:47.674+0000 7f5e5847e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 34695) UID: 0 2024-02-05T04:53:47.737 INFO:tasks.ceph.osd.1.smithi091.stderr:2024-02-05T04:53:47.736+0000 7ff9dc283640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34275) UID: 0 2024-02-05T04:53:47.738 INFO:teuthology.orchestra.run.smithi062.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:47.738 INFO:teuthology.orchestra.run.smithi062.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:47.746 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-02-05T04:53:47.838 INFO:tasks.ceph.osd.11.smithi196.stderr:2024-02-05T04:53:47.837+0000 7f85a8d86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 34609) UID: 0 2024-02-05T04:53:47.894 INFO:teuthology.orchestra.run.smithi062.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:47.894 INFO:teuthology.orchestra.run.smithi062.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:47.903 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-02-05T04:53:47.947 INFO:tasks.ceph.osd.0.smithi062.stderr:2024-02-05T04:53:47.946+0000 7fece8e03640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34688) UID: 0 2024-02-05T04:53:47.955 INFO:teuthology.orchestra.run.smithi062.stderr:nodeep-scrub is set 2024-02-05T04:53:47.978 INFO:tasks.rados.rados.0.smithi196.stdout:198: done (1 left) 2024-02-05T04:53:47.978 INFO:tasks.rados.rados.0.smithi196.stdout:199: setattr oid 40 current snap is 14 2024-02-05T04:53:47.981 INFO:tasks.rados.rados.0.smithi196.stdout:200: copy_from oid 8 from oid 30 current snap is 14 2024-02-05T04:53:47.981 INFO:tasks.rados.rados.0.smithi196.stdout:201: snap_remove snap 7 2024-02-05T04:53:48.025 INFO:teuthology.orchestra.run.smithi062.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:48.026 INFO:teuthology.orchestra.run.smithi062.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:48.036 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-02-05T04:53:48.040 INFO:tasks.ceph.osd.9.smithi091.stderr:2024-02-05T04:53:48.038+0000 7efc18827640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 34279) UID: 0 2024-02-05T04:53:48.141 INFO:tasks.ceph.osd.8.smithi062.stderr:2024-02-05T04:53:48.141+0000 7f9385bf2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 34693) UID: 0 2024-02-05T04:53:48.145 INFO:teuthology.orchestra.run.smithi062.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:48.145 INFO:teuthology.orchestra.run.smithi062.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:48.154 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-02-05T04:53:48.243 INFO:tasks.ceph.osd.9.smithi091.stderr:2024-02-05T04:53:48.242+0000 7efc18827640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 34279) UID: 0 2024-02-05T04:53:48.301 INFO:teuthology.orchestra.run.smithi062.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:48.301 INFO:teuthology.orchestra.run.smithi062.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:48.310 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-02-05T04:53:48.353 INFO:tasks.ceph.osd.12.smithi062.stderr:2024-02-05T04:53:48.352+0000 7f5e5847e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 34695) UID: 0 2024-02-05T04:53:48.416 INFO:teuthology.orchestra.run.smithi062.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:48.416 INFO:teuthology.orchestra.run.smithi062.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:48.424 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-02-05T04:53:48.466 INFO:tasks.ceph.osd.8.smithi062.stderr:2024-02-05T04:53:48.465+0000 7f9385bf2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 34693) UID: 0 2024-02-05T04:53:48.546 INFO:tasks.ceph.osd.10.smithi150.stderr:2024-02-05T04:53:48.545+0000 7f3075a4d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 34018) UID: 0 2024-02-05T04:53:48.572 INFO:teuthology.orchestra.run.smithi062.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:48.572 INFO:teuthology.orchestra.run.smithi062.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:48.581 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-02-05T04:53:48.639 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T04:53:48.684 INFO:tasks.ceph.osd.8.smithi062.stderr:2024-02-05T04:53:48.683+0000 7f9385bf2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 34693) UID: 0 2024-02-05T04:53:48.697 INFO:teuthology.orchestra.run.smithi091.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:48.697 INFO:teuthology.orchestra.run.smithi091.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:48.707 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-02-05T04:53:48.753 INFO:tasks.ceph.osd.9.smithi091.stderr:2024-02-05T04:53:48.752+0000 7efc18827640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 34279) UID: 0 2024-02-05T04:53:48.764 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0: {} 2024-02-05T04:53:48.764 INFO:teuthology.orchestra.run.smithi062.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T04:53:48.783 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1: {} 2024-02-05T04:53:48.783 INFO:teuthology.orchestra.run.smithi062.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T04:53:48.801 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2: {} 2024-02-05T04:53:48.801 INFO:teuthology.orchestra.run.smithi062.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T04:53:48.814 INFO:teuthology.orchestra.run.smithi091.stdout:{ 2024-02-05T04:53:48.814 INFO:teuthology.orchestra.run.smithi091.stdout: "ops": [], 2024-02-05T04:53:48.814 INFO:teuthology.orchestra.run.smithi091.stdout: "complaint_time": 30, 2024-02-05T04:53:48.814 INFO:teuthology.orchestra.run.smithi091.stdout: "num_blocked_ops": 0 2024-02-05T04:53:48.814 INFO:teuthology.orchestra.run.smithi091.stdout:} 2024-02-05T04:53:48.819 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3: {} 2024-02-05T04:53:48.819 INFO:teuthology.orchestra.run.smithi062.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T04:53:48.823 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-02-05T04:53:48.836 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4: {} 2024-02-05T04:53:48.836 INFO:teuthology.orchestra.run.smithi062.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T04:53:48.850 INFO:tasks.ceph.osd.11.smithi196.stderr:2024-02-05T04:53:48.849+0000 7f85a8d86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 34609) UID: 0 2024-02-05T04:53:48.854 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5: {} 2024-02-05T04:53:48.854 INFO:teuthology.orchestra.run.smithi062.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T04:53:48.872 INFO:teuthology.orchestra.run.smithi062.stdout:osd.6: {} 2024-02-05T04:53:48.872 INFO:teuthology.orchestra.run.smithi062.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T04:53:48.890 INFO:teuthology.orchestra.run.smithi062.stdout:osd.7: {} 2024-02-05T04:53:48.890 INFO:teuthology.orchestra.run.smithi062.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T04:53:48.906 INFO:teuthology.orchestra.run.smithi062.stdout:osd.8: {} 2024-02-05T04:53:48.906 INFO:teuthology.orchestra.run.smithi062.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-05T04:53:48.923 INFO:teuthology.orchestra.run.smithi062.stdout:osd.9: {} 2024-02-05T04:53:48.924 INFO:teuthology.orchestra.run.smithi062.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-05T04:53:48.942 INFO:teuthology.orchestra.run.smithi062.stdout:osd.10: {} 2024-02-05T04:53:48.943 INFO:teuthology.orchestra.run.smithi062.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-05T04:53:48.944 INFO:tasks.rados.rados.0.smithi196.stdout:201: done (3 left) 2024-02-05T04:53:48.944 INFO:tasks.rados.rados.0.smithi196.stdout:202: read oid 50 snap -1 2024-02-05T04:53:48.944 INFO:tasks.rados.rados.0.smithi196.stdout:202: expect deleted 2024-02-05T04:53:48.944 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 40 v 30 (ObjNum 66 snap 2 seq_num 66) dirty exists 2024-02-05T04:53:48.944 INFO:tasks.rados.rados.0.smithi196.stdout:199: done (3 left) 2024-02-05T04:53:48.944 INFO:tasks.rados.rados.0.smithi196.stdout:203: delete oid 41 current snap is 14 2024-02-05T04:53:48.944 INFO:tasks.rados.rados.0.smithi196.stdout:195: finishing rollback tid 1 to smithi19639362-3 2024-02-05T04:53:48.944 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 3 v 0 (ObjNum 129 snap 0 seq_num 3192812832) dirty dne 2024-02-05T04:53:48.944 INFO:tasks.rados.rados.0.smithi196.stdout:200: finishing copy_from racing read to smithi19639362-8 2024-02-05T04:53:48.944 INFO:tasks.rados.rados.0.smithi196.stdout:200: finishing copy_from to smithi19639362-8 2024-02-05T04:53:48.944 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 8 v 20 (ObjNum 60 snap 2 seq_num 60) dirty exists 2024-02-05T04:53:48.952 INFO:tasks.ceph.osd.9.smithi091.stderr:2024-02-05T04:53:48.950+0000 7efc18827640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 34279) UID: 0 2024-02-05T04:53:48.958 INFO:tasks.rados.rados.0.smithi196.stdout:195: done (3 left) 2024-02-05T04:53:48.958 INFO:tasks.rados.rados.0.smithi196.stdout:200: done (2 left) 2024-02-05T04:53:48.958 INFO:tasks.rados.rados.0.smithi196.stdout:202: done (1 left) 2024-02-05T04:53:48.958 INFO:tasks.rados.rados.0.smithi196.stdout:203: done (0 left) 2024-02-05T04:53:48.958 INFO:tasks.rados.rados.0.smithi196.stdout:204: copy_from oid 36 from oid 33 current snap is 14 2024-02-05T04:53:48.958 INFO:tasks.rados.rados.0.smithi196.stdout:205: copy_from oid 28 from oid 8 current snap is 14 2024-02-05T04:53:48.958 INFO:tasks.rados.rados.0.smithi196.stdout:206: read oid 48 snap -1 2024-02-05T04:53:48.958 INFO:tasks.rados.rados.0.smithi196.stdout:206: expect (ObjNum 49 snap 0 seq_num 49) 2024-02-05T04:53:48.958 INFO:tasks.rados.rados.0.smithi196.stdout:207: delete oid 44 current snap is 14 2024-02-05T04:53:48.966 INFO:tasks.rados.rados.0.smithi196.stdout:207: done (3 left) 2024-02-05T04:53:48.966 INFO:tasks.rados.rados.0.smithi196.stdout:208: copy_from oid 40 from oid 23 current snap is 14 2024-02-05T04:53:48.966 INFO:tasks.rados.rados.0.smithi196.stdout:209: read oid 24 snap -1 2024-02-05T04:53:48.966 INFO:tasks.rados.rados.0.smithi196.stdout:209: expect (ObjNum 23 snap 0 seq_num 23) 2024-02-05T04:53:48.967 INFO:tasks.rados.rados.0.smithi196.stdout:append oid 4 current snap is 14 2024-02-05T04:53:48.967 INFO:tasks.rados.rados.0.smithi196.stdout:210: seq_num 105 ranges {540672=606208,1146880=688128,1835008=475136} 2024-02-05T04:53:48.968 INFO:teuthology.orchestra.run.smithi062.stdout:osd.11: {} 2024-02-05T04:53:48.968 INFO:teuthology.orchestra.run.smithi062.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-05T04:53:48.973 INFO:tasks.rados.rados.0.smithi196.stdout:210: writing smithi19639362-4 from 540672 to 1146880 tid 1 2024-02-05T04:53:48.977 INFO:tasks.rados.rados.0.smithi196.stdout:210: writing smithi19639362-4 from 1146880 to 1835008 tid 2 2024-02-05T04:53:48.980 INFO:tasks.rados.rados.0.smithi196.stdout:210: writing smithi19639362-4 from 1835008 to 2310144 tid 3 2024-02-05T04:53:48.980 INFO:tasks.rados.rados.0.smithi196.stdout:211: snap_create 2024-02-05T04:53:48.980 INFO:tasks.rados.rados.0.smithi196.stdout:206: expect (ObjNum 49 snap 0 seq_num 49) 2024-02-05T04:53:48.982 INFO:teuthology.orchestra.run.smithi091.stdout:{ 2024-02-05T04:53:48.982 INFO:teuthology.orchestra.run.smithi091.stdout: "size": 20, 2024-02-05T04:53:48.982 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 600, 2024-02-05T04:53:48.982 INFO:teuthology.orchestra.run.smithi091.stdout: "ops": [ 2024-02-05T04:53:48.983 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:48.983 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:216 3.4s0 3:25248d28:::smithi19639362-18:head [read 0~294912,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-02-05T04:53:48.983 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:28.122406+0000", 2024-02-05T04:53:48.983 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 20.855574636, 2024-02-05T04:53:48.983 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0056680589999999996, 2024-02-05T04:53:48.983 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:48.984 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "started", 2024-02-05T04:53:48.984 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:53:48.984 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:53:48.984 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:48.984 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 216 2024-02-05T04:53:48.984 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:48.984 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:48.984 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:48.984 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:48.984 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.122406+0000", 2024-02-05T04:53:48.984 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:48.984 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:48.984 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:48.985 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:48.985 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.122406+0000", 2024-02-05T04:53:48.985 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:48.985 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:48.985 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:48.985 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:48.985 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.122406+0000", 2024-02-05T04:53:48.985 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.8000000000000001e-07 2024-02-05T04:53:48.985 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:48.985 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:48.985 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:48.985 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.122408+0000", 2024-02-05T04:53:48.985 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.4600000000000002e-06 2024-02-05T04:53:48.985 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:48.985 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:48.986 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:48.986 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.122413+0000", 2024-02-05T04:53:48.986 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.0209999999999999e-06 2024-02-05T04:53:48.986 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:48.986 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:48.986 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:48.986 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.122417+0000", 2024-02-05T04:53:48.986 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.6239999999999999e-06 2024-02-05T04:53:48.986 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:48.986 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:48.986 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:48.986 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.122541+0000", 2024-02-05T04:53:48.986 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00012391100000000001 2024-02-05T04:53:48.986 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:48.986 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:48.987 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "waiting_for_map not empty", 2024-02-05T04:53:48.987 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.122547+0000", 2024-02-05T04:53:48.987 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.682e-06 2024-02-05T04:53:48.987 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:48.987 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:48.987 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:48.987 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.127053+0000", 2024-02-05T04:53:48.988 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0045067170000000004 2024-02-05T04:53:48.988 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:48.988 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:48.988 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:53:48.988 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.127105+0000", 2024-02-05T04:53:48.988 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.1458000000000001e-05 2024-02-05T04:53:48.989 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:48.989 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:48.989 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:48.989 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:28.128074+0000", 2024-02-05T04:53:48.989 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00096880600000000001 2024-02-05T04:53:48.989 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:48.989 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:48.990 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:48.990 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:48.990 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:48.990 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.3s2 37/31 ECSubWrite(tid=30, reqid=client.4634.0:219, at_version=37'13, trim_to=0'0, roll_forward_to=37'10))", 2024-02-05T04:53:48.990 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:29.136955+0000", 2024-02-05T04:53:48.990 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 19.841024815000001, 2024-02-05T04:53:48.991 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0062065059999999997, 2024-02-05T04:53:48.991 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:48.991 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:48.991 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:48.991 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:48.991 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:48.991 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.136955+0000", 2024-02-05T04:53:48.992 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:48.992 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:48.992 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:48.992 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:48.992 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.136955+0000", 2024-02-05T04:53:48.992 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:48.993 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:48.993 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:48.993 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:48.993 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.136956+0000", 2024-02-05T04:53:48.993 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.8299999999999996e-07 2024-02-05T04:53:48.993 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:48.993 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:48.994 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:48.994 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.137016+0000", 2024-02-05T04:53:48.994 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.961e-05 2024-02-05T04:53:48.994 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:48.994 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:48.994 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:48.995 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.137030+0000", 2024-02-05T04:53:48.995 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.4121000000000001e-05 2024-02-05T04:53:48.995 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:48.995 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:48.995 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:48.995 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.137040+0000", 2024-02-05T04:53:48.995 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.7350000000000008e-06 2024-02-05T04:53:48.996 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:48.996 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:48.996 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:48.996 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.137102+0000", 2024-02-05T04:53:48.996 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.1977000000000006e-05 2024-02-05T04:53:48.996 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:48.997 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:48.997 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:48.997 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.137111+0000", 2024-02-05T04:53:48.997 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.7690000000000008e-06 2024-02-05T04:53:48.997 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:48.997 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:48.998 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:48.998 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.143129+0000", 2024-02-05T04:53:48.998 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0060175410000000004 2024-02-05T04:53:48.998 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:48.998 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:48.998 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:48.998 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.143162+0000", 2024-02-05T04:53:48.999 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.307e-05 2024-02-05T04:53:48.999 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:48.999 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:48.999 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:48.999 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:48.999 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.000 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:239 3.4s0 3:2c242b83:::smithi19639362-9:head [append 0~655360] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:53:49.000 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:29.155211+0000", 2024-02-05T04:53:49.000 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 19.822769482999998, 2024-02-05T04:53:49.000 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0067722930000000004, 2024-02-05T04:53:49.000 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:49.000 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:49.001 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:53:49.001 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:53:49.001 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:49.001 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 239 2024-02-05T04:53:49.001 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.001 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:49.002 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.002 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:49.002 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.155211+0000", 2024-02-05T04:53:49.002 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.002 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.002 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.002 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:49.003 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.155211+0000", 2024-02-05T04:53:49.003 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.003 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.003 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.003 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:49.003 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.155212+0000", 2024-02-05T04:53:49.004 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.302e-06 2024-02-05T04:53:49.004 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.004 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.004 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:49.004 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.156080+0000", 2024-02-05T04:53:49.004 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00086819000000000002 2024-02-05T04:53:49.005 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.005 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.005 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:49.005 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.156094+0000", 2024-02-05T04:53:49.005 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.3637e-05 2024-02-05T04:53:49.005 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.006 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.006 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:49.006 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.156101+0000", 2024-02-05T04:53:49.006 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.5619999999999998e-06 2024-02-05T04:53:49.006 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.006 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.006 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:49.007 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.156158+0000", 2024-02-05T04:53:49.007 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.6595999999999998e-05 2024-02-05T04:53:49.007 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.007 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.007 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:53:49.007 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.156235+0000", 2024-02-05T04:53:49.008 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.7181999999999994e-05 2024-02-05T04:53:49.008 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.008 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.008 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:49.008 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.157325+0000", 2024-02-05T04:53:49.009 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.001089558 2024-02-05T04:53:49.009 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.009 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.009 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:49.009 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.160456+0000", 2024-02-05T04:53:49.009 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.003131694 2024-02-05T04:53:49.010 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.010 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.010 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "commit_sent", 2024-02-05T04:53:49.010 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.161914+0000", 2024-02-05T04:53:49.010 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.001457814 2024-02-05T04:53:49.010 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.011 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.011 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:49.011 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.161983+0000", 2024-02-05T04:53:49.011 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.8757999999999995e-05 2024-02-05T04:53:49.011 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.011 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:49.012 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.012 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.012 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.012 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:242 3.4s0 3:2c242b83:::smithi19639362-9:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:53:49.012 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:29.158493+0000", 2024-02-05T04:53:49.012 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 19.819487250999998, 2024-02-05T04:53:49.013 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0052039679999999998, 2024-02-05T04:53:49.013 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:49.013 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "started", 2024-02-05T04:53:49.013 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:53:49.013 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:53:49.013 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:49.014 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 242 2024-02-05T04:53:49.014 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.014 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:49.014 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.014 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:49.014 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.158493+0000", 2024-02-05T04:53:49.015 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.015 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.015 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.015 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:49.015 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.158493+0000", 2024-02-05T04:53:49.015 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.016 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.016 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.016 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:49.016 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.158494+0000", 2024-02-05T04:53:49.016 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.06e-07 2024-02-05T04:53:49.016 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.017 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.017 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:49.017 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.158496+0000", 2024-02-05T04:53:49.017 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.3709999999999998e-06 2024-02-05T04:53:49.017 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.017 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.018 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:49.018 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.158503+0000", 2024-02-05T04:53:49.018 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.1459999999999999e-06 2024-02-05T04:53:49.018 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.018 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.018 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:49.019 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.158509+0000", 2024-02-05T04:53:49.019 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.7960000000000001e-06 2024-02-05T04:53:49.019 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.019 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.019 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:49.019 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.160639+0000", 2024-02-05T04:53:49.020 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0021304269999999998 2024-02-05T04:53:49.020 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.020 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.020 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "waiting for rw locks", 2024-02-05T04:53:49.020 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.160718+0000", 2024-02-05T04:53:49.020 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.8591999999999996e-05 2024-02-05T04:53:49.021 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.021 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.021 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:49.021 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.162682+0000", 2024-02-05T04:53:49.021 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.001964155 2024-02-05T04:53:49.021 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.022 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.022 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:53:49.022 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.162771+0000", 2024-02-05T04:53:49.022 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.8739999999999994e-05 2024-02-05T04:53:49.022 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.022 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.023 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:49.023 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.163697+0000", 2024-02-05T04:53:49.023 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00092623499999999997 2024-02-05T04:53:49.023 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.023 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:49.023 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.023 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.023 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.023 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:284 3.4s0 3:264f0b4b:::smithi19639362-35:head [setxattr _header (56) in=7b] snapc 3=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-02-05T04:53:49.024 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:30.205620+0000", 2024-02-05T04:53:49.024 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 18.772360536000001, 2024-02-05T04:53:49.024 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0067013009999999998, 2024-02-05T04:53:49.024 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:49.024 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:49.024 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:53:49.024 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:53:49.024 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:49.024 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 284 2024-02-05T04:53:49.024 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.025 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:49.025 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.025 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:49.025 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.205620+0000", 2024-02-05T04:53:49.025 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.025 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.025 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.025 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:49.025 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.205620+0000", 2024-02-05T04:53:49.025 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.025 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.026 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.026 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:49.026 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.205621+0000", 2024-02-05T04:53:49.026 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.8800000000000001e-07 2024-02-05T04:53:49.026 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.026 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.026 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:49.026 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.205626+0000", 2024-02-05T04:53:49.026 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.028e-06 2024-02-05T04:53:49.026 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.027 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.027 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:49.027 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.205634+0000", 2024-02-05T04:53:49.027 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.8820000000000001e-06 2024-02-05T04:53:49.027 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.027 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.027 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:49.027 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.205639+0000", 2024-02-05T04:53:49.027 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.6670000000000001e-06 2024-02-05T04:53:49.027 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.027 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.028 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:49.028 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.207319+0000", 2024-02-05T04:53:49.028 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.001680029 2024-02-05T04:53:49.028 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.028 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.028 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:53:49.028 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.207416+0000", 2024-02-05T04:53:49.028 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.6576000000000005e-05 2024-02-05T04:53:49.028 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.029 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.029 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:49.029 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.207826+0000", 2024-02-05T04:53:49.029 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00041016499999999999 2024-02-05T04:53:49.029 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.029 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.029 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:49.029 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.208446+0000", 2024-02-05T04:53:49.029 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00061980300000000004 2024-02-05T04:53:49.029 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.030 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.030 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "commit_sent", 2024-02-05T04:53:49.030 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.212268+0000", 2024-02-05T04:53:49.030 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0038218890000000002 2024-02-05T04:53:49.030 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.030 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.030 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:49.030 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.212321+0000", 2024-02-05T04:53:49.030 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.3374000000000001e-05 2024-02-05T04:53:49.030 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.031 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:49.031 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.031 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.031 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.031 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:285 3.4s0 3:264f0b4b:::smithi19639362-35:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-02-05T04:53:49.031 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:30.205648+0000", 2024-02-05T04:53:49.031 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 18.77233232, 2024-02-05T04:53:49.031 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0099691669999999993, 2024-02-05T04:53:49.031 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:49.031 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "started", 2024-02-05T04:53:49.031 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:53:49.032 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:53:49.032 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:49.032 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 285 2024-02-05T04:53:49.032 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.032 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:49.032 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.032 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:49.032 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.205648+0000", 2024-02-05T04:53:49.032 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.033 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.033 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.033 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:49.033 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.205648+0000", 2024-02-05T04:53:49.033 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.033 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.033 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.033 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:49.033 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.205648+0000", 2024-02-05T04:53:49.033 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.7100000000000002e-07 2024-02-05T04:53:49.034 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.034 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.034 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:49.034 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.205651+0000", 2024-02-05T04:53:49.034 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.277e-06 2024-02-05T04:53:49.034 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.034 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.034 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:49.034 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.205658+0000", 2024-02-05T04:53:49.035 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.1709999999999996e-06 2024-02-05T04:53:49.035 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.035 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.035 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:49.035 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.205663+0000", 2024-02-05T04:53:49.035 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.4009999999999996e-06 2024-02-05T04:53:49.036 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.036 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.036 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:49.036 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.208044+0000", 2024-02-05T04:53:49.036 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.002380957 2024-02-05T04:53:49.036 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.037 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.037 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "waiting for rw locks", 2024-02-05T04:53:49.037 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.208122+0000", 2024-02-05T04:53:49.037 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.7688999999999994e-05 2024-02-05T04:53:49.037 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.037 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.038 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:49.038 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.212367+0000", 2024-02-05T04:53:49.038 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0042449569999999997 2024-02-05T04:53:49.038 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.038 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.038 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:53:49.039 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.212502+0000", 2024-02-05T04:53:49.039 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00013503199999999999 2024-02-05T04:53:49.039 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.039 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.039 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:49.039 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.215617+0000", 2024-02-05T04:53:49.040 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0031151120000000002 2024-02-05T04:53:49.040 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.040 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:49.040 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.040 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.040 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.041 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:289 3.4s0 3:2c15cb2e:::smithi19639362-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-02-05T04:53:49.041 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:30.212758+0000", 2024-02-05T04:53:49.041 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 18.765222704999999, 2024-02-05T04:53:49.041 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0060231160000000002, 2024-02-05T04:53:49.041 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:49.041 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "started", 2024-02-05T04:53:49.042 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:53:49.042 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:53:49.042 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:49.042 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 289 2024-02-05T04:53:49.042 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.042 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:49.043 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.043 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:49.043 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.212758+0000", 2024-02-05T04:53:49.043 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.043 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.043 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.044 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:49.044 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.212758+0000", 2024-02-05T04:53:49.044 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.044 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.044 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.044 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:49.044 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.212758+0000", 2024-02-05T04:53:49.044 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.0400000000000002e-07 2024-02-05T04:53:49.045 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.045 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.045 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:49.045 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.212760+0000", 2024-02-05T04:53:49.045 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.8500000000000001e-06 2024-02-05T04:53:49.045 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.045 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.045 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:49.045 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.212765+0000", 2024-02-05T04:53:49.045 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.4920000000000001e-06 2024-02-05T04:53:49.045 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.046 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.046 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:49.046 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.212768+0000", 2024-02-05T04:53:49.046 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.112e-06 2024-02-05T04:53:49.046 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.046 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.046 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:49.046 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.215646+0000", 2024-02-05T04:53:49.046 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.002877395 2024-02-05T04:53:49.046 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.046 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.046 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "waiting for rw locks", 2024-02-05T04:53:49.046 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.215694+0000", 2024-02-05T04:53:49.046 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.7831999999999999e-05 2024-02-05T04:53:49.046 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.046 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.047 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:49.047 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.217839+0000", 2024-02-05T04:53:49.047 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0021454450000000002 2024-02-05T04:53:49.047 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.047 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.047 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:53:49.047 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.217915+0000", 2024-02-05T04:53:49.047 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.6347000000000006e-05 2024-02-05T04:53:49.047 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.047 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.047 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:49.047 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.218781+0000", 2024-02-05T04:53:49.047 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00086523899999999996 2024-02-05T04:53:49.047 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.047 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:49.048 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.048 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.048 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.048 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:393 3.4s0 3:2c242b83:::smithi19639362-9:head [append 0~770048] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-02-05T04:53:49.048 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:34.186553+0000", 2024-02-05T04:53:49.048 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 14.791426859, 2024-02-05T04:53:49.048 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0062969649999999999, 2024-02-05T04:53:49.048 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:49.048 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:49.048 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:53:49.048 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:53:49.048 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:49.048 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 393 2024-02-05T04:53:49.048 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.048 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:49.049 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.049 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:49.049 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.186553+0000", 2024-02-05T04:53:49.049 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.049 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.049 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.049 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:49.049 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.186553+0000", 2024-02-05T04:53:49.049 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.049 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.049 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.049 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:49.049 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.186555+0000", 2024-02-05T04:53:49.049 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.5960000000000001e-06 2024-02-05T04:53:49.049 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.049 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.050 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:49.050 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.187770+0000", 2024-02-05T04:53:49.050 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0012145389999999999 2024-02-05T04:53:49.050 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.050 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.050 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:49.050 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.187786+0000", 2024-02-05T04:53:49.050 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.6719e-05 2024-02-05T04:53:49.050 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.050 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.050 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:49.050 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.187795+0000", 2024-02-05T04:53:49.050 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.1649999999999995e-06 2024-02-05T04:53:49.050 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.050 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.050 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:49.051 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.187861+0000", 2024-02-05T04:53:49.051 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.5399000000000002e-05 2024-02-05T04:53:49.051 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.051 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.051 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:53:49.051 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.187956+0000", 2024-02-05T04:53:49.051 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.4853999999999997e-05 2024-02-05T04:53:49.051 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.051 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.051 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:49.051 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.188717+0000", 2024-02-05T04:53:49.051 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00076123399999999996 2024-02-05T04:53:49.051 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.051 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.051 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:49.051 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.191130+0000", 2024-02-05T04:53:49.052 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0024133660000000001 2024-02-05T04:53:49.052 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.052 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.052 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "commit_sent", 2024-02-05T04:53:49.052 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.192780+0000", 2024-02-05T04:53:49.052 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0016500320000000001 2024-02-05T04:53:49.052 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.052 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.052 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:49.052 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.192850+0000", 2024-02-05T04:53:49.052 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.0061000000000006e-05 2024-02-05T04:53:49.052 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.052 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:49.052 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.052 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.052 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.053 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:394 3.4s0 3:2c242b83:::smithi19639362-9:head [setxattr _header (55) in=7b] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-02-05T04:53:49.053 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:34.187824+0000", 2024-02-05T04:53:49.053 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 14.790155996999999, 2024-02-05T04:53:49.053 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0063472709999999998, 2024-02-05T04:53:49.053 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:49.053 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:49.053 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:53:49.053 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:53:49.053 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:49.053 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 394 2024-02-05T04:53:49.053 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.053 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:49.053 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.053 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:49.054 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.187824+0000", 2024-02-05T04:53:49.054 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.054 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.054 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.054 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:49.054 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.187824+0000", 2024-02-05T04:53:49.054 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.054 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.054 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.054 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:49.054 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.187826+0000", 2024-02-05T04:53:49.054 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.342e-06 2024-02-05T04:53:49.054 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.054 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.054 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:49.054 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.187832+0000", 2024-02-05T04:53:49.055 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.2040000000000002e-06 2024-02-05T04:53:49.055 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.055 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.055 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:49.055 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.187841+0000", 2024-02-05T04:53:49.055 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.0529999999999996e-06 2024-02-05T04:53:49.055 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.055 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.055 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:49.055 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.187847+0000", 2024-02-05T04:53:49.055 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.3570000000000003e-06 2024-02-05T04:53:49.055 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.055 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.055 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:49.055 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.189963+0000", 2024-02-05T04:53:49.055 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0021155969999999999 2024-02-05T04:53:49.056 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.056 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.056 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:53:49.056 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.190114+0000", 2024-02-05T04:53:49.056 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00015165600000000001 2024-02-05T04:53:49.056 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.056 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.056 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:49.056 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.190493+0000", 2024-02-05T04:53:49.056 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00037816 2024-02-05T04:53:49.056 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.056 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.056 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:49.056 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.192865+0000", 2024-02-05T04:53:49.056 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0023725840000000001 2024-02-05T04:53:49.057 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.057 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.057 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "commit_sent", 2024-02-05T04:53:49.057 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.194107+0000", 2024-02-05T04:53:49.057 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.001242139 2024-02-05T04:53:49.057 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.057 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.057 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:49.057 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.194172+0000", 2024-02-05T04:53:49.057 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.4178999999999997e-05 2024-02-05T04:53:49.057 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.057 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:49.057 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.057 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.057 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.057 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:395 3.4s0 3:2c242b83:::smithi19639362-9:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-02-05T04:53:49.058 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:34.187868+0000", 2024-02-05T04:53:49.058 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 14.79011266, 2024-02-05T04:53:49.058 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0080281970000000008, 2024-02-05T04:53:49.058 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:49.058 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "started", 2024-02-05T04:53:49.058 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:53:49.058 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:53:49.058 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:49.058 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 395 2024-02-05T04:53:49.058 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.058 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:49.058 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.058 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:49.058 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.187868+0000", 2024-02-05T04:53:49.058 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.058 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.059 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.059 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:49.059 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.187868+0000", 2024-02-05T04:53:49.059 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.059 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.059 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.059 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:49.059 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.187868+0000", 2024-02-05T04:53:49.059 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.6499999999999999e-07 2024-02-05T04:53:49.059 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.059 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.059 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:49.059 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.187872+0000", 2024-02-05T04:53:49.059 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.9659999999999998e-06 2024-02-05T04:53:49.059 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.059 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.060 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:49.060 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.187881+0000", 2024-02-05T04:53:49.060 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.8519999999999993e-06 2024-02-05T04:53:49.060 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.060 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.060 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:49.060 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.187887+0000", 2024-02-05T04:53:49.060 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.3579999999999999e-06 2024-02-05T04:53:49.060 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.060 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.060 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:49.060 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.191261+0000", 2024-02-05T04:53:49.060 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.003373095 2024-02-05T04:53:49.060 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.060 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.061 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "waiting for rw locks", 2024-02-05T04:53:49.061 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.191336+0000", 2024-02-05T04:53:49.061 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.5731999999999999e-05 2024-02-05T04:53:49.061 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.061 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.061 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:49.061 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.194503+0000", 2024-02-05T04:53:49.061 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0031670069999999999 2024-02-05T04:53:49.061 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.061 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.061 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:53:49.061 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.194590+0000", 2024-02-05T04:53:49.061 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.6650000000000006e-05 2024-02-05T04:53:49.061 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.061 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.061 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:49.062 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.195896+0000", 2024-02-05T04:53:49.062 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.001305872 2024-02-05T04:53:49.062 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.062 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:49.062 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.062 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.062 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.062 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.6s4 43/31 ECSubWrite(tid=138, reqid=osd.13.0:138, at_version=43'24, trim_to=0'0, roll_forward_to=40'22))", 2024-02-05T04:53:49.062 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:35.172360+0000", 2024-02-05T04:53:49.062 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 13.805620455, 2024-02-05T04:53:49.062 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00794804, 2024-02-05T04:53:49.062 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:49.062 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:49.062 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:49.062 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.062 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:49.063 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.172360+0000", 2024-02-05T04:53:49.063 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.063 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.063 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.063 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:49.063 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.172360+0000", 2024-02-05T04:53:49.063 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.063 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.063 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.063 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:49.063 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.172360+0000", 2024-02-05T04:53:49.063 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.9400000000000005e-07 2024-02-05T04:53:49.063 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.063 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.063 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:49.063 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.172384+0000", 2024-02-05T04:53:49.064 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.3186000000000001e-05 2024-02-05T04:53:49.064 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.064 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.064 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:49.064 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.172425+0000", 2024-02-05T04:53:49.064 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.1069000000000003e-05 2024-02-05T04:53:49.064 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.064 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.064 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:49.064 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.172435+0000", 2024-02-05T04:53:49.064 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.9150000000000008e-06 2024-02-05T04:53:49.064 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.064 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.064 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:49.064 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.172511+0000", 2024-02-05T04:53:49.065 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.6118000000000005e-05 2024-02-05T04:53:49.065 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.065 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.065 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:49.065 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.172521+0000", 2024-02-05T04:53:49.065 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.0429e-05 2024-02-05T04:53:49.065 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.065 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.065 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:49.065 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.180273+0000", 2024-02-05T04:53:49.065 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0077523109999999996 2024-02-05T04:53:49.065 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.065 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.065 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:49.065 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.180308+0000", 2024-02-05T04:53:49.066 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.4418000000000001e-05 2024-02-05T04:53:49.066 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.066 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:49.066 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.066 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.066 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.066 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.8s1 43/31 ECSubWrite(tid=62, reqid=osd.4.0:62, at_version=43'19, trim_to=0'0, roll_forward_to=42'17))", 2024-02-05T04:53:49.066 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:35.172380+0000", 2024-02-05T04:53:49.066 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 13.805600675000001, 2024-02-05T04:53:49.066 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0081089249999999995, 2024-02-05T04:53:49.066 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:49.066 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:49.066 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:49.066 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.066 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:49.067 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.172380+0000", 2024-02-05T04:53:49.067 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.067 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.067 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.067 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:49.067 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.172380+0000", 2024-02-05T04:53:49.067 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.067 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.067 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.067 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:49.067 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.172380+0000", 2024-02-05T04:53:49.067 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.48e-07 2024-02-05T04:53:49.067 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.067 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.067 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:49.067 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.172399+0000", 2024-02-05T04:53:49.068 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.9377000000000001e-05 2024-02-05T04:53:49.068 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.068 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.068 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:49.068 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.172407+0000", 2024-02-05T04:53:49.068 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.7729999999999999e-06 2024-02-05T04:53:49.068 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.068 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.068 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:49.068 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.172415+0000", 2024-02-05T04:53:49.068 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.8010000000000004e-06 2024-02-05T04:53:49.068 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.068 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.068 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:49.068 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.172494+0000", 2024-02-05T04:53:49.069 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.9208999999999993e-05 2024-02-05T04:53:49.069 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.069 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.069 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:49.069 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.172503+0000", 2024-02-05T04:53:49.069 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.1439999999999992e-06 2024-02-05T04:53:49.069 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.069 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.069 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:49.069 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.180452+0000", 2024-02-05T04:53:49.069 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0079488509999999998 2024-02-05T04:53:49.069 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.069 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.069 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:49.069 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.180488+0000", 2024-02-05T04:53:49.070 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.6622000000000003e-05 2024-02-05T04:53:49.070 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.070 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:49.070 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.070 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.070 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.070 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.3s2 43/31 ECSubWrite(tid=63, reqid=osd.4.0:63, at_version=43'27, trim_to=0'0, roll_forward_to=40'25))", 2024-02-05T04:53:49.070 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:35.173633+0000", 2024-02-05T04:53:49.070 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 13.804347288000001, 2024-02-05T04:53:49.070 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0074948189999999998, 2024-02-05T04:53:49.070 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:49.070 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:49.070 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:49.070 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.070 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:49.071 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.173633+0000", 2024-02-05T04:53:49.071 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.071 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.071 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.071 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:49.071 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.173633+0000", 2024-02-05T04:53:49.071 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.071 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.071 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.071 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:49.071 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.173634+0000", 2024-02-05T04:53:49.071 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.2099999999999996e-07 2024-02-05T04:53:49.071 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.071 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.071 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:49.071 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.173648+0000", 2024-02-05T04:53:49.072 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.3957999999999999e-05 2024-02-05T04:53:49.072 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.072 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.072 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:49.072 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.173657+0000", 2024-02-05T04:53:49.072 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.9289999999999993e-06 2024-02-05T04:53:49.072 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.072 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.072 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:49.072 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.173663+0000", 2024-02-05T04:53:49.072 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.2689999999999999e-06 2024-02-05T04:53:49.072 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.072 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.072 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:49.072 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.173709+0000", 2024-02-05T04:53:49.072 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.6238e-05 2024-02-05T04:53:49.073 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.073 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.073 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:49.073 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.173714+0000", 2024-02-05T04:53:49.073 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.4199999999999998e-06 2024-02-05T04:53:49.073 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.073 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.073 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:49.073 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.181098+0000", 2024-02-05T04:53:49.073 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0073840570000000003 2024-02-05T04:53:49.073 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.073 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.073 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:49.073 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.181128+0000", 2024-02-05T04:53:49.073 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.9326999999999999e-05 2024-02-05T04:53:49.074 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.074 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:49.074 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.074 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.074 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.074 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.6s4 43/31 ECSubWrite(tid=141, reqid=osd.13.0:138, at_version=0'0, trim_to=0'0, roll_forward_to=43'24))", 2024-02-05T04:53:49.074 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:35.182634+0000", 2024-02-05T04:53:49.074 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 13.795346464, 2024-02-05T04:53:49.074 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0062931330000000002, 2024-02-05T04:53:49.074 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:49.074 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:49.074 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:49.074 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.074 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:49.074 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.182634+0000", 2024-02-05T04:53:49.074 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.075 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.075 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.075 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:49.075 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.182634+0000", 2024-02-05T04:53:49.075 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.075 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.075 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.075 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:49.075 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.182634+0000", 2024-02-05T04:53:49.075 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.0999999999999998e-07 2024-02-05T04:53:49.075 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.075 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.075 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:49.075 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.182644+0000", 2024-02-05T04:53:49.075 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.5170000000000005e-06 2024-02-05T04:53:49.075 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.076 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.076 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:49.076 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.182656+0000", 2024-02-05T04:53:49.076 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.1623999999999999e-05 2024-02-05T04:53:49.076 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.076 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.076 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:49.076 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.182663+0000", 2024-02-05T04:53:49.076 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.5870000000000004e-06 2024-02-05T04:53:49.076 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.076 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.076 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:49.076 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.182729+0000", 2024-02-05T04:53:49.076 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.5804999999999998e-05 2024-02-05T04:53:49.076 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.077 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.077 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:49.077 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.182742+0000", 2024-02-05T04:53:49.077 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.2612000000000001e-05 2024-02-05T04:53:49.077 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.077 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.077 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:49.077 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.188886+0000", 2024-02-05T04:53:49.077 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0061443640000000002 2024-02-05T04:53:49.077 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.077 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.077 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:49.077 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.188927+0000", 2024-02-05T04:53:49.077 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.1013999999999997e-05 2024-02-05T04:53:49.077 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.077 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:49.077 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.078 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.078 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.078 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.5s5 52/31 ECSubWrite(tid=43, reqid=osd.15.0:42, at_version=0'0, trim_to=0'0, roll_forward_to=52'32))", 2024-02-05T04:53:49.078 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:43.972905+0000", 2024-02-05T04:53:49.078 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 5.0050749750000003, 2024-02-05T04:53:49.078 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0057872820000000004, 2024-02-05T04:53:49.078 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:49.078 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:49.078 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:49.078 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.078 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:49.078 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:43.972905+0000", 2024-02-05T04:53:49.078 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.078 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.078 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.079 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:49.079 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:43.972905+0000", 2024-02-05T04:53:49.079 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.079 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.079 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.079 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:49.079 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:43.972906+0000", 2024-02-05T04:53:49.079 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.0599999999999999e-07 2024-02-05T04:53:49.079 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.079 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.079 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:49.079 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:43.972917+0000", 2024-02-05T04:53:49.079 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.1257000000000001e-05 2024-02-05T04:53:49.079 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.079 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.079 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:49.080 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:43.972930+0000", 2024-02-05T04:53:49.080 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.2241e-05 2024-02-05T04:53:49.080 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.080 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.080 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:49.080 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:43.972938+0000", 2024-02-05T04:53:49.080 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.3729999999999993e-06 2024-02-05T04:53:49.080 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.080 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.080 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:49.080 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:43.973008+0000", 2024-02-05T04:53:49.080 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.0173999999999996e-05 2024-02-05T04:53:49.080 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.080 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.081 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:49.081 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:43.973017+0000", 2024-02-05T04:53:49.081 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.9209999999999998e-06 2024-02-05T04:53:49.081 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.081 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.081 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:49.081 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:43.978655+0000", 2024-02-05T04:53:49.081 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0056377170000000004 2024-02-05T04:53:49.081 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.081 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.081 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:49.081 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:43.978693+0000", 2024-02-05T04:53:49.081 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.7793000000000002e-05 2024-02-05T04:53:49.081 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.081 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:49.081 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.082 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.082 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.082 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:500 3.4s0 3:2c15cb2e:::smithi19639362-6:head [append 0~524288] snapc d=[d,b,9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-02-05T04:53:49.082 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:44.963298+0000", 2024-02-05T04:53:49.082 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 4.0146820380000001, 2024-02-05T04:53:49.082 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.010488153, 2024-02-05T04:53:49.082 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:49.082 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:49.082 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:53:49.082 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:53:49.082 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:49.082 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 500 2024-02-05T04:53:49.082 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.082 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:49.082 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.083 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:49.083 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.963298+0000", 2024-02-05T04:53:49.083 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.083 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.083 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.083 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:49.083 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.963298+0000", 2024-02-05T04:53:49.083 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.083 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.083 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.083 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:49.083 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.963301+0000", 2024-02-05T04:53:49.083 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.6769999999999999e-06 2024-02-05T04:53:49.083 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.083 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.083 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:49.084 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.965096+0000", 2024-02-05T04:53:49.084 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.001794618 2024-02-05T04:53:49.084 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.084 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.084 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:49.084 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.965120+0000", 2024-02-05T04:53:49.084 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.4161000000000001e-05 2024-02-05T04:53:49.084 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.084 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.084 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:49.084 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.965132+0000", 2024-02-05T04:53:49.084 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.2658e-05 2024-02-05T04:53:49.084 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.084 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.084 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:49.084 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.965206+0000", 2024-02-05T04:53:49.085 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.3632000000000002e-05 2024-02-05T04:53:49.085 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.085 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.085 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "op must wait for map", 2024-02-05T04:53:49.085 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.965213+0000", 2024-02-05T04:53:49.085 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.8399999999999997e-06 2024-02-05T04:53:49.085 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.085 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.085 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:49.085 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.967118+0000", 2024-02-05T04:53:49.085 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.001905688 2024-02-05T04:53:49.085 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.085 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.085 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:53:49.085 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.967190+0000", 2024-02-05T04:53:49.085 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.161e-05 2024-02-05T04:53:49.086 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.086 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.086 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:49.086 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.968120+0000", 2024-02-05T04:53:49.086 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00092962600000000002 2024-02-05T04:53:49.086 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.086 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.086 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:49.086 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.973623+0000", 2024-02-05T04:53:49.086 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0055030770000000003 2024-02-05T04:53:49.086 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.086 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.086 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "commit_sent", 2024-02-05T04:53:49.086 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.973717+0000", 2024-02-05T04:53:49.086 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.3845999999999996e-05 2024-02-05T04:53:49.086 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.087 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.087 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:49.087 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.973786+0000", 2024-02-05T04:53:49.087 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.9720000000000003e-05 2024-02-05T04:53:49.087 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.087 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:49.087 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.087 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.087 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.087 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:501 3.4s0 3:2c15cb2e:::smithi19639362-6:head [append 0~753664] snapc d=[d,b,9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-02-05T04:53:49.087 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:44.967519+0000", 2024-02-05T04:53:49.087 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 4.0104614139999999, 2024-02-05T04:53:49.087 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.010834552000000001, 2024-02-05T04:53:49.087 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:49.087 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:49.087 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:53:49.088 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:53:49.088 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:49.088 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 501 2024-02-05T04:53:49.088 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.088 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:49.088 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.088 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:49.088 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.967519+0000", 2024-02-05T04:53:49.088 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.088 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.088 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.088 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:49.088 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.967519+0000", 2024-02-05T04:53:49.088 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.089 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.089 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.089 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:49.089 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.967520+0000", 2024-02-05T04:53:49.089 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.142e-06 2024-02-05T04:53:49.089 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.089 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.089 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:49.089 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.969864+0000", 2024-02-05T04:53:49.089 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0023435349999999999 2024-02-05T04:53:49.089 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.089 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.089 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:49.089 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.969881+0000", 2024-02-05T04:53:49.089 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.7592999999999999e-05 2024-02-05T04:53:49.089 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.090 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.090 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:49.090 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.969890+0000", 2024-02-05T04:53:49.090 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.2609999999999999e-06 2024-02-05T04:53:49.090 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.090 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.090 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:49.090 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.969975+0000", 2024-02-05T04:53:49.090 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.4382999999999995e-05 2024-02-05T04:53:49.090 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.090 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.090 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:53:49.090 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.970100+0000", 2024-02-05T04:53:49.090 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00012555199999999999 2024-02-05T04:53:49.091 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.091 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.091 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:49.091 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.971116+0000", 2024-02-05T04:53:49.091 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.001015665 2024-02-05T04:53:49.091 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.091 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.091 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:49.091 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.977653+0000", 2024-02-05T04:53:49.091 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0065369429999999999 2024-02-05T04:53:49.091 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.091 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.091 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "commit_sent", 2024-02-05T04:53:49.091 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.978281+0000", 2024-02-05T04:53:49.091 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00062775600000000004 2024-02-05T04:53:49.091 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.092 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.092 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:49.092 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.978353+0000", 2024-02-05T04:53:49.092 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.2721999999999999e-05 2024-02-05T04:53:49.092 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.092 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:49.092 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.092 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.092 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.092 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:502 3.4s0 3:2c15cb2e:::smithi19639362-6:head [append 0~409600] snapc d=[d,b,9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-02-05T04:53:49.092 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:44.970001+0000", 2024-02-05T04:53:49.092 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 4.0079787949999996, 2024-02-05T04:53:49.092 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0088644669999999991, 2024-02-05T04:53:49.092 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:49.092 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:49.093 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:53:49.093 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:53:49.093 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:49.093 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 502 2024-02-05T04:53:49.093 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.093 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:49.093 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.093 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:49.093 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.970001+0000", 2024-02-05T04:53:49.093 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.093 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.093 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.093 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:49.093 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.970001+0000", 2024-02-05T04:53:49.093 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.093 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.094 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.094 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:49.094 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.970003+0000", 2024-02-05T04:53:49.094 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.7120000000000001e-06 2024-02-05T04:53:49.094 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.094 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.094 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:49.094 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.971437+0000", 2024-02-05T04:53:49.094 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.001433911 2024-02-05T04:53:49.094 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.094 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.094 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:49.094 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.971460+0000", 2024-02-05T04:53:49.094 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.3394e-05 2024-02-05T04:53:49.094 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.095 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.095 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:49.095 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.971482+0000", 2024-02-05T04:53:49.095 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.1137000000000001e-05 2024-02-05T04:53:49.095 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.095 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.095 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:49.095 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.972029+0000", 2024-02-05T04:53:49.095 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00054736499999999996 2024-02-05T04:53:49.095 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.095 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.095 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:53:49.095 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.972141+0000", 2024-02-05T04:53:49.095 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.000111742 2024-02-05T04:53:49.095 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.095 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.096 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:49.096 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.972917+0000", 2024-02-05T04:53:49.096 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00077584400000000003 2024-02-05T04:53:49.096 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.096 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.096 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:49.096 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.978062+0000", 2024-02-05T04:53:49.096 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0051453000000000002 2024-02-05T04:53:49.096 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.096 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.096 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "commit_sent", 2024-02-05T04:53:49.096 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.978801+0000", 2024-02-05T04:53:49.096 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00073907499999999995 2024-02-05T04:53:49.096 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.096 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.097 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:49.097 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.978866+0000", 2024-02-05T04:53:49.097 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.4987000000000006e-05 2024-02-05T04:53:49.097 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.097 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:49.097 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.097 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.097 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.097 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:503 3.4s0 3:2c15cb2e:::smithi19639362-6:head [setxattr _header (56) in=7b] snapc d=[d,b,9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-02-05T04:53:49.097 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:44.971507+0000", 2024-02-05T04:53:49.097 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 4.0064732909999998, 2024-02-05T04:53:49.097 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0075038589999999999, 2024-02-05T04:53:49.097 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:49.097 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:49.097 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:53:49.097 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:53:49.098 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:49.098 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 503 2024-02-05T04:53:49.098 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.098 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:49.098 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.098 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:49.098 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.971507+0000", 2024-02-05T04:53:49.098 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.098 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.098 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.098 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:49.098 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.971507+0000", 2024-02-05T04:53:49.098 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.098 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.098 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.098 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:49.099 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.971508+0000", 2024-02-05T04:53:49.099 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.229e-06 2024-02-05T04:53:49.099 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.099 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.099 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:49.099 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.971513+0000", 2024-02-05T04:53:49.099 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.6430000000000003e-06 2024-02-05T04:53:49.099 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.099 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.099 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:49.099 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.971523+0000", 2024-02-05T04:53:49.099 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.7589999999999992e-06 2024-02-05T04:53:49.099 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.099 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.099 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:49.100 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.971529+0000", 2024-02-05T04:53:49.100 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.7109999999999998e-06 2024-02-05T04:53:49.100 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.100 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.100 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:49.100 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.973834+0000", 2024-02-05T04:53:49.100 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0023050010000000001 2024-02-05T04:53:49.100 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.100 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.100 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:53:49.100 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.973944+0000", 2024-02-05T04:53:49.100 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.000110202 2024-02-05T04:53:49.100 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.100 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.100 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:49.101 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.974385+0000", 2024-02-05T04:53:49.101 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.000440884 2024-02-05T04:53:49.101 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.101 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.101 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:49.101 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.978066+0000", 2024-02-05T04:53:49.101 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0036805060000000001 2024-02-05T04:53:49.101 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.101 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.101 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "commit_sent", 2024-02-05T04:53:49.101 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.978951+0000", 2024-02-05T04:53:49.101 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.000884756 2024-02-05T04:53:49.101 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.101 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.101 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:49.101 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.979011+0000", 2024-02-05T04:53:49.102 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.0167999999999997e-05 2024-02-05T04:53:49.102 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.102 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:49.102 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.102 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.102 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.102 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:504 3.4s0 3:2c15cb2e:::smithi19639362-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-02-05T04:53:49.102 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:44.971539+0000", 2024-02-05T04:53:49.102 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 4.0064414050000003, 2024-02-05T04:53:49.102 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0087845130000000007, 2024-02-05T04:53:49.102 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:49.102 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "started", 2024-02-05T04:53:49.102 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:53:49.102 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:53:49.102 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:49.103 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 504 2024-02-05T04:53:49.103 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.103 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:49.103 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.103 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:49.103 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.971539+0000", 2024-02-05T04:53:49.103 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.103 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.103 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.103 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:49.103 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.971539+0000", 2024-02-05T04:53:49.103 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.103 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.103 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.103 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:49.103 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.971539+0000", 2024-02-05T04:53:49.104 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.0500000000000003e-07 2024-02-05T04:53:49.104 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.104 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.104 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:49.104 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.971542+0000", 2024-02-05T04:53:49.104 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.6970000000000002e-06 2024-02-05T04:53:49.104 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.104 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.104 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:49.104 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.971549+0000", 2024-02-05T04:53:49.104 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.9890000000000003e-06 2024-02-05T04:53:49.104 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.104 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.104 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:49.104 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.971555+0000", 2024-02-05T04:53:49.104 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.6690000000000003e-06 2024-02-05T04:53:49.105 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.105 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.105 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:49.105 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.974686+0000", 2024-02-05T04:53:49.105 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0031309939999999998 2024-02-05T04:53:49.105 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.105 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.105 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "waiting for rw locks", 2024-02-05T04:53:49.105 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.974749+0000", 2024-02-05T04:53:49.105 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.3167e-05 2024-02-05T04:53:49.105 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.105 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.105 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:49.105 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.979371+0000", 2024-02-05T04:53:49.105 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0046216349999999998 2024-02-05T04:53:49.105 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.106 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.106 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:53:49.106 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.979458+0000", 2024-02-05T04:53:49.106 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.7661e-05 2024-02-05T04:53:49.106 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.106 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.106 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:49.106 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.980323+0000", 2024-02-05T04:53:49.106 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00086509599999999999 2024-02-05T04:53:49.106 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.106 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:49.106 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.106 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.106 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:49.106 INFO:teuthology.orchestra.run.smithi091.stdout:} 2024-02-05T04:53:49.108 INFO:tasks.rados.rados.0.smithi196.stdout:210: finishing write tid 1 to smithi19639362-4 2024-02-05T04:53:49.108 INFO:tasks.rados.rados.0.smithi196.stdout:210: finishing write tid 2 to smithi19639362-4 2024-02-05T04:53:49.108 INFO:tasks.rados.rados.0.smithi196.stdout:210: finishing write tid 3 to smithi19639362-4 2024-02-05T04:53:49.108 INFO:tasks.rados.rados.0.smithi196.stdout:210: finishing write tid 4 to smithi19639362-4 2024-02-05T04:53:49.108 INFO:tasks.rados.rados.0.smithi196.stdout:210: finishing write tid 5 to smithi19639362-4 2024-02-05T04:53:49.108 INFO:tasks.rados.rados.0.smithi196.stdout:210: oid 4 updating version 0 to 16 2024-02-05T04:53:49.108 INFO:tasks.rados.rados.0.smithi196.stdout:210: oid 4 updating version 16 to 18 2024-02-05T04:53:49.108 INFO:tasks.rados.rados.0.smithi196.stdout:210: oid 4 version 18 is already newer than 15 2024-02-05T04:53:49.108 INFO:tasks.rados.rados.0.smithi196.stdout:210: oid 4 version 18 is already newer than 17 2024-02-05T04:53:49.108 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 4 v 18 (ObjNum 105 snap 14 seq_num 105) dirty exists 2024-02-05T04:53:49.108 INFO:tasks.rados.rados.0.smithi196.stdout:210: left oid 4 (ObjNum 105 snap 14 seq_num 105) 2024-02-05T04:53:49.109 INFO:teuthology.orchestra.run.smithi062.stdout:osd.12: {} 2024-02-05T04:53:49.109 INFO:tasks.ceph.osd.3.smithi196.stderr:2024-02-05T04:53:49.107+0000 7fc8817c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34605) UID: 0 2024-02-05T04:53:49.109 INFO:teuthology.orchestra.run.smithi062.stderr:osd.12: osd_enable_op_tracker = '' 2024-02-05T04:53:49.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-02-05T04:53:49.002+0000 7fc42659c640 0 -- 172.21.15.62:0/1041806717 >> [v2:172.21.15.91:6809/3656255795,v1:172.21.15.91:6812/3656255795] conn(0x7fc428219ea0 msgr2=0x7fc42821c290 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-02-05T04:53:49.110 INFO:teuthology.orchestra.run.smithi062.stdout:osd.13: {} 2024-02-05T04:53:49.110 INFO:teuthology.orchestra.run.smithi062.stdout:osd.14: {} 2024-02-05T04:53:49.110 INFO:teuthology.orchestra.run.smithi062.stderr:osd.14: osd_enable_op_tracker = '' 2024-02-05T04:53:49.111 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-02-05T04:53:49.111 INFO:teuthology.orchestra.run.smithi062.stdout:osd.15: {} 2024-02-05T04:53:49.112 INFO:teuthology.orchestra.run.smithi062.stderr:osd.15: osd_enable_op_tracker = '' 2024-02-05T04:53:49.208 INFO:tasks.ceph.osd.8.smithi062.stderr:2024-02-05T04:53:49.207+0000 7f9385bf2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 34693) UID: 0 2024-02-05T04:53:49.257 INFO:teuthology.orchestra.run.smithi091.stdout:{ 2024-02-05T04:53:49.257 INFO:teuthology.orchestra.run.smithi091.stdout: "ops": [], 2024-02-05T04:53:49.257 INFO:teuthology.orchestra.run.smithi091.stdout: "num_ops": 0 2024-02-05T04:53:49.258 INFO:teuthology.orchestra.run.smithi091.stdout:} 2024-02-05T04:53:49.265 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-02-05T04:53:49.309 INFO:tasks.ceph.osd.6.smithi150.stderr:2024-02-05T04:53:49.308+0000 7f6bea9f0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34015) UID: 0 2024-02-05T04:53:49.370 INFO:teuthology.orchestra.run.smithi091.stdout:{ 2024-02-05T04:53:49.371 INFO:teuthology.orchestra.run.smithi091.stdout: "ops": [], 2024-02-05T04:53:49.371 INFO:teuthology.orchestra.run.smithi091.stdout: "complaint_time": 30, 2024-02-05T04:53:49.371 INFO:teuthology.orchestra.run.smithi091.stdout: "num_blocked_ops": 0 2024-02-05T04:53:49.371 INFO:teuthology.orchestra.run.smithi091.stdout:} 2024-02-05T04:53:49.378 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-02-05T04:53:49.410 INFO:tasks.ceph.osd.2.smithi150.stderr:2024-02-05T04:53:49.409+0000 7f7337bf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 34014) UID: 0 2024-02-05T04:53:49.485 INFO:teuthology.orchestra.run.smithi091.stdout:{ 2024-02-05T04:53:49.485 INFO:teuthology.orchestra.run.smithi091.stdout: "size": 20, 2024-02-05T04:53:49.486 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 600, 2024-02-05T04:53:49.486 INFO:teuthology.orchestra.run.smithi091.stdout: "ops": [ 2024-02-05T04:53:49.486 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.486 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.5s2 37/31 ECSubWrite(tid=20, reqid=osd.15.0:19, at_version=0'0, trim_to=0'0, roll_forward_to=37'12))", 2024-02-05T04:53:49.486 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:29.137152+0000", 2024-02-05T04:53:49.486 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 20.346474255, 2024-02-05T04:53:49.486 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0059073499999999996, 2024-02-05T04:53:49.486 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:49.486 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:49.486 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:49.486 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.486 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:49.486 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.137152+0000", 2024-02-05T04:53:49.486 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.487 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.487 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.487 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:49.487 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.137152+0000", 2024-02-05T04:53:49.487 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.487 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.487 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.487 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:49.487 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.137153+0000", 2024-02-05T04:53:49.487 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.8299999999999996e-07 2024-02-05T04:53:49.487 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.487 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.488 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:49.488 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.137166+0000", 2024-02-05T04:53:49.488 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.3200000000000001e-05 2024-02-05T04:53:49.488 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.488 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.488 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:49.488 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.137184+0000", 2024-02-05T04:53:49.488 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.7927999999999998e-05 2024-02-05T04:53:49.488 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.488 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.488 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:49.488 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.137191+0000", 2024-02-05T04:53:49.489 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.559e-06 2024-02-05T04:53:49.489 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.489 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.489 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:49.489 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.137379+0000", 2024-02-05T04:53:49.489 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00018750599999999999 2024-02-05T04:53:49.489 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.489 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.489 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:49.489 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.137384+0000", 2024-02-05T04:53:49.489 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.237e-06 2024-02-05T04:53:49.489 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.489 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.490 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:49.490 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.143018+0000", 2024-02-05T04:53:49.490 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0056338990000000004 2024-02-05T04:53:49.490 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.490 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.490 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:49.490 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.143059+0000", 2024-02-05T04:53:49.490 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.1338000000000003e-05 2024-02-05T04:53:49.490 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.490 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:49.490 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.490 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.490 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.491 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.3s5 37/31 ECSubWrite(tid=31, reqid=client.4634.0:220, at_version=37'14, trim_to=0'0, roll_forward_to=37'10))", 2024-02-05T04:53:49.491 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:29.138016+0000", 2024-02-05T04:53:49.491 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 20.345610046000001, 2024-02-05T04:53:49.491 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.005054846, 2024-02-05T04:53:49.491 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:49.491 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:49.491 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:49.491 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.491 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:49.491 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.138016+0000", 2024-02-05T04:53:49.491 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.491 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.491 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.492 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:49.492 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.138016+0000", 2024-02-05T04:53:49.492 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.492 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.492 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.492 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:49.492 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.138017+0000", 2024-02-05T04:53:49.492 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.0200000000000002e-07 2024-02-05T04:53:49.492 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.492 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.492 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:49.492 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.138032+0000", 2024-02-05T04:53:49.492 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.5006e-05 2024-02-05T04:53:49.493 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.493 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.493 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:49.493 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.138045+0000", 2024-02-05T04:53:49.493 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.3471e-05 2024-02-05T04:53:49.493 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.493 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.493 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:49.493 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.138057+0000", 2024-02-05T04:53:49.493 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.17e-05 2024-02-05T04:53:49.493 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.493 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.493 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:49.494 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.138112+0000", 2024-02-05T04:53:49.494 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.5319999999999999e-05 2024-02-05T04:53:49.494 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.494 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.494 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:49.494 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.138121+0000", 2024-02-05T04:53:49.494 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.1360000000000003e-06 2024-02-05T04:53:49.494 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.494 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.494 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:49.494 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.143042+0000", 2024-02-05T04:53:49.494 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0049209989999999997 2024-02-05T04:53:49.494 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.495 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.495 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:49.495 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.143071+0000", 2024-02-05T04:53:49.495 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.8612e-05 2024-02-05T04:53:49.495 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.495 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:49.495 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.495 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.495 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.495 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.ds6 37/31 ECSubWrite(tid=23, reqid=client.4634.0:225, at_version=37'14, trim_to=0'0, roll_forward_to=34'12))", 2024-02-05T04:53:49.495 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:29.146353+0000", 2024-02-05T04:53:49.495 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 20.337273572000001, 2024-02-05T04:53:49.495 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0052857729999999997, 2024-02-05T04:53:49.496 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:49.496 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:49.496 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:49.496 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.496 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:49.496 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.146353+0000", 2024-02-05T04:53:49.496 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.496 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.496 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.496 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:49.496 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.146353+0000", 2024-02-05T04:53:49.496 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.497 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.497 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.497 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:49.497 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.146353+0000", 2024-02-05T04:53:49.497 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.0899999999999997e-07 2024-02-05T04:53:49.497 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.497 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.497 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:49.497 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.146651+0000", 2024-02-05T04:53:49.497 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00029824799999999998 2024-02-05T04:53:49.497 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.497 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.497 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:49.498 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.146661+0000", 2024-02-05T04:53:49.498 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.9590000000000007e-06 2024-02-05T04:53:49.498 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.498 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.498 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:49.498 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.146668+0000", 2024-02-05T04:53:49.498 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.3319999999999997e-06 2024-02-05T04:53:49.498 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.498 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.498 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:49.498 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.146716+0000", 2024-02-05T04:53:49.498 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.8705999999999998e-05 2024-02-05T04:53:49.498 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.499 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.499 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:49.499 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.146723+0000", 2024-02-05T04:53:49.499 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.0129999999999996e-06 2024-02-05T04:53:49.499 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.499 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.499 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:49.499 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.151591+0000", 2024-02-05T04:53:49.499 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0048681369999999998 2024-02-05T04:53:49.499 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.499 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.499 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:49.499 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.151638+0000", 2024-02-05T04:53:49.500 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.7068999999999999e-05 2024-02-05T04:53:49.500 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.500 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:49.500 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.500 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.500 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.500 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.es6 37/31 ECSubWrite(tid=21, reqid=client.4634.0:246, at_version=37'14, trim_to=0'0, roll_forward_to=34'10))", 2024-02-05T04:53:49.500 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:29.190781+0000", 2024-02-05T04:53:49.500 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 20.292845283999998, 2024-02-05T04:53:49.500 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0047737830000000002, 2024-02-05T04:53:49.500 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:49.500 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:49.500 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:49.501 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.501 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:49.501 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.190781+0000", 2024-02-05T04:53:49.501 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.501 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.501 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.501 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:49.501 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.190781+0000", 2024-02-05T04:53:49.501 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.501 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.501 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.502 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:49.502 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.190782+0000", 2024-02-05T04:53:49.502 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.1699999999999998e-07 2024-02-05T04:53:49.502 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.502 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.502 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:49.502 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.190797+0000", 2024-02-05T04:53:49.502 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.5131e-05 2024-02-05T04:53:49.502 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.502 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.502 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:49.502 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.190807+0000", 2024-02-05T04:53:49.503 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.0360000000000001e-05 2024-02-05T04:53:49.503 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.503 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.503 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:49.503 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.190813+0000", 2024-02-05T04:53:49.503 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.0299999999999999e-06 2024-02-05T04:53:49.503 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.503 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.503 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:49.503 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.190872+0000", 2024-02-05T04:53:49.503 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.8900000000000002e-05 2024-02-05T04:53:49.503 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.503 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.504 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:49.504 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.190883+0000", 2024-02-05T04:53:49.504 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.0834e-05 2024-02-05T04:53:49.504 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.504 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.504 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:49.504 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.195517+0000", 2024-02-05T04:53:49.504 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0046344619999999998 2024-02-05T04:53:49.504 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.504 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.504 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:49.504 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.195555+0000", 2024-02-05T04:53:49.505 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.7449e-05 2024-02-05T04:53:49.505 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.505 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:49.505 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.505 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.505 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.505 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.7s5 37/31 ECSubWrite(tid=85, reqid=client.4634.0:243, at_version=0'0, trim_to=0'0, roll_forward_to=37'6))", 2024-02-05T04:53:49.505 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:29.191229+0000", 2024-02-05T04:53:49.505 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 20.292397661999999, 2024-02-05T04:53:49.505 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0043406060000000003, 2024-02-05T04:53:49.505 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:49.505 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:49.506 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:49.506 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.506 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:49.506 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.191229+0000", 2024-02-05T04:53:49.506 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.506 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.506 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.506 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:49.506 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.191229+0000", 2024-02-05T04:53:49.506 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.506 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.506 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.506 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:49.507 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.191229+0000", 2024-02-05T04:53:49.507 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.9899999999999999e-07 2024-02-05T04:53:49.507 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.507 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.507 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:49.507 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.191240+0000", 2024-02-05T04:53:49.507 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.098e-05 2024-02-05T04:53:49.507 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.507 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.507 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:49.507 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.191258+0000", 2024-02-05T04:53:49.507 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.7847e-05 2024-02-05T04:53:49.507 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.508 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.508 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:49.508 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.191267+0000", 2024-02-05T04:53:49.508 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.0720000000000006e-06 2024-02-05T04:53:49.508 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.508 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.508 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:49.508 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.191340+0000", 2024-02-05T04:53:49.508 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.2276999999999999e-05 2024-02-05T04:53:49.508 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.508 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.508 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:49.508 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.191349+0000", 2024-02-05T04:53:49.509 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.6579999999999997e-06 2024-02-05T04:53:49.509 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.509 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.509 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:49.509 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.195532+0000", 2024-02-05T04:53:49.509 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0041825339999999999 2024-02-05T04:53:49.509 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.509 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.509 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:49.509 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.195569+0000", 2024-02-05T04:53:49.509 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.7438999999999998e-05 2024-02-05T04:53:49.509 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.510 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:49.510 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.510 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.510 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.510 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.es6 37/31 ECSubWrite(tid=22, reqid=client.4634.0:249, at_version=37'16, trim_to=0'0, roll_forward_to=37'12))", 2024-02-05T04:53:49.510 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:29.192133+0000", 2024-02-05T04:53:49.510 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 20.291493405000001, 2024-02-05T04:53:49.510 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0060557249999999996, 2024-02-05T04:53:49.510 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:49.510 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:49.510 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:49.510 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.510 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:49.511 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.192133+0000", 2024-02-05T04:53:49.511 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.511 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.511 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.511 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:49.511 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.192133+0000", 2024-02-05T04:53:49.511 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.511 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.511 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.511 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:49.511 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.192133+0000", 2024-02-05T04:53:49.511 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.6000000000000003e-07 2024-02-05T04:53:49.512 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.512 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.512 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:49.512 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.192150+0000", 2024-02-05T04:53:49.512 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.6765e-05 2024-02-05T04:53:49.512 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.512 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.512 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:49.512 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.192160+0000", 2024-02-05T04:53:49.512 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.0185e-05 2024-02-05T04:53:49.512 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.512 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.512 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:49.513 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.192166+0000", 2024-02-05T04:53:49.513 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.7030000000000003e-06 2024-02-05T04:53:49.513 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.513 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.513 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:49.513 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.192212+0000", 2024-02-05T04:53:49.513 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.5952999999999999e-05 2024-02-05T04:53:49.513 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.513 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.513 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:49.513 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.192218+0000", 2024-02-05T04:53:49.513 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.5790000000000003e-06 2024-02-05T04:53:49.514 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.514 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.514 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:49.514 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.198162+0000", 2024-02-05T04:53:49.514 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0059439580000000001 2024-02-05T04:53:49.514 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.514 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.514 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:49.514 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.198189+0000", 2024-02-05T04:53:49.514 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.6922000000000001e-05 2024-02-05T04:53:49.514 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.514 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:49.514 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.515 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.515 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.515 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.4s2 38/31 ECSubWrite(tid=27, reqid=osd.1.0:27, at_version=38'16, trim_to=0'0, roll_forward_to=37'14))", 2024-02-05T04:53:49.515 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:30.134010+0000", 2024-02-05T04:53:49.515 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 19.349616397999998, 2024-02-05T04:53:49.515 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.004375487, 2024-02-05T04:53:49.515 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:49.515 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:49.515 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:49.515 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.515 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:49.515 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.134010+0000", 2024-02-05T04:53:49.515 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.516 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.516 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.516 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:49.516 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.134010+0000", 2024-02-05T04:53:49.516 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.516 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.516 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.516 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:49.516 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.134010+0000", 2024-02-05T04:53:49.516 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.7500000000000001e-07 2024-02-05T04:53:49.516 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.516 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.517 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:49.517 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.134037+0000", 2024-02-05T04:53:49.517 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.7259000000000001e-05 2024-02-05T04:53:49.517 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.517 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.517 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:49.517 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.134049+0000", 2024-02-05T04:53:49.517 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.1642e-05 2024-02-05T04:53:49.517 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.517 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.517 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:49.517 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.134060+0000", 2024-02-05T04:53:49.518 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.1289e-05 2024-02-05T04:53:49.518 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.518 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.518 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:49.518 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.134124+0000", 2024-02-05T04:53:49.518 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.3734999999999999e-05 2024-02-05T04:53:49.518 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.518 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.518 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:49.518 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.134133+0000", 2024-02-05T04:53:49.518 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.9169999999999995e-06 2024-02-05T04:53:49.518 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.519 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.519 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:49.519 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.138341+0000", 2024-02-05T04:53:49.519 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0042083880000000004 2024-02-05T04:53:49.519 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.519 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.519 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:49.519 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.138385+0000", 2024-02-05T04:53:49.519 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.3982e-05 2024-02-05T04:53:49.519 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.519 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:49.519 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.519 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.520 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.520 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.3s5 38/31 ECSubWrite(tid=39, reqid=osd.4.0:35, at_version=0'0, trim_to=0'0, roll_forward_to=38'16))", 2024-02-05T04:53:49.520 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:30.136126+0000", 2024-02-05T04:53:49.520 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 19.347500410999999, 2024-02-05T04:53:49.520 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.004733073, 2024-02-05T04:53:49.520 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:49.520 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:49.520 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:49.520 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.520 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:49.520 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.136126+0000", 2024-02-05T04:53:49.520 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.520 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.521 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.521 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:49.521 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.136126+0000", 2024-02-05T04:53:49.521 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.521 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.521 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.521 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:49.521 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.136127+0000", 2024-02-05T04:53:49.521 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.1500000000000004e-07 2024-02-05T04:53:49.521 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.521 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.521 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:49.522 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.136136+0000", 2024-02-05T04:53:49.522 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.7410000000000004e-06 2024-02-05T04:53:49.522 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.522 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.522 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:49.522 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.136150+0000", 2024-02-05T04:53:49.522 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.3781000000000001e-05 2024-02-05T04:53:49.522 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.522 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.522 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:49.522 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.136160+0000", 2024-02-05T04:53:49.522 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.9129999999999998e-06 2024-02-05T04:53:49.522 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.523 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.523 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:49.523 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.136203+0000", 2024-02-05T04:53:49.523 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.2571000000000003e-05 2024-02-05T04:53:49.523 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.523 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.523 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:49.523 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.136208+0000", 2024-02-05T04:53:49.523 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.5480000000000001e-06 2024-02-05T04:53:49.523 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.523 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.523 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:49.523 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.140825+0000", 2024-02-05T04:53:49.524 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0046168169999999996 2024-02-05T04:53:49.524 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.524 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.524 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:49.524 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.140859+0000", 2024-02-05T04:53:49.524 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.3986999999999998e-05 2024-02-05T04:53:49.524 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.524 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:49.524 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.524 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.524 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.524 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.cs2 38/31 ECSubWrite(tid=41, reqid=client.4634.0:267, at_version=38'3, trim_to=0'0, roll_forward_to=33'2))", 2024-02-05T04:53:49.525 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:30.181074+0000", 2024-02-05T04:53:49.525 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 19.302552340999998, 2024-02-05T04:53:49.525 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.005888557, 2024-02-05T04:53:49.525 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:49.525 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:49.525 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:49.525 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.525 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:49.525 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.181074+0000", 2024-02-05T04:53:49.525 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.525 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.525 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.526 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:49.526 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.181074+0000", 2024-02-05T04:53:49.526 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.526 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.526 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.526 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:49.526 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.181074+0000", 2024-02-05T04:53:49.526 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.6400000000000002e-07 2024-02-05T04:53:49.526 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.526 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.526 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:49.526 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.181763+0000", 2024-02-05T04:53:49.526 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00068900500000000002 2024-02-05T04:53:49.527 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.527 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.527 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:49.527 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.181773+0000", 2024-02-05T04:53:49.527 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.9680000000000001e-06 2024-02-05T04:53:49.527 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.527 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.527 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:49.527 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.181780+0000", 2024-02-05T04:53:49.527 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.528e-06 2024-02-05T04:53:49.527 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.527 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.528 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:49.528 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.181845+0000", 2024-02-05T04:53:49.528 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.5100999999999999e-05 2024-02-05T04:53:49.528 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.528 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.528 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:49.528 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.181854+0000", 2024-02-05T04:53:49.528 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.6759999999999996e-06 2024-02-05T04:53:49.528 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.528 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.528 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:49.528 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.186903+0000", 2024-02-05T04:53:49.528 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0050493580000000003 2024-02-05T04:53:49.529 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.529 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.529 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:49.529 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.186962+0000", 2024-02-05T04:53:49.529 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.9357000000000001e-05 2024-02-05T04:53:49.529 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.529 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:49.529 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.529 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.529 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.529 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.cs2 38/31 ECSubWrite(tid=42, reqid=client.4634.0:268, at_version=38'4, trim_to=0'0, roll_forward_to=33'2))", 2024-02-05T04:53:49.529 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:30.183587+0000", 2024-02-05T04:53:49.529 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 19.300038829000002, 2024-02-05T04:53:49.530 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0055219010000000001, 2024-02-05T04:53:49.530 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:49.530 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:49.530 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:49.530 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.530 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:49.530 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.183587+0000", 2024-02-05T04:53:49.530 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.530 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.530 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.530 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:49.530 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.183587+0000", 2024-02-05T04:53:49.530 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.531 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.531 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.531 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:49.531 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.183588+0000", 2024-02-05T04:53:49.531 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.4899999999999995e-07 2024-02-05T04:53:49.531 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.531 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.531 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:49.531 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.183765+0000", 2024-02-05T04:53:49.531 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.000177257 2024-02-05T04:53:49.531 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.531 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.531 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:49.532 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.183781+0000", 2024-02-05T04:53:49.532 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.5284000000000001e-05 2024-02-05T04:53:49.532 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.532 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.532 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:49.532 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.183789+0000", 2024-02-05T04:53:49.532 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.524e-06 2024-02-05T04:53:49.532 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.532 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.532 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:49.532 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.183850+0000", 2024-02-05T04:53:49.532 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.0494000000000001e-05 2024-02-05T04:53:49.533 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.533 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.533 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:49.533 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.183859+0000", 2024-02-05T04:53:49.533 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.8749999999999995e-06 2024-02-05T04:53:49.533 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.533 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.533 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:49.533 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.189078+0000", 2024-02-05T04:53:49.533 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0052183480000000003 2024-02-05T04:53:49.533 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.533 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.533 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:49.533 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.189109+0000", 2024-02-05T04:53:49.534 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.1470000000000002e-05 2024-02-05T04:53:49.534 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.534 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:49.534 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.534 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.534 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.534 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.ds6 38/31 ECSubWrite(tid=36, reqid=client.4634.0:276, at_version=0'0, trim_to=0'0, roll_forward_to=38'22))", 2024-02-05T04:53:49.534 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:30.198151+0000", 2024-02-05T04:53:49.534 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 19.28547498, 2024-02-05T04:53:49.534 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0045376519999999997, 2024-02-05T04:53:49.534 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:49.534 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:49.534 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:49.535 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.535 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:49.535 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.198151+0000", 2024-02-05T04:53:49.535 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.535 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.535 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.535 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:49.535 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.198151+0000", 2024-02-05T04:53:49.535 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.535 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.535 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.535 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:49.535 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.198152+0000", 2024-02-05T04:53:49.536 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.8999999999999997e-07 2024-02-05T04:53:49.536 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.536 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.536 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:49.536 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.198163+0000", 2024-02-05T04:53:49.536 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.0969e-05 2024-02-05T04:53:49.536 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.536 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.536 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:49.536 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.198172+0000", 2024-02-05T04:53:49.536 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.516e-06 2024-02-05T04:53:49.536 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.536 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.537 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:49.537 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.198180+0000", 2024-02-05T04:53:49.537 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.3459999999999996e-06 2024-02-05T04:53:49.537 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.537 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.537 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:49.537 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.198240+0000", 2024-02-05T04:53:49.537 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.0052e-05 2024-02-05T04:53:49.537 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.537 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.537 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:49.537 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.198250+0000", 2024-02-05T04:53:49.537 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.0795000000000001e-05 2024-02-05T04:53:49.538 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.538 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.538 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:49.538 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.202647+0000", 2024-02-05T04:53:49.538 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0043961729999999997 2024-02-05T04:53:49.538 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.538 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.538 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:49.538 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.202689+0000", 2024-02-05T04:53:49.538 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.2311000000000002e-05 2024-02-05T04:53:49.538 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.538 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:49.538 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.539 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.539 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.539 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.4s2 38/31 ECSubWrite(tid=30, reqid=client.4634.0:278, at_version=0'0, trim_to=0'0, roll_forward_to=38'17))", 2024-02-05T04:53:49.539 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:30.200115+0000", 2024-02-05T04:53:49.539 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 19.283511600000001, 2024-02-05T04:53:49.539 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0055056649999999999, 2024-02-05T04:53:49.539 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:49.539 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:49.539 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:49.539 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.539 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:49.539 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.200115+0000", 2024-02-05T04:53:49.539 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.540 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.540 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.540 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:49.540 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.200115+0000", 2024-02-05T04:53:49.540 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.540 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.540 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.540 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:49.540 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.200115+0000", 2024-02-05T04:53:49.540 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.6300000000000005e-07 2024-02-05T04:53:49.540 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.540 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.540 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:49.541 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.200125+0000", 2024-02-05T04:53:49.541 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.3060000000000003e-06 2024-02-05T04:53:49.541 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.541 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.541 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:49.541 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.200139+0000", 2024-02-05T04:53:49.541 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.4252e-05 2024-02-05T04:53:49.541 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.541 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.541 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:49.541 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.200148+0000", 2024-02-05T04:53:49.541 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.7050000000000005e-06 2024-02-05T04:53:49.542 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.542 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.542 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:49.542 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.200211+0000", 2024-02-05T04:53:49.542 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.3193000000000003e-05 2024-02-05T04:53:49.542 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.542 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.542 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:49.542 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.200222+0000", 2024-02-05T04:53:49.542 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.1204e-05 2024-02-05T04:53:49.542 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.542 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.542 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:49.543 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.205586+0000", 2024-02-05T04:53:49.543 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.005364382 2024-02-05T04:53:49.543 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.543 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.543 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:49.543 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.205620+0000", 2024-02-05T04:53:49.543 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.396e-05 2024-02-05T04:53:49.543 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.543 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:49.543 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.543 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.543 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.543 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.3s5 43/31 ECSubWrite(tid=63, reqid=osd.4.0:63, at_version=43'27, trim_to=0'0, roll_forward_to=40'25))", 2024-02-05T04:53:49.544 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:35.173301+0000", 2024-02-05T04:53:49.544 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 14.310325216000001, 2024-02-05T04:53:49.544 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.006998181, 2024-02-05T04:53:49.544 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:49.544 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:49.544 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:49.544 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.544 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:49.544 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.173301+0000", 2024-02-05T04:53:49.544 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.544 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.544 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.545 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:49.545 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.173301+0000", 2024-02-05T04:53:49.545 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.545 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.545 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.545 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:49.545 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.173301+0000", 2024-02-05T04:53:49.545 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.9600000000000001e-07 2024-02-05T04:53:49.545 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.545 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.545 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:49.545 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.173316+0000", 2024-02-05T04:53:49.546 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.5274e-05 2024-02-05T04:53:49.546 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.546 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.546 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:49.546 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.173328+0000", 2024-02-05T04:53:49.546 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.1199000000000001e-05 2024-02-05T04:53:49.546 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.546 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.546 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:49.546 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.173336+0000", 2024-02-05T04:53:49.546 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.5779999999999999e-06 2024-02-05T04:53:49.546 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.546 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.547 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:49.547 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.173389+0000", 2024-02-05T04:53:49.547 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.2818999999999997e-05 2024-02-05T04:53:49.547 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.547 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.547 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:49.547 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.173395+0000", 2024-02-05T04:53:49.547 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.0769999999999996e-06 2024-02-05T04:53:49.547 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.547 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.547 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:49.547 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.180250+0000", 2024-02-05T04:53:49.548 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0068552889999999997 2024-02-05T04:53:49.548 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.548 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.548 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:49.548 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.180299+0000", 2024-02-05T04:53:49.548 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.8748999999999997e-05 2024-02-05T04:53:49.548 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.548 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:49.548 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.548 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.548 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.548 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.es6 43/31 ECSubWrite(tid=41, reqid=osd.2.0:40, at_version=0'0, trim_to=0'0, roll_forward_to=43'34))", 2024-02-05T04:53:49.549 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:35.176532+0000", 2024-02-05T04:53:49.549 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 14.307094168999999, 2024-02-05T04:53:49.549 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0048031920000000004, 2024-02-05T04:53:49.549 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:49.549 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:49.549 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:49.549 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.549 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:49.549 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.176532+0000", 2024-02-05T04:53:49.549 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.549 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.549 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.549 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:49.550 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.176532+0000", 2024-02-05T04:53:49.550 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.550 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.550 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.550 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:49.550 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.176533+0000", 2024-02-05T04:53:49.550 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.7299999999999996e-07 2024-02-05T04:53:49.550 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.550 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.550 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:49.550 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.176541+0000", 2024-02-05T04:53:49.550 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.8879999999999997e-06 2024-02-05T04:53:49.551 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.551 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.551 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:49.551 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.176551+0000", 2024-02-05T04:53:49.551 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.025e-05 2024-02-05T04:53:49.551 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.551 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.551 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:49.551 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.176557+0000", 2024-02-05T04:53:49.551 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.8499999999999999e-06 2024-02-05T04:53:49.551 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.551 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.551 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:49.552 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.176608+0000", 2024-02-05T04:53:49.552 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.1629000000000003e-05 2024-02-05T04:53:49.552 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.552 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.552 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:49.552 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.176618+0000", 2024-02-05T04:53:49.552 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.8879999999999992e-06 2024-02-05T04:53:49.552 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.552 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.552 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:49.552 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.180882+0000", 2024-02-05T04:53:49.552 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0042635110000000002 2024-02-05T04:53:49.552 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.553 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.553 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:49.553 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.181335+0000", 2024-02-05T04:53:49.553 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00045360300000000001 2024-02-05T04:53:49.553 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.553 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:49.553 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.553 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.553 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.553 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.7s5 52/31 ECSubWrite(tid=201, reqid=osd.13.0:201, at_version=52'21, trim_to=0'0, roll_forward_to=51'17))", 2024-02-05T04:53:49.553 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:43.954997+0000", 2024-02-05T04:53:49.553 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 5.5286288519999998, 2024-02-05T04:53:49.553 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0048872229999999996, 2024-02-05T04:53:49.554 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:49.554 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:49.554 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:49.554 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.554 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:49.554 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:43.954997+0000", 2024-02-05T04:53:49.554 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.554 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.554 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.554 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:49.554 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:43.954997+0000", 2024-02-05T04:53:49.554 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.554 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.555 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.555 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:49.555 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:43.954998+0000", 2024-02-05T04:53:49.555 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.4099999999999998e-07 2024-02-05T04:53:49.555 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.555 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.555 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:49.555 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:43.955023+0000", 2024-02-05T04:53:49.555 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.4655e-05 2024-02-05T04:53:49.555 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.555 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.555 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:49.556 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:43.955037+0000", 2024-02-05T04:53:49.556 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.4107999999999999e-05 2024-02-05T04:53:49.556 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.556 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.556 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:49.556 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:43.955045+0000", 2024-02-05T04:53:49.556 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.3610000000000001e-06 2024-02-05T04:53:49.556 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.556 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.556 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:49.556 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:43.955098+0000", 2024-02-05T04:53:49.556 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.2533e-05 2024-02-05T04:53:49.557 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.557 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.557 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:49.557 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:43.955106+0000", 2024-02-05T04:53:49.557 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.6589999999999996e-06 2024-02-05T04:53:49.557 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.557 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.557 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:49.557 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:43.959849+0000", 2024-02-05T04:53:49.557 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0047420609999999997 2024-02-05T04:53:49.557 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.557 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.558 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:49.558 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:43.959885+0000", 2024-02-05T04:53:49.558 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.6105000000000001e-05 2024-02-05T04:53:49.558 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.558 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:49.558 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.558 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.558 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.558 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.4s2 53/31 ECSubWrite(tid=81, reqid=client.4634.0:502, at_version=53'56, trim_to=0'0, roll_forward_to=52'52))", 2024-02-05T04:53:49.558 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:44.972956+0000", 2024-02-05T04:53:49.558 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 4.5106706580000004, 2024-02-05T04:53:49.558 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0052956460000000002, 2024-02-05T04:53:49.558 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:49.559 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:49.559 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:49.559 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.559 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:49.559 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.972956+0000", 2024-02-05T04:53:49.559 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.559 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.559 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.559 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:49.559 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.972956+0000", 2024-02-05T04:53:49.559 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.559 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.560 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.560 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:49.560 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.972956+0000", 2024-02-05T04:53:49.560 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.0400000000000005e-07 2024-02-05T04:53:49.560 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.560 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.560 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:49.560 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.973057+0000", 2024-02-05T04:53:49.560 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.000100616 2024-02-05T04:53:49.560 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.560 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.561 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:49.561 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.973070+0000", 2024-02-05T04:53:49.561 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.2927e-05 2024-02-05T04:53:49.561 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.561 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.561 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:49.561 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.973083+0000", 2024-02-05T04:53:49.561 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.293e-05 2024-02-05T04:53:49.561 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.561 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.561 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:49.561 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.973154+0000", 2024-02-05T04:53:49.562 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.0687999999999998e-05 2024-02-05T04:53:49.562 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.562 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.562 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:49.562 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.973164+0000", 2024-02-05T04:53:49.562 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.0746e-05 2024-02-05T04:53:49.562 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.562 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.562 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:49.562 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.978218+0000", 2024-02-05T04:53:49.562 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0050538409999999999 2024-02-05T04:53:49.562 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.562 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.563 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:49.563 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.978251+0000", 2024-02-05T04:53:49.563 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.3093999999999999e-05 2024-02-05T04:53:49.563 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.563 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:49.563 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.563 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.563 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.563 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.7s5 57/31 ECSubWrite(tid=238, reqid=osd.13.0:238, at_version=57'27, trim_to=0'0, roll_forward_to=52'25))", 2024-02-05T04:53:49.563 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:48.946664+0000", 2024-02-05T04:53:49.563 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 0.53696217000000002, 2024-02-05T04:53:49.563 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0057548510000000001, 2024-02-05T04:53:49.564 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:49.564 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:49.564 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:49.564 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.564 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:49.564 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.946664+0000", 2024-02-05T04:53:49.564 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.564 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.564 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.564 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:49.564 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.946664+0000", 2024-02-05T04:53:49.564 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.564 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.565 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.565 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:49.565 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.946664+0000", 2024-02-05T04:53:49.565 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.4299999999999999e-07 2024-02-05T04:53:49.565 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.565 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.565 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:49.565 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.946685+0000", 2024-02-05T04:53:49.565 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.0449999999999999e-05 2024-02-05T04:53:49.565 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.565 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.565 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:49.566 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.946700+0000", 2024-02-05T04:53:49.566 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.5608e-05 2024-02-05T04:53:49.566 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.566 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.566 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:49.566 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.946711+0000", 2024-02-05T04:53:49.566 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.0373999999999999e-05 2024-02-05T04:53:49.566 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.566 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.566 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:49.566 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.946778+0000", 2024-02-05T04:53:49.566 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.6884000000000005e-05 2024-02-05T04:53:49.566 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.567 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.567 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:49.567 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.946786+0000", 2024-02-05T04:53:49.567 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.0690000000000002e-06 2024-02-05T04:53:49.567 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.567 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.567 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:49.567 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.952361+0000", 2024-02-05T04:53:49.567 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0055754480000000002 2024-02-05T04:53:49.567 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.567 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.567 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:49.568 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.952419+0000", 2024-02-05T04:53:49.568 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.7775000000000002e-05 2024-02-05T04:53:49.568 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.568 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:49.568 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.568 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.568 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.568 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.7s5 57/31 ECSubWrite(tid=239, reqid=osd.13.0:239, at_version=57'29, trim_to=0'0, roll_forward_to=52'25))", 2024-02-05T04:53:49.568 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:48.947110+0000", 2024-02-05T04:53:49.568 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 0.53651632599999999, 2024-02-05T04:53:49.568 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0062897489999999999, 2024-02-05T04:53:49.568 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:49.568 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:49.569 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:49.569 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.569 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:49.569 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.947110+0000", 2024-02-05T04:53:49.569 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.569 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.569 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.569 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:49.569 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.947110+0000", 2024-02-05T04:53:49.569 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.569 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.569 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.570 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:49.570 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.947110+0000", 2024-02-05T04:53:49.570 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.6899999999999998e-07 2024-02-05T04:53:49.570 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.570 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.570 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:49.570 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.947130+0000", 2024-02-05T04:53:49.570 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.9894e-05 2024-02-05T04:53:49.570 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.570 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.570 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:49.570 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.947140+0000", 2024-02-05T04:53:49.570 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.4599999999999992e-06 2024-02-05T04:53:49.571 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.571 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.571 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:49.571 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.947147+0000", 2024-02-05T04:53:49.571 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.6629999999999993e-06 2024-02-05T04:53:49.571 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.571 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.571 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:49.571 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.947202+0000", 2024-02-05T04:53:49.571 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.4768999999999997e-05 2024-02-05T04:53:49.571 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.572 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.572 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:49.572 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.947211+0000", 2024-02-05T04:53:49.572 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.9819999999999997e-06 2024-02-05T04:53:49.572 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.572 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.572 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:49.572 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.953367+0000", 2024-02-05T04:53:49.572 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0061559199999999996 2024-02-05T04:53:49.572 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.572 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.572 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:49.572 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.953400+0000", 2024-02-05T04:53:49.573 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.2592000000000002e-05 2024-02-05T04:53:49.573 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.573 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:49.573 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.573 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.573 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.573 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.cs2 57/31 ECSubWrite(tid=90, reqid=osd.11.0:90, at_version=57'13, trim_to=0'0, roll_forward_to=52'11))", 2024-02-05T04:53:49.573 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:48.949235+0000", 2024-02-05T04:53:49.573 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 0.53439169900000005, 2024-02-05T04:53:49.573 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0047863089999999999, 2024-02-05T04:53:49.573 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:49.573 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:49.573 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:49.573 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.574 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:49.574 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.949235+0000", 2024-02-05T04:53:49.574 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.574 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.574 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.574 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:49.574 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.949235+0000", 2024-02-05T04:53:49.574 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.574 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.574 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.574 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:49.574 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.949235+0000", 2024-02-05T04:53:49.574 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.0699999999999997e-07 2024-02-05T04:53:49.575 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.575 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.575 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:49.575 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.949274+0000", 2024-02-05T04:53:49.575 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.8625999999999999e-05 2024-02-05T04:53:49.575 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.575 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.575 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:49.575 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.949286+0000", 2024-02-05T04:53:49.575 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.1895e-05 2024-02-05T04:53:49.575 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.575 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.575 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:49.575 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.949306+0000", 2024-02-05T04:53:49.576 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.9887000000000001e-05 2024-02-05T04:53:49.576 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.576 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.576 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:49.576 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.951204+0000", 2024-02-05T04:53:49.576 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0018986350000000001 2024-02-05T04:53:49.576 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.576 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.576 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:49.576 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.951213+0000", 2024-02-05T04:53:49.576 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.0739999999999999e-06 2024-02-05T04:53:49.576 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.576 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.577 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:49.577 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.953988+0000", 2024-02-05T04:53:49.577 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.002775193 2024-02-05T04:53:49.577 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.577 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.577 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:49.577 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.954021+0000", 2024-02-05T04:53:49.577 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.2391999999999998e-05 2024-02-05T04:53:49.577 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.577 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:49.577 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.577 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.577 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.577 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.cs2 57/31 ECSubWrite(tid=100, reqid=client.4634.0:533, at_version=57'17, trim_to=0'0, roll_forward_to=57'13))", 2024-02-05T04:53:49.578 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:48.981181+0000", 2024-02-05T04:53:49.578 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 0.50244540900000001, 2024-02-05T04:53:49.578 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0044356550000000002, 2024-02-05T04:53:49.578 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:49.578 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:49.578 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:49.578 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.578 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:49.578 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.981181+0000", 2024-02-05T04:53:49.578 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.578 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.578 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.579 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:49.579 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.981181+0000", 2024-02-05T04:53:49.579 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.579 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.579 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.579 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:49.579 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.981182+0000", 2024-02-05T04:53:49.579 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.2399999999999997e-07 2024-02-05T04:53:49.579 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.579 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.579 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:49.579 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.981634+0000", 2024-02-05T04:53:49.579 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00045244700000000003 2024-02-05T04:53:49.579 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.580 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.580 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:49.580 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.981646+0000", 2024-02-05T04:53:49.580 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.1534e-05 2024-02-05T04:53:49.580 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.580 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.580 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:49.580 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.981655+0000", 2024-02-05T04:53:49.580 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.0180000000000007e-06 2024-02-05T04:53:49.580 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.580 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.580 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:49.580 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.981727+0000", 2024-02-05T04:53:49.581 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.2708999999999998e-05 2024-02-05T04:53:49.581 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.581 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.581 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:49.581 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.981738+0000", 2024-02-05T04:53:49.581 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.0511e-05 2024-02-05T04:53:49.581 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.581 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.581 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:49.581 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.985567+0000", 2024-02-05T04:53:49.581 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0038296110000000001 2024-02-05T04:53:49.581 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.581 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.581 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:49.582 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.985616+0000", 2024-02-05T04:53:49.582 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.9101000000000003e-05 2024-02-05T04:53:49.582 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.582 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:49.582 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.582 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.582 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:49.582 INFO:teuthology.orchestra.run.smithi091.stdout:} 2024-02-05T04:53:49.584 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-02-05T04:53:49.585 INFO:tasks.ceph.osd.14.smithi150.stderr:2024-02-05T04:53:49.582+0000 7f5f81083640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 34020) UID: 0 2024-02-05T04:53:49.684 INFO:tasks.ceph.osd.2.smithi150.stderr:2024-02-05T04:53:49.683+0000 7f7337bf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 34014) UID: 0 2024-02-05T04:53:49.690 INFO:teuthology.orchestra.run.smithi091.stdout:{ 2024-02-05T04:53:49.690 INFO:teuthology.orchestra.run.smithi091.stdout: "ops": [], 2024-02-05T04:53:49.690 INFO:teuthology.orchestra.run.smithi091.stdout: "num_ops": 0 2024-02-05T04:53:49.690 INFO:teuthology.orchestra.run.smithi091.stdout:} 2024-02-05T04:53:49.699 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-02-05T04:53:49.785 INFO:tasks.ceph.osd.6.smithi150.stderr:2024-02-05T04:53:49.784+0000 7f6bea9f0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34015) UID: 0 2024-02-05T04:53:49.804 INFO:teuthology.orchestra.run.smithi091.stdout:{ 2024-02-05T04:53:49.804 INFO:teuthology.orchestra.run.smithi091.stdout: "ops": [], 2024-02-05T04:53:49.804 INFO:teuthology.orchestra.run.smithi091.stdout: "complaint_time": 30, 2024-02-05T04:53:49.804 INFO:teuthology.orchestra.run.smithi091.stdout: "num_blocked_ops": 0 2024-02-05T04:53:49.804 INFO:teuthology.orchestra.run.smithi091.stdout:} 2024-02-05T04:53:49.813 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-02-05T04:53:49.886 INFO:tasks.ceph.osd.12.smithi062.stderr:2024-02-05T04:53:49.886+0000 7f5e5847e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 34695) UID: 0 2024-02-05T04:53:49.919 INFO:teuthology.orchestra.run.smithi091.stdout:{ 2024-02-05T04:53:49.920 INFO:teuthology.orchestra.run.smithi091.stdout: "size": 20, 2024-02-05T04:53:49.920 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 600, 2024-02-05T04:53:49.920 INFO:teuthology.orchestra.run.smithi091.stdout: "ops": [ 2024-02-05T04:53:49.920 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.920 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.ds4 37/31 ECSubWrite(tid=23, reqid=client.4634.0:225, at_version=37'14, trim_to=0'0, roll_forward_to=34'12))", 2024-02-05T04:53:49.920 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:29.146542+0000", 2024-02-05T04:53:49.920 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 20.771487594, 2024-02-05T04:53:49.920 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0049434240000000001, 2024-02-05T04:53:49.920 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:49.920 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:49.920 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:49.920 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.921 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:49.921 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.146542+0000", 2024-02-05T04:53:49.921 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.921 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.921 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.921 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:49.921 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.146542+0000", 2024-02-05T04:53:49.921 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.921 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.921 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.921 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:49.921 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.146542+0000", 2024-02-05T04:53:49.921 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.1199999999999999e-07 2024-02-05T04:53:49.921 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.921 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.921 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:49.922 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.147100+0000", 2024-02-05T04:53:49.922 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00055791499999999997 2024-02-05T04:53:49.922 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.922 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.922 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:49.922 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.147111+0000", 2024-02-05T04:53:49.922 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.1038e-05 2024-02-05T04:53:49.922 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.922 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.922 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:49.922 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.147119+0000", 2024-02-05T04:53:49.922 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.3389999999999993e-06 2024-02-05T04:53:49.922 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.922 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.923 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:49.923 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.147173+0000", 2024-02-05T04:53:49.923 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.3823999999999998e-05 2024-02-05T04:53:49.923 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.923 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.923 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:49.923 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.147180+0000", 2024-02-05T04:53:49.923 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.3899999999999998e-06 2024-02-05T04:53:49.923 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.923 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.923 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:49.923 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.151438+0000", 2024-02-05T04:53:49.923 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0042581229999999999 2024-02-05T04:53:49.923 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.923 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.924 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:49.924 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.151485+0000", 2024-02-05T04:53:49.924 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.7482999999999999e-05 2024-02-05T04:53:49.924 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.924 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:49.924 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.924 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.924 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.924 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.ds4 37/31 ECSubWrite(tid=24, reqid=client.4634.0:225, at_version=0'0, trim_to=0'0, roll_forward_to=37'14))", 2024-02-05T04:53:49.924 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:29.152884+0000", 2024-02-05T04:53:49.924 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 20.765144932999998, 2024-02-05T04:53:49.924 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0041159070000000002, 2024-02-05T04:53:49.924 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:49.924 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:49.924 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:49.925 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.925 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:49.925 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.152884+0000", 2024-02-05T04:53:49.925 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.925 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.925 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.925 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:49.925 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.152884+0000", 2024-02-05T04:53:49.925 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.925 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.925 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.925 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:49.925 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.152885+0000", 2024-02-05T04:53:49.925 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.3200000000000004e-07 2024-02-05T04:53:49.925 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.926 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.926 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:49.926 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.152896+0000", 2024-02-05T04:53:49.926 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.0485999999999999e-05 2024-02-05T04:53:49.926 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.926 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.926 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:49.926 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.152910+0000", 2024-02-05T04:53:49.926 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.3862e-05 2024-02-05T04:53:49.926 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.926 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.926 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:49.926 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.152918+0000", 2024-02-05T04:53:49.926 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.0709999999999995e-06 2024-02-05T04:53:49.926 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.926 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.927 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:49.927 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.152983+0000", 2024-02-05T04:53:49.927 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.5445999999999996e-05 2024-02-05T04:53:49.927 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.927 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.927 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:49.927 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.152993+0000", 2024-02-05T04:53:49.927 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.6590000000000002e-06 2024-02-05T04:53:49.927 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.927 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.927 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:49.927 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.156968+0000", 2024-02-05T04:53:49.927 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0039752030000000001 2024-02-05T04:53:49.927 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.927 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.928 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:49.928 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.157000+0000", 2024-02-05T04:53:49.928 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.2348000000000003e-05 2024-02-05T04:53:49.928 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.928 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:49.928 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.928 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.928 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.928 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.4s4 37/31 ECSubWrite(tid=22, reqid=client.4634.0:239, at_version=37'12, trim_to=0'0, roll_forward_to=34'10))", 2024-02-05T04:53:49.928 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:29.157627+0000", 2024-02-05T04:53:49.928 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 20.760402081999999, 2024-02-05T04:53:49.928 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0038875250000000002, 2024-02-05T04:53:49.928 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:49.928 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:49.928 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:49.929 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.929 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:49.929 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.157627+0000", 2024-02-05T04:53:49.929 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.929 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.929 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.929 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:49.929 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.157627+0000", 2024-02-05T04:53:49.929 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.929 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.929 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.929 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:49.929 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.157627+0000", 2024-02-05T04:53:49.929 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.48e-07 2024-02-05T04:53:49.929 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.930 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.930 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:49.930 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.157791+0000", 2024-02-05T04:53:49.930 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00016373700000000001 2024-02-05T04:53:49.930 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.930 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.930 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:49.930 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.157802+0000", 2024-02-05T04:53:49.930 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.0591000000000001e-05 2024-02-05T04:53:49.930 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.930 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.930 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:49.930 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.157810+0000", 2024-02-05T04:53:49.930 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.8730000000000007e-06 2024-02-05T04:53:49.930 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.930 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.931 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:49.931 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.157880+0000", 2024-02-05T04:53:49.931 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.0251000000000002e-05 2024-02-05T04:53:49.931 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.931 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.931 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:49.931 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.157890+0000", 2024-02-05T04:53:49.931 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.0111e-05 2024-02-05T04:53:49.931 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.931 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.931 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:49.931 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.161485+0000", 2024-02-05T04:53:49.931 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0035949609999999998 2024-02-05T04:53:49.932 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.932 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.932 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:49.932 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.161515+0000", 2024-02-05T04:53:49.932 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.9753000000000001e-05 2024-02-05T04:53:49.932 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.932 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:49.932 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.932 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.932 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.932 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.ds4 38/31 ECSubWrite(tid=26, reqid=osd.7.0:26, at_version=38'16, trim_to=0'0, roll_forward_to=37'14))", 2024-02-05T04:53:49.932 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:30.134730+0000", 2024-02-05T04:53:49.932 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 19.783299513999999, 2024-02-05T04:53:49.932 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0033467420000000002, 2024-02-05T04:53:49.933 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:49.933 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:49.933 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:49.933 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.933 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:49.933 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.134730+0000", 2024-02-05T04:53:49.933 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.933 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.933 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.933 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:49.933 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.134730+0000", 2024-02-05T04:53:49.933 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.933 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.933 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.933 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:49.934 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.134730+0000", 2024-02-05T04:53:49.934 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.06e-07 2024-02-05T04:53:49.934 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.934 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.934 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:49.934 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.134743+0000", 2024-02-05T04:53:49.934 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.2712e-05 2024-02-05T04:53:49.934 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.934 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.934 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:49.934 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.134751+0000", 2024-02-05T04:53:49.934 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.2439999999999993e-06 2024-02-05T04:53:49.934 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.934 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.934 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:49.934 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.134756+0000", 2024-02-05T04:53:49.935 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.3630000000000002e-06 2024-02-05T04:53:49.935 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.935 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.935 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:49.935 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.134806+0000", 2024-02-05T04:53:49.935 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.9654999999999998e-05 2024-02-05T04:53:49.935 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.935 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.935 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:49.935 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.134817+0000", 2024-02-05T04:53:49.935 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.1107000000000001e-05 2024-02-05T04:53:49.935 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.935 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.935 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:49.935 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.138044+0000", 2024-02-05T04:53:49.936 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0032267480000000002 2024-02-05T04:53:49.936 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.936 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.936 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:49.936 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.138077+0000", 2024-02-05T04:53:49.936 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.2806999999999999e-05 2024-02-05T04:53:49.936 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.936 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:49.936 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.936 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.936 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.936 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.cs6 38/31 ECSubWrite(tid=42, reqid=client.4634.0:268, at_version=38'4, trim_to=0'0, roll_forward_to=33'2))", 2024-02-05T04:53:49.936 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:30.183371+0000", 2024-02-05T04:53:49.936 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 19.734658670000002, 2024-02-05T04:53:49.936 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0037992030000000001, 2024-02-05T04:53:49.937 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:49.937 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:49.937 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:49.937 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.937 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:49.937 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.183371+0000", 2024-02-05T04:53:49.937 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.937 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.937 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.937 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:49.937 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.183371+0000", 2024-02-05T04:53:49.937 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.937 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.937 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.937 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:49.938 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.183371+0000", 2024-02-05T04:53:49.938 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.13e-07 2024-02-05T04:53:49.938 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.938 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.938 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:49.938 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.183571+0000", 2024-02-05T04:53:49.938 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00019966499999999999 2024-02-05T04:53:49.938 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.938 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.938 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:49.938 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.183581+0000", 2024-02-05T04:53:49.938 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.9690000000000006e-06 2024-02-05T04:53:49.938 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.938 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.938 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:49.938 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.183587+0000", 2024-02-05T04:53:49.939 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.3110000000000002e-06 2024-02-05T04:53:49.939 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.939 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.939 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:49.939 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.183635+0000", 2024-02-05T04:53:49.939 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.7466000000000003e-05 2024-02-05T04:53:49.939 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.939 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.939 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:49.939 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.183640+0000", 2024-02-05T04:53:49.939 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.4229999999999996e-06 2024-02-05T04:53:49.939 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.939 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.939 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:49.940 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.187146+0000", 2024-02-05T04:53:49.940 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.003506239 2024-02-05T04:53:49.940 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.940 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.940 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:49.940 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.187170+0000", 2024-02-05T04:53:49.940 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.3617000000000001e-05 2024-02-05T04:53:49.940 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.940 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:49.940 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.940 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.940 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.940 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.4s4 38/31 ECSubWrite(tid=29, reqid=client.4634.0:278, at_version=38'17, trim_to=0'0, roll_forward_to=38'16))", 2024-02-05T04:53:49.940 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:30.196038+0000", 2024-02-05T04:53:49.940 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 19.721991488, 2024-02-05T04:53:49.940 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0036163229999999998, 2024-02-05T04:53:49.941 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:49.941 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:49.941 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:49.941 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.941 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:49.941 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.196038+0000", 2024-02-05T04:53:49.941 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.941 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.941 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.941 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:49.941 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.196038+0000", 2024-02-05T04:53:49.941 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.942 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.942 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.942 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:49.942 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.196039+0000", 2024-02-05T04:53:49.942 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.9100000000000003e-07 2024-02-05T04:53:49.942 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.942 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.942 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:49.942 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.196068+0000", 2024-02-05T04:53:49.942 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.9391e-05 2024-02-05T04:53:49.942 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.942 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.942 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:49.942 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.196082+0000", 2024-02-05T04:53:49.943 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.4226999999999999e-05 2024-02-05T04:53:49.943 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.943 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.943 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:49.943 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.196090+0000", 2024-02-05T04:53:49.943 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.0909999999999993e-06 2024-02-05T04:53:49.943 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.943 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.943 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:49.943 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.196168+0000", 2024-02-05T04:53:49.943 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.7968999999999998e-05 2024-02-05T04:53:49.943 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.943 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.943 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:49.944 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.196178+0000", 2024-02-05T04:53:49.944 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.3000000000000007e-06 2024-02-05T04:53:49.944 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.944 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.944 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:49.944 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.199621+0000", 2024-02-05T04:53:49.944 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.003443083 2024-02-05T04:53:49.944 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.944 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.944 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:49.944 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.199654+0000", 2024-02-05T04:53:49.944 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.3571000000000001e-05 2024-02-05T04:53:49.944 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.944 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:49.945 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.945 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.945 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.945 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.ds4 38/31 ECSubWrite(tid=36, reqid=client.4634.0:276, at_version=0'0, trim_to=0'0, roll_forward_to=38'22))", 2024-02-05T04:53:49.945 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:30.198156+0000", 2024-02-05T04:53:49.945 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 19.719873139000001, 2024-02-05T04:53:49.945 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0045276530000000004, 2024-02-05T04:53:49.945 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:49.945 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:49.945 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:49.945 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.945 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:49.945 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.198156+0000", 2024-02-05T04:53:49.945 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.945 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.945 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.946 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:49.946 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.198156+0000", 2024-02-05T04:53:49.946 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.946 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.946 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.946 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:49.946 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.198157+0000", 2024-02-05T04:53:49.946 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.9599999999999999e-07 2024-02-05T04:53:49.946 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.946 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.946 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:49.946 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.198164+0000", 2024-02-05T04:53:49.946 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.8519999999999998e-06 2024-02-05T04:53:49.946 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.946 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.947 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:49.947 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.198175+0000", 2024-02-05T04:53:49.947 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.0896999999999999e-05 2024-02-05T04:53:49.947 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.947 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.947 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:49.947 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.198184+0000", 2024-02-05T04:53:49.947 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.1530000000000003e-06 2024-02-05T04:53:49.947 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.947 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.947 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:49.947 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.198249+0000", 2024-02-05T04:53:49.947 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.5270000000000004e-05 2024-02-05T04:53:49.947 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.947 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.948 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:49.948 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.198259+0000", 2024-02-05T04:53:49.948 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.0180000000000001e-05 2024-02-05T04:53:49.948 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.948 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.948 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:49.948 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.202654+0000", 2024-02-05T04:53:49.948 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0043949999999999996 2024-02-05T04:53:49.948 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.948 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.948 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:49.948 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.202684+0000", 2024-02-05T04:53:49.948 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.9705000000000001e-05 2024-02-05T04:53:49.948 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.948 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:49.949 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.949 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.949 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.949 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.4s4 38/31 ECSubWrite(tid=30, reqid=client.4634.0:278, at_version=0'0, trim_to=0'0, roll_forward_to=38'17))", 2024-02-05T04:53:49.949 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:30.200120+0000", 2024-02-05T04:53:49.949 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 19.717909628000001, 2024-02-05T04:53:49.949 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0055354169999999999, 2024-02-05T04:53:49.949 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:49.949 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:49.949 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:49.949 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.949 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:49.949 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.200120+0000", 2024-02-05T04:53:49.949 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.949 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.949 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.950 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:49.950 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.200120+0000", 2024-02-05T04:53:49.950 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.950 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.950 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.950 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:49.950 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.200120+0000", 2024-02-05T04:53:49.950 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.5000000000000003e-07 2024-02-05T04:53:49.950 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.950 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.950 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:49.950 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.200127+0000", 2024-02-05T04:53:49.950 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.1570000000000003e-06 2024-02-05T04:53:49.950 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.950 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.951 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:49.951 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.200139+0000", 2024-02-05T04:53:49.951 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.1484000000000001e-05 2024-02-05T04:53:49.951 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.951 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.951 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:49.951 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.200147+0000", 2024-02-05T04:53:49.951 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.9249999999999995e-06 2024-02-05T04:53:49.951 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.951 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.951 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:49.951 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.200215+0000", 2024-02-05T04:53:49.951 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.8570000000000002e-05 2024-02-05T04:53:49.951 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.951 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.952 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:49.952 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.200224+0000", 2024-02-05T04:53:49.952 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.9670000000000007e-06 2024-02-05T04:53:49.952 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.952 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.952 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:49.952 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.205618+0000", 2024-02-05T04:53:49.952 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.005393994 2024-02-05T04:53:49.952 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.952 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.952 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:49.952 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.205655+0000", 2024-02-05T04:53:49.952 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.6770000000000002e-05 2024-02-05T04:53:49.952 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.952 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:49.953 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.953 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.953 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.953 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.ds4 43/31 ECSubWrite(tid=44, reqid=client.4634.0:401, at_version=43'24, trim_to=0'0, roll_forward_to=38'22))", 2024-02-05T04:53:49.953 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:35.183654+0000", 2024-02-05T04:53:49.953 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 14.734375681, 2024-02-05T04:53:49.953 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0052912829999999999, 2024-02-05T04:53:49.953 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:49.953 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:49.953 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:49.953 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.953 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:49.953 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.183654+0000", 2024-02-05T04:53:49.953 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.953 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.954 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.954 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:49.954 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.183654+0000", 2024-02-05T04:53:49.954 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.954 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.954 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.954 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:49.954 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.183654+0000", 2024-02-05T04:53:49.954 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.4999999999999998e-07 2024-02-05T04:53:49.954 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.954 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.954 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:49.954 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.183674+0000", 2024-02-05T04:53:49.954 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.0027e-05 2024-02-05T04:53:49.954 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.955 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.955 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:49.955 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.183684+0000", 2024-02-05T04:53:49.955 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.5000000000000005e-06 2024-02-05T04:53:49.955 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.955 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.955 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:49.955 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.183691+0000", 2024-02-05T04:53:49.955 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.1629999999999999e-06 2024-02-05T04:53:49.955 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.955 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.955 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:49.955 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.183759+0000", 2024-02-05T04:53:49.955 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.8559999999999994e-05 2024-02-05T04:53:49.955 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.956 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.956 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:49.956 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.183768+0000", 2024-02-05T04:53:49.956 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.0059999999999998e-06 2024-02-05T04:53:49.956 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.956 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.956 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:49.956 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.188906+0000", 2024-02-05T04:53:49.956 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0051378040000000002 2024-02-05T04:53:49.956 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.956 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.956 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:49.956 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.188945+0000", 2024-02-05T04:53:49.956 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.8872999999999999e-05 2024-02-05T04:53:49.956 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.957 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:49.957 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.957 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.957 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.957 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.4s4 53/31 ECSubWrite(tid=80, reqid=client.4634.0:501, at_version=53'55, trim_to=0'0, roll_forward_to=52'52))", 2024-02-05T04:53:49.957 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:44.971367+0000", 2024-02-05T04:53:49.957 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 4.9466623050000003, 2024-02-05T04:53:49.957 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0062083889999999999, 2024-02-05T04:53:49.957 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:49.957 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:49.957 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:49.957 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.957 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:49.957 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.971367+0000", 2024-02-05T04:53:49.957 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.958 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.958 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.958 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:49.958 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.971367+0000", 2024-02-05T04:53:49.958 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.958 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.958 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.958 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:49.958 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.971368+0000", 2024-02-05T04:53:49.958 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.95e-07 2024-02-05T04:53:49.958 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.958 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.958 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:49.958 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.971551+0000", 2024-02-05T04:53:49.958 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00018280999999999999 2024-02-05T04:53:49.958 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.959 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.959 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:49.959 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.971566+0000", 2024-02-05T04:53:49.959 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.5654999999999998e-05 2024-02-05T04:53:49.959 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.959 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.959 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:49.959 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.971577+0000", 2024-02-05T04:53:49.959 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.0268999999999999e-05 2024-02-05T04:53:49.959 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.959 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.959 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:49.959 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.971654+0000", 2024-02-05T04:53:49.959 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.7328000000000002e-05 2024-02-05T04:53:49.959 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.960 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.960 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:49.960 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.971665+0000", 2024-02-05T04:53:49.960 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.0713000000000001e-05 2024-02-05T04:53:49.960 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.960 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.960 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:49.960 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.977541+0000", 2024-02-05T04:53:49.960 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0058759399999999996 2024-02-05T04:53:49.960 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.960 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.960 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:49.960 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.977575+0000", 2024-02-05T04:53:49.960 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.4678999999999998e-05 2024-02-05T04:53:49.961 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.961 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:49.961 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.961 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.961 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.961 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.4s4 53/31 ECSubWrite(tid=81, reqid=client.4634.0:502, at_version=53'56, trim_to=0'0, roll_forward_to=52'52))", 2024-02-05T04:53:49.961 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:44.973033+0000", 2024-02-05T04:53:49.961 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 4.9449961709999997, 2024-02-05T04:53:49.961 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0055856899999999999, 2024-02-05T04:53:49.961 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:49.961 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:49.961 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:49.961 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.961 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:49.961 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.973033+0000", 2024-02-05T04:53:49.961 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.962 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.962 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.962 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:49.962 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.973033+0000", 2024-02-05T04:53:49.962 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.962 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.962 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.962 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:49.962 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.973034+0000", 2024-02-05T04:53:49.962 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.0400000000000005e-07 2024-02-05T04:53:49.962 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.962 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.962 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:49.962 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.973150+0000", 2024-02-05T04:53:49.962 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00011555199999999999 2024-02-05T04:53:49.963 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.963 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.963 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:49.963 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.973174+0000", 2024-02-05T04:53:49.963 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.4099000000000001e-05 2024-02-05T04:53:49.963 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.963 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.963 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:49.963 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.973183+0000", 2024-02-05T04:53:49.963 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.4609999999999997e-06 2024-02-05T04:53:49.963 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.963 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.963 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:49.963 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.973293+0000", 2024-02-05T04:53:49.963 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.000110318 2024-02-05T04:53:49.964 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.964 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.964 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:49.964 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.973307+0000", 2024-02-05T04:53:49.964 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.3613e-05 2024-02-05T04:53:49.964 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.964 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.964 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:49.964 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.978589+0000", 2024-02-05T04:53:49.964 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0052819920000000001 2024-02-05T04:53:49.964 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.964 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.964 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:49.964 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.978619+0000", 2024-02-05T04:53:49.964 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.9751e-05 2024-02-05T04:53:49.964 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.965 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:49.965 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.965 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.965 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.965 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.4s4 53/31 ECSubWrite(tid=82, reqid=client.4634.0:503, at_version=53'57, trim_to=0'0, roll_forward_to=53'54))", 2024-02-05T04:53:49.965 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:44.974421+0000", 2024-02-05T04:53:49.965 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 4.9436086970000002, 2024-02-05T04:53:49.965 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0042189580000000001, 2024-02-05T04:53:49.965 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:49.965 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:49.965 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:49.965 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.965 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:49.965 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.974421+0000", 2024-02-05T04:53:49.965 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.966 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.966 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.966 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:49.966 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.974421+0000", 2024-02-05T04:53:49.966 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.966 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.966 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.966 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:49.966 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.974421+0000", 2024-02-05T04:53:49.966 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.7199999999999998e-07 2024-02-05T04:53:49.966 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.966 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.966 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:49.966 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.974440+0000", 2024-02-05T04:53:49.966 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.8626000000000001e-05 2024-02-05T04:53:49.966 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.967 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.967 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:49.967 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.974469+0000", 2024-02-05T04:53:49.967 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.9323e-05 2024-02-05T04:53:49.967 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.967 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.967 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:49.967 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.974486+0000", 2024-02-05T04:53:49.967 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.6820000000000002e-05 2024-02-05T04:53:49.967 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.967 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.967 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:49.967 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.975268+0000", 2024-02-05T04:53:49.967 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00078173700000000001 2024-02-05T04:53:49.967 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.967 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.968 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:49.968 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.975279+0000", 2024-02-05T04:53:49.968 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.1423999999999999e-05 2024-02-05T04:53:49.968 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.968 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.968 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:49.968 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.978625+0000", 2024-02-05T04:53:49.968 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0033460059999999999 2024-02-05T04:53:49.968 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.968 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.969 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:49.969 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.978640+0000", 2024-02-05T04:53:49.969 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.4250000000000001e-05 2024-02-05T04:53:49.969 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.969 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:49.969 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.969 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.969 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.969 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.9s5 57/31 ECSubWrite(tid=47, reqid=osd.14.0:47, at_version=57'21, trim_to=0'0, roll_forward_to=52'19))", 2024-02-05T04:53:49.969 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:48.946207+0000", 2024-02-05T04:53:49.969 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 0.97182268900000002, 2024-02-05T04:53:49.969 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0044579049999999999, 2024-02-05T04:53:49.969 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:49.969 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:49.969 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:49.969 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.970 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:49.970 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.946207+0000", 2024-02-05T04:53:49.970 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.970 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.970 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.970 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:49.970 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.946207+0000", 2024-02-05T04:53:49.970 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.970 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.970 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.970 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:49.970 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.946207+0000", 2024-02-05T04:53:49.970 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.0800000000000004e-07 2024-02-05T04:53:49.970 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.970 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.970 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:49.971 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.946230+0000", 2024-02-05T04:53:49.971 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.3244e-05 2024-02-05T04:53:49.971 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.971 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.971 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:49.971 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.946243+0000", 2024-02-05T04:53:49.971 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.2174000000000001e-05 2024-02-05T04:53:49.971 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.971 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.971 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:49.971 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.946254+0000", 2024-02-05T04:53:49.971 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.1012000000000001e-05 2024-02-05T04:53:49.971 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.971 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.971 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:49.972 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.946315+0000", 2024-02-05T04:53:49.972 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.1278000000000005e-05 2024-02-05T04:53:49.972 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.972 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.972 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:49.972 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.946327+0000", 2024-02-05T04:53:49.972 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.189e-05 2024-02-05T04:53:49.972 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.972 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.972 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:49.972 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.950627+0000", 2024-02-05T04:53:49.972 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0043000499999999997 2024-02-05T04:53:49.972 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.972 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.972 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:49.973 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.950665+0000", 2024-02-05T04:53:49.973 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.7648999999999998e-05 2024-02-05T04:53:49.973 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.973 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:49.973 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.973 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.973 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.973 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.cs6 57/31 ECSubWrite(tid=90, reqid=osd.11.0:90, at_version=57'13, trim_to=0'0, roll_forward_to=52'11))", 2024-02-05T04:53:49.973 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:48.949341+0000", 2024-02-05T04:53:49.973 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 0.96868879500000005, 2024-02-05T04:53:49.973 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.003600116, 2024-02-05T04:53:49.973 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:49.973 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:49.973 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:49.973 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.973 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:49.974 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.949341+0000", 2024-02-05T04:53:49.974 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.974 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.974 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.974 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:49.974 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.949341+0000", 2024-02-05T04:53:49.974 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.974 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.974 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.974 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:49.974 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.949341+0000", 2024-02-05T04:53:49.974 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.31e-07 2024-02-05T04:53:49.974 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.974 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.974 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:49.974 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.949735+0000", 2024-02-05T04:53:49.975 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00039458899999999999 2024-02-05T04:53:49.975 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.975 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.975 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:49.975 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.949745+0000", 2024-02-05T04:53:49.975 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.8349999999999999e-06 2024-02-05T04:53:49.975 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.975 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.975 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:49.975 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.949753+0000", 2024-02-05T04:53:49.975 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.4290000000000004e-06 2024-02-05T04:53:49.975 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.975 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.975 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:49.975 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.949808+0000", 2024-02-05T04:53:49.976 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.5973000000000001e-05 2024-02-05T04:53:49.976 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.976 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.976 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:49.976 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.949818+0000", 2024-02-05T04:53:49.976 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.3470000000000005e-06 2024-02-05T04:53:49.976 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.976 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.976 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:49.976 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.952895+0000", 2024-02-05T04:53:49.976 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0030774919999999998 2024-02-05T04:53:49.976 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.976 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.976 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:49.976 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.952941+0000", 2024-02-05T04:53:49.977 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.532e-05 2024-02-05T04:53:49.977 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.977 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:49.977 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.977 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.977 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.977 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.cs6 57/31 ECSubWrite(tid=94, reqid=osd.11.0:90, at_version=0'0, trim_to=0'0, roll_forward_to=57'13))", 2024-02-05T04:53:49.977 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:48.958470+0000", 2024-02-05T04:53:49.977 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 0.95955971600000001, 2024-02-05T04:53:49.977 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00581906, 2024-02-05T04:53:49.977 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:49.977 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:49.977 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:49.977 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.977 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:49.978 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.958470+0000", 2024-02-05T04:53:49.978 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.978 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.978 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.978 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:49.978 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.958470+0000", 2024-02-05T04:53:49.978 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.978 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.978 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.978 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:49.978 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.958475+0000", 2024-02-05T04:53:49.978 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.1320000000000002e-06 2024-02-05T04:53:49.978 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.978 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.978 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:49.978 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.958483+0000", 2024-02-05T04:53:49.979 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.123e-06 2024-02-05T04:53:49.979 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.979 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.979 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:49.979 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.958492+0000", 2024-02-05T04:53:49.979 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.5000000000000005e-06 2024-02-05T04:53:49.979 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.979 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.979 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:49.979 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.958500+0000", 2024-02-05T04:53:49.979 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.515e-06 2024-02-05T04:53:49.979 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.979 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.979 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:49.979 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.958568+0000", 2024-02-05T04:53:49.979 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.7806999999999996e-05 2024-02-05T04:53:49.980 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.980 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.980 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:49.980 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.958577+0000", 2024-02-05T04:53:49.980 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.3880000000000006e-06 2024-02-05T04:53:49.980 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.980 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.980 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:49.980 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.964250+0000", 2024-02-05T04:53:49.980 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0056730360000000002 2024-02-05T04:53:49.980 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.980 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.980 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:49.980 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.964289+0000", 2024-02-05T04:53:49.980 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.8559000000000001e-05 2024-02-05T04:53:49.981 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.981 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:49.981 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.981 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.981 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.981 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.ds4 57/31 ECSubWrite(tid=54, reqid=client.4634.0:525, at_version=57'34, trim_to=0'0, roll_forward_to=57'32))", 2024-02-05T04:53:49.981 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:48.959440+0000", 2024-02-05T04:53:49.981 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 0.95858951800000003, 2024-02-05T04:53:49.981 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0048957699999999998, 2024-02-05T04:53:49.981 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:49.981 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:49.981 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:49.981 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.981 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:49.981 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.959440+0000", 2024-02-05T04:53:49.981 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.982 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.982 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.982 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:49.982 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.959440+0000", 2024-02-05T04:53:49.982 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.982 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.982 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.982 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:49.982 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.959440+0000", 2024-02-05T04:53:49.982 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.1500000000000004e-07 2024-02-05T04:53:49.982 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.982 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.982 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:49.982 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.959460+0000", 2024-02-05T04:53:49.982 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.9531999999999999e-05 2024-02-05T04:53:49.983 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.983 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.983 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:49.983 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.959480+0000", 2024-02-05T04:53:49.983 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.0401999999999999e-05 2024-02-05T04:53:49.983 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.983 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.983 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:49.983 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.959489+0000", 2024-02-05T04:53:49.983 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.2889999999999998e-06 2024-02-05T04:53:49.983 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.983 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.983 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:49.983 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.959562+0000", 2024-02-05T04:53:49.983 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.3528000000000005e-05 2024-02-05T04:53:49.983 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.984 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.984 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:49.984 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.959571+0000", 2024-02-05T04:53:49.984 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.6850000000000007e-06 2024-02-05T04:53:49.984 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.984 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.984 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:49.984 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.964303+0000", 2024-02-05T04:53:49.984 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0047316709999999998 2024-02-05T04:53:49.984 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.984 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.985 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:49.985 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.964336+0000", 2024-02-05T04:53:49.985 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.3047999999999999e-05 2024-02-05T04:53:49.985 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.985 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:49.985 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.985 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.985 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.985 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.cs6 57/31 ECSubWrite(tid=98, reqid=client.4634.0:531, at_version=57'15, trim_to=0'0, roll_forward_to=57'13))", 2024-02-05T04:53:49.985 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:48.974750+0000", 2024-02-05T04:53:49.985 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 0.94327905499999998, 2024-02-05T04:53:49.985 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.010037023000000001, 2024-02-05T04:53:49.985 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:49.985 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:49.985 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:49.985 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.986 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:49.986 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.974750+0000", 2024-02-05T04:53:49.986 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.986 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.986 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.986 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:49.986 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.974750+0000", 2024-02-05T04:53:49.986 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.986 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.986 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.986 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:49.986 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.974751+0000", 2024-02-05T04:53:49.986 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.0500000000000003e-07 2024-02-05T04:53:49.986 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.986 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.987 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:49.987 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.975454+0000", 2024-02-05T04:53:49.987 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00070269799999999997 2024-02-05T04:53:49.987 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.987 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.987 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:49.987 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.975467+0000", 2024-02-05T04:53:49.987 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.3737000000000001e-05 2024-02-05T04:53:49.987 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.987 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.987 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:49.987 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.975482+0000", 2024-02-05T04:53:49.987 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.4531e-05 2024-02-05T04:53:49.987 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.987 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.987 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:49.988 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.975585+0000", 2024-02-05T04:53:49.988 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00010341 2024-02-05T04:53:49.988 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.988 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.988 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:49.988 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.975596+0000", 2024-02-05T04:53:49.988 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.0621999999999999e-05 2024-02-05T04:53:49.988 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.988 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.988 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:49.988 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.984752+0000", 2024-02-05T04:53:49.988 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0091561630000000001 2024-02-05T04:53:49.988 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.988 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.988 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:49.989 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.984787+0000", 2024-02-05T04:53:49.989 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.5256999999999998e-05 2024-02-05T04:53:49.989 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.989 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:49.989 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.989 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.989 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.989 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.cs6 57/31 ECSubWrite(tid=99, reqid=client.4634.0:532, at_version=57'16, trim_to=0'0, roll_forward_to=57'13))", 2024-02-05T04:53:49.989 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:48.978847+0000", 2024-02-05T04:53:49.989 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 0.93918214300000002, 2024-02-05T04:53:49.989 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0068457910000000004, 2024-02-05T04:53:49.989 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:49.989 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:49.989 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:49.989 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.989 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:49.990 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.978847+0000", 2024-02-05T04:53:49.990 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.990 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.990 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.990 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:49.990 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.978847+0000", 2024-02-05T04:53:49.990 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.990 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.990 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.990 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:49.991 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.978848+0000", 2024-02-05T04:53:49.991 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.4399999999999999e-07 2024-02-05T04:53:49.991 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.991 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.991 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:49.991 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.979286+0000", 2024-02-05T04:53:49.991 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00043832000000000002 2024-02-05T04:53:49.991 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.991 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.991 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:49.991 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.979305+0000", 2024-02-05T04:53:49.991 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.9196999999999999e-05 2024-02-05T04:53:49.991 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.991 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.992 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:49.992 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.979314+0000", 2024-02-05T04:53:49.992 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.8929999999999994e-06 2024-02-05T04:53:49.992 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.992 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.992 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:49.992 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.979368+0000", 2024-02-05T04:53:49.992 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.3378000000000003e-05 2024-02-05T04:53:49.992 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.992 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.992 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:49.992 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.979376+0000", 2024-02-05T04:53:49.992 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.5180000000000004e-06 2024-02-05T04:53:49.992 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.992 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.993 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:49.993 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.985661+0000", 2024-02-05T04:53:49.993 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0062852289999999998 2024-02-05T04:53:49.993 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.993 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.993 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:49.993 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.985693+0000", 2024-02-05T04:53:49.993 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.1511999999999999e-05 2024-02-05T04:53:49.993 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.993 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:49.993 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.993 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.993 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.993 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.cs6 57/31 ECSubWrite(tid=100, reqid=client.4634.0:533, at_version=57'17, trim_to=0'0, roll_forward_to=57'13))", 2024-02-05T04:53:49.993 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:48.981395+0000", 2024-02-05T04:53:49.994 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 0.93663418499999995, 2024-02-05T04:53:49.994 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0046430849999999999, 2024-02-05T04:53:49.994 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:49.994 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:49.994 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:49.994 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.994 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:49.994 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.981395+0000", 2024-02-05T04:53:49.994 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.994 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.994 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.994 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:49.994 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.981395+0000", 2024-02-05T04:53:49.994 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.994 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.995 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.995 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:49.995 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.981396+0000", 2024-02-05T04:53:49.995 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.9800000000000003e-07 2024-02-05T04:53:49.995 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.995 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.995 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:49.995 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.981770+0000", 2024-02-05T04:53:49.995 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.000373715 2024-02-05T04:53:49.995 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.995 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.995 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:49.995 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.981786+0000", 2024-02-05T04:53:49.995 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.5954999999999999e-05 2024-02-05T04:53:49.995 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.995 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.996 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:49.996 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.981794+0000", 2024-02-05T04:53:49.996 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.1629999999999996e-06 2024-02-05T04:53:49.996 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.996 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.996 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:49.996 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.982196+0000", 2024-02-05T04:53:49.996 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00040278900000000002 2024-02-05T04:53:49.996 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.996 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.996 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:49.996 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.982207+0000", 2024-02-05T04:53:49.996 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.0288999999999999e-05 2024-02-05T04:53:49.996 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.996 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.997 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:49.997 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.986015+0000", 2024-02-05T04:53:49.997 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.003807773 2024-02-05T04:53:49.997 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.997 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.997 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:49.997 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.986038+0000", 2024-02-05T04:53:49.998 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.3703e-05 2024-02-05T04:53:49.998 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.998 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:49.998 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:49.998 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.998 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.998 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.cs6 57/31 ECSubWrite(tid=101, reqid=client.4634.0:534, at_version=57'18, trim_to=0'0, roll_forward_to=57'13))", 2024-02-05T04:53:49.998 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:48.982192+0000", 2024-02-05T04:53:49.998 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 0.93583719600000004, 2024-02-05T04:53:49.999 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0038834870000000001, 2024-02-05T04:53:49.999 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:49.999 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:49.999 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:49.999 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.999 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:49.999 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.982192+0000", 2024-02-05T04:53:49.999 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:49.999 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:49.999 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:49.999 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:49.999 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.982192+0000", 2024-02-05T04:53:50.000 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:50.000 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.000 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.000 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:50.000 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.982193+0000", 2024-02-05T04:53:50.000 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.4300000000000003e-07 2024-02-05T04:53:50.000 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.000 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.000 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:50.000 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.982227+0000", 2024-02-05T04:53:50.000 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.3847999999999998e-05 2024-02-05T04:53:50.001 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.001 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.001 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:50.001 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.982237+0000", 2024-02-05T04:53:50.001 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.0679999999999999e-05 2024-02-05T04:53:50.001 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.001 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.001 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:50.001 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.982247+0000", 2024-02-05T04:53:50.001 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.5759999999999995e-06 2024-02-05T04:53:50.001 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.001 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.001 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:50.001 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.982613+0000", 2024-02-05T04:53:50.001 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00036610599999999999 2024-02-05T04:53:50.001 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.002 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.002 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:50.002 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.982623+0000", 2024-02-05T04:53:50.002 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.6690000000000001e-06 2024-02-05T04:53:50.002 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.002 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.002 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:50.002 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.986063+0000", 2024-02-05T04:53:50.002 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0034401660000000001 2024-02-05T04:53:50.002 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.002 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.002 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:50.002 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.986076+0000", 2024-02-05T04:53:50.002 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.2799000000000001e-05 2024-02-05T04:53:50.002 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:50.003 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:50.003 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:50.003 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:50.003 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:50.003 INFO:teuthology.orchestra.run.smithi091.stdout:} 2024-02-05T04:53:50.003 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-02-05T04:53:50.006 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-02-05T04:53:50.006 INFO:tasks.ceph.osd.13.smithi091.stderr:2024-02-05T04:53:49.960+0000 7f5254320640 -1 snap_mapper.get_next_objects_to_trim::New Clone-Objects were added to Snap 9 after trimming was started 2024-02-05T04:53:50.007 INFO:tasks.rados.rados.0.smithi196.stdout:206: done (6 left) 2024-02-05T04:53:50.007 INFO:tasks.rados.rados.0.smithi196.stdout:210: done (5 left) 2024-02-05T04:53:50.007 INFO:tasks.rados.rados.0.smithi196.stdout:211: done (4 left) 2024-02-05T04:53:50.007 INFO:tasks.rados.rados.0.smithi196.stdout:212: snap_create 2024-02-05T04:53:50.007 INFO:tasks.rados.rados.0.smithi196.stdout:205: finishing copy_from racing read to smithi19639362-28 2024-02-05T04:53:50.007 INFO:tasks.rados.rados.0.smithi196.stdout:209: expect (ObjNum 23 snap 0 seq_num 23) 2024-02-05T04:53:50.007 INFO:tasks.rados.rados.0.smithi196.stdout:205: finishing copy_from to smithi19639362-28 2024-02-05T04:53:50.007 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 28 v 20 (ObjNum 60 snap 2 seq_num 60) dirty exists 2024-02-05T04:53:50.007 INFO:tasks.rados.rados.0.smithi196.stdout:208: finishing copy_from racing read to smithi19639362-40 2024-02-05T04:53:50.007 INFO:tasks.rados.rados.0.smithi196.stdout:204: finishing copy_from racing read to smithi19639362-36 2024-02-05T04:53:50.007 INFO:tasks.rados.rados.0.smithi196.stdout:208: finishing copy_from to smithi19639362-40 2024-02-05T04:53:50.007 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 40 v 29 (ObjNum 73 snap 3 seq_num 73) dirty exists 2024-02-05T04:53:50.007 INFO:tasks.rados.rados.0.smithi196.stdout:204: finishing copy_from to smithi19639362-36 2024-02-05T04:53:50.008 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 36 v 44 (ObjNum 94 snap 11 seq_num 94) dirty exists 2024-02-05T04:53:50.008 INFO:tasks.ceph.osd.9.smithi091.stderr:2024-02-05T04:53:50.004+0000 7efc18827640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 34279) UID: 0 2024-02-05T04:53:50.105 INFO:tasks.ceph.osd.9.smithi091.stderr:2024-02-05T04:53:50.104+0000 7efc18827640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 34279) UID: 0 2024-02-05T04:53:50.153 INFO:teuthology.orchestra.run.smithi091.stdout:{ 2024-02-05T04:53:50.153 INFO:teuthology.orchestra.run.smithi091.stdout: "ops": [], 2024-02-05T04:53:50.153 INFO:teuthology.orchestra.run.smithi091.stdout: "num_ops": 0 2024-02-05T04:53:50.153 INFO:teuthology.orchestra.run.smithi091.stdout:} 2024-02-05T04:53:50.161 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-02-05T04:53:50.206 INFO:tasks.ceph.osd.14.smithi150.stderr:2024-02-05T04:53:50.205+0000 7f5f81083640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 34020) UID: 0 2024-02-05T04:53:50.307 INFO:teuthology.orchestra.run.smithi091.stdout:{ 2024-02-05T04:53:50.307 INFO:teuthology.orchestra.run.smithi091.stdout: "ops": [], 2024-02-05T04:53:50.307 INFO:teuthology.orchestra.run.smithi091.stdout: "complaint_time": 30, 2024-02-05T04:53:50.307 INFO:teuthology.orchestra.run.smithi091.stdout: "num_blocked_ops": 0 2024-02-05T04:53:50.307 INFO:teuthology.orchestra.run.smithi091.stdout:} 2024-02-05T04:53:50.308 INFO:tasks.ceph.osd.6.smithi150.stderr:2024-02-05T04:53:50.306+0000 7f6bea9f0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34015) UID: 0 2024-02-05T04:53:50.315 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-02-05T04:53:50.408 INFO:tasks.ceph.osd.5.smithi091.stderr:2024-02-05T04:53:50.406+0000 7f63983b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34277) UID: 0 2024-02-05T04:53:50.422 INFO:teuthology.orchestra.run.smithi091.stdout:{ 2024-02-05T04:53:50.422 INFO:teuthology.orchestra.run.smithi091.stdout: "size": 20, 2024-02-05T04:53:50.422 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 600, 2024-02-05T04:53:50.422 INFO:teuthology.orchestra.run.smithi091.stdout: "ops": [ 2024-02-05T04:53:50.422 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.422 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:228 3.6s0 3:6dc5620a:::smithi19639362-11:head [rollback 3] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:53:50.423 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:29.147258+0000", 2024-02-05T04:53:50.423 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 21.273061043999999, 2024-02-05T04:53:50.423 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0097154210000000001, 2024-02-05T04:53:50.423 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:50.423 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:50.423 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:53:50.423 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:53:50.423 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:50.423 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 228 2024-02-05T04:53:50.423 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.423 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:50.423 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.424 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:50.424 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.147258+0000", 2024-02-05T04:53:50.424 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:50.424 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.424 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.424 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:50.424 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.147258+0000", 2024-02-05T04:53:50.424 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:50.424 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.424 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.424 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:50.425 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.147258+0000", 2024-02-05T04:53:50.425 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.3000000000000001e-07 2024-02-05T04:53:50.425 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.425 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.425 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:50.425 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.147261+0000", 2024-02-05T04:53:50.425 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.429e-06 2024-02-05T04:53:50.425 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.425 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.425 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:50.425 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.147268+0000", 2024-02-05T04:53:50.425 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.7109999999999995e-06 2024-02-05T04:53:50.426 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.426 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.426 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:50.426 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.147275+0000", 2024-02-05T04:53:50.426 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.2140000000000001e-06 2024-02-05T04:53:50.426 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.426 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.426 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:50.426 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.148902+0000", 2024-02-05T04:53:50.426 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0016272630000000001 2024-02-05T04:53:50.426 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.426 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.426 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:53:50.427 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.148962+0000", 2024-02-05T04:53:50.427 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.0158999999999998e-05 2024-02-05T04:53:50.427 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.427 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.427 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:50.427 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.149301+0000", 2024-02-05T04:53:50.427 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.000339227 2024-02-05T04:53:50.427 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.427 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.427 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:50.427 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.156821+0000", 2024-02-05T04:53:50.427 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0075194629999999997 2024-02-05T04:53:50.427 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.428 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.428 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "commit_sent", 2024-02-05T04:53:50.428 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.156887+0000", 2024-02-05T04:53:50.428 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.6748000000000005e-05 2024-02-05T04:53:50.428 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.428 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.428 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:50.428 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.156973+0000", 2024-02-05T04:53:50.428 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.5677e-05 2024-02-05T04:53:50.428 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:50.428 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:50.428 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:50.429 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.429 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.429 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:229 3.6s0 3:6dc5620a:::smithi19639362-11:head [append 0~0] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:53:50.429 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:29.147292+0000", 2024-02-05T04:53:50.429 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 21.273026527999999, 2024-02-05T04:53:50.429 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0099616229999999993, 2024-02-05T04:53:50.429 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:50.429 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:50.429 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:53:50.429 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:53:50.429 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:50.429 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 229 2024-02-05T04:53:50.429 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.430 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:50.430 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.430 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:50.430 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.147292+0000", 2024-02-05T04:53:50.430 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:50.430 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.430 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.430 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:50.430 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.147292+0000", 2024-02-05T04:53:50.430 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:50.430 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.430 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.430 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:50.430 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.147293+0000", 2024-02-05T04:53:50.431 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.1200000000000003e-07 2024-02-05T04:53:50.431 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.431 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.431 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:50.431 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.147295+0000", 2024-02-05T04:53:50.431 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.3309999999999998e-06 2024-02-05T04:53:50.431 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.431 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.431 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:50.431 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.147302+0000", 2024-02-05T04:53:50.431 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.3769999999999998e-06 2024-02-05T04:53:50.431 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.431 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.432 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:50.432 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.147308+0000", 2024-02-05T04:53:50.432 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.8499999999999999e-06 2024-02-05T04:53:50.432 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.432 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.432 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:50.432 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.151615+0000", 2024-02-05T04:53:50.432 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.004306374 2024-02-05T04:53:50.432 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.432 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.432 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:53:50.432 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.151698+0000", 2024-02-05T04:53:50.432 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.3741999999999999e-05 2024-02-05T04:53:50.433 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.433 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.433 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:50.433 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.152107+0000", 2024-02-05T04:53:50.433 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.000408265 2024-02-05T04:53:50.433 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.433 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.433 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:50.433 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.156975+0000", 2024-02-05T04:53:50.433 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0048685459999999996 2024-02-05T04:53:50.433 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.433 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.434 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "commit_sent", 2024-02-05T04:53:50.434 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.157198+0000", 2024-02-05T04:53:50.434 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00022250500000000001 2024-02-05T04:53:50.434 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.434 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.434 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:50.434 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.157254+0000", 2024-02-05T04:53:50.434 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.6121e-05 2024-02-05T04:53:50.434 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:50.434 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:50.434 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:50.434 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.434 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.435 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:247 3.1s0 3:8451ec0c:::smithi19639362-8:head [copy-from ver 6] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:53:50.435 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:29.186841+0000", 2024-02-05T04:53:50.435 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 21.233477821000001, 2024-02-05T04:53:50.435 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.011720046, 2024-02-05T04:53:50.435 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:50.435 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:50.435 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:53:50.435 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:53:50.435 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:50.435 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 247 2024-02-05T04:53:50.435 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.435 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:50.435 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.436 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:50.436 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.186841+0000", 2024-02-05T04:53:50.436 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:50.436 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.436 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.436 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:50.436 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.186841+0000", 2024-02-05T04:53:50.436 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:50.436 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.436 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.436 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:50.436 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.186841+0000", 2024-02-05T04:53:50.437 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.0900000000000002e-07 2024-02-05T04:53:50.437 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.437 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.437 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:50.437 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.186844+0000", 2024-02-05T04:53:50.437 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.0970000000000002e-06 2024-02-05T04:53:50.437 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.437 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.437 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:50.437 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.186849+0000", 2024-02-05T04:53:50.437 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.9150000000000004e-06 2024-02-05T04:53:50.437 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.437 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.438 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:50.438 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.186853+0000", 2024-02-05T04:53:50.438 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.41e-06 2024-02-05T04:53:50.438 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.438 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.438 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:50.438 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.186906+0000", 2024-02-05T04:53:50.438 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.3505999999999999e-05 2024-02-05T04:53:50.438 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.438 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.438 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:53:50.438 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.187011+0000", 2024-02-05T04:53:50.439 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.000104478 2024-02-05T04:53:50.439 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.439 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.439 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:50.439 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.192835+0000", 2024-02-05T04:53:50.439 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0058247689999999996 2024-02-05T04:53:50.439 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.439 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.439 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:50.439 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.198154+0000", 2024-02-05T04:53:50.439 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0053189470000000001 2024-02-05T04:53:50.439 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.440 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.440 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "commit_sent", 2024-02-05T04:53:50.440 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.198508+0000", 2024-02-05T04:53:50.440 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.000353333 2024-02-05T04:53:50.440 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.440 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.440 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:50.440 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.198561+0000", 2024-02-05T04:53:50.440 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.3182e-05 2024-02-05T04:53:50.440 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:50.440 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:50.440 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:50.441 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.441 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.441 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:248 3.1s0 3:8451ec0c:::smithi19639362-8:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:53:50.441 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:29.186861+0000", 2024-02-05T04:53:50.441 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 21.233457264999998, 2024-02-05T04:53:50.441 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0075015220000000001, 2024-02-05T04:53:50.441 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:50.441 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "started", 2024-02-05T04:53:50.441 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:53:50.441 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:53:50.441 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:50.441 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 248 2024-02-05T04:53:50.441 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.442 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:50.442 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.442 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:50.442 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.186861+0000", 2024-02-05T04:53:50.442 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:50.442 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.442 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.442 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:50.442 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.186861+0000", 2024-02-05T04:53:50.442 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:50.443 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.443 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.443 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:50.443 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.186862+0000", 2024-02-05T04:53:50.443 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.6199999999999999e-07 2024-02-05T04:53:50.443 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.443 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.443 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:50.443 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.186863+0000", 2024-02-05T04:53:50.443 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.378e-06 2024-02-05T04:53:50.443 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.443 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.444 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:50.444 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.186869+0000", 2024-02-05T04:53:50.444 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.4990000000000002e-06 2024-02-05T04:53:50.444 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.444 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.444 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:50.444 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.186873+0000", 2024-02-05T04:53:50.444 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.6550000000000003e-06 2024-02-05T04:53:50.444 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.444 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.444 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:50.444 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.187178+0000", 2024-02-05T04:53:50.445 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00030503099999999999 2024-02-05T04:53:50.445 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.445 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.445 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "waiting for blocked object", 2024-02-05T04:53:50.445 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.187222+0000", 2024-02-05T04:53:50.445 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.3726000000000001e-05 2024-02-05T04:53:50.445 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.445 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.445 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:50.445 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.194173+0000", 2024-02-05T04:53:50.445 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0069509139999999999 2024-02-05T04:53:50.445 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.446 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.446 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:53:50.446 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.194269+0000", 2024-02-05T04:53:50.446 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.6391999999999995e-05 2024-02-05T04:53:50.446 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.446 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.446 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:50.446 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.194363+0000", 2024-02-05T04:53:50.446 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.3665000000000005e-05 2024-02-05T04:53:50.446 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:50.446 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:50.446 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:50.447 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.447 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.447 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:262 3.6s0 3:62c1182c:::smithi19639362-26:head [delete] snapc 3=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e38)", 2024-02-05T04:53:50.447 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:30.132144+0000", 2024-02-05T04:53:50.447 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 20.288174274999999, 2024-02-05T04:53:50.447 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.014886422999999999, 2024-02-05T04:53:50.447 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:50.447 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:50.447 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:53:50.447 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:53:50.447 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:50.447 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 262 2024-02-05T04:53:50.447 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.448 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:50.448 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.448 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:50.448 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.132144+0000", 2024-02-05T04:53:50.448 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:50.448 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.448 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.448 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:50.448 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.132144+0000", 2024-02-05T04:53:50.448 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:50.448 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.448 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.449 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:50.449 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.132146+0000", 2024-02-05T04:53:50.449 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.3990000000000001e-06 2024-02-05T04:53:50.449 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.449 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.449 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:50.449 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.132152+0000", 2024-02-05T04:53:50.449 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.2600000000000002e-06 2024-02-05T04:53:50.449 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.449 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.449 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:50.450 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.132166+0000", 2024-02-05T04:53:50.450 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.3526999999999999e-05 2024-02-05T04:53:50.450 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.450 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.450 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:50.450 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.132175+0000", 2024-02-05T04:53:50.450 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.8020000000000003e-06 2024-02-05T04:53:50.450 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.450 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.450 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:50.450 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.132249+0000", 2024-02-05T04:53:50.450 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.3250000000000005e-05 2024-02-05T04:53:50.451 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.451 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.451 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "op must wait for map", 2024-02-05T04:53:50.451 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.132256+0000", 2024-02-05T04:53:50.451 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.3749999999999997e-06 2024-02-05T04:53:50.451 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.451 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.451 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:50.451 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.139131+0000", 2024-02-05T04:53:50.451 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.006875038 2024-02-05T04:53:50.451 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.451 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.452 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:53:50.452 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.139221+0000", 2024-02-05T04:53:50.452 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.9709000000000005e-05 2024-02-05T04:53:50.452 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.452 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.452 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:50.452 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.140034+0000", 2024-02-05T04:53:50.452 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00081289899999999996 2024-02-05T04:53:50.452 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.452 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.453 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:50.453 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.142644+0000", 2024-02-05T04:53:50.453 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.002610008 2024-02-05T04:53:50.453 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.453 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.453 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "commit_sent", 2024-02-05T04:53:50.453 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.146991+0000", 2024-02-05T04:53:50.453 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0043471509999999996 2024-02-05T04:53:50.453 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.453 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.453 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:50.454 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.147031+0000", 2024-02-05T04:53:50.454 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.0005000000000001e-05 2024-02-05T04:53:50.454 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:50.454 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:50.454 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:50.454 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.454 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.454 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.cs1 38/31 ECSubWrite(tid=41, reqid=client.4634.0:267, at_version=38'3, trim_to=0'0, roll_forward_to=33'2))", 2024-02-05T04:53:50.454 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:30.181544+0000", 2024-02-05T04:53:50.454 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 20.238774504999999, 2024-02-05T04:53:50.454 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0075671979999999998, 2024-02-05T04:53:50.454 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:50.455 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:50.455 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:50.455 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.455 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:50.455 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.181544+0000", 2024-02-05T04:53:50.455 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:50.455 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.455 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.455 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:50.455 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.181544+0000", 2024-02-05T04:53:50.455 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:50.455 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.456 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.456 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:50.456 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.181545+0000", 2024-02-05T04:53:50.456 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.0399999999999996e-07 2024-02-05T04:53:50.456 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.456 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.456 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:50.456 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.182132+0000", 2024-02-05T04:53:50.456 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00058728000000000003 2024-02-05T04:53:50.456 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.456 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.457 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:50.457 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.182146+0000", 2024-02-05T04:53:50.457 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.4334e-05 2024-02-05T04:53:50.457 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.457 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.457 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:50.457 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.182156+0000", 2024-02-05T04:53:50.457 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.4239999999999999e-06 2024-02-05T04:53:50.457 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.457 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.457 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:50.457 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.182229+0000", 2024-02-05T04:53:50.458 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.3676000000000004e-05 2024-02-05T04:53:50.458 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.458 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.458 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:50.458 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.182239+0000", 2024-02-05T04:53:50.458 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.0219999999999993e-06 2024-02-05T04:53:50.458 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.458 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.458 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:50.458 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.189084+0000", 2024-02-05T04:53:50.458 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0068452369999999997 2024-02-05T04:53:50.458 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.458 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.459 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:50.459 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.189111+0000", 2024-02-05T04:53:50.459 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.7620999999999999e-05 2024-02-05T04:53:50.459 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:50.459 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:50.459 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:50.459 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.459 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.460 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:379 3.1s0 3:86872077:::smithi19639362-21:head [copy-from ver 6] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-02-05T04:53:50.460 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:34.158922+0000", 2024-02-05T04:53:50.460 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 16.261396239, 2024-02-05T04:53:50.460 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.012016882, 2024-02-05T04:53:50.461 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:50.461 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:50.461 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:53:50.461 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:53:50.461 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:50.461 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 379 2024-02-05T04:53:50.462 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.462 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:50.462 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.462 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:50.462 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.158922+0000", 2024-02-05T04:53:50.462 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:50.462 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.463 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.463 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:50.463 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.158922+0000", 2024-02-05T04:53:50.463 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:50.463 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.463 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.463 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:50.463 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.158925+0000", 2024-02-05T04:53:50.463 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.3329999999999999e-06 2024-02-05T04:53:50.463 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.463 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.463 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:50.464 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.158938+0000", 2024-02-05T04:53:50.464 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.3373000000000001e-05 2024-02-05T04:53:50.464 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.464 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.464 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:50.464 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.158970+0000", 2024-02-05T04:53:50.465 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.1449000000000003e-05 2024-02-05T04:53:50.465 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.465 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.465 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:50.465 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.158994+0000", 2024-02-05T04:53:50.465 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.4539999999999999e-05 2024-02-05T04:53:50.466 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.466 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.466 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:50.466 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.159134+0000", 2024-02-05T04:53:50.466 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00013956 2024-02-05T04:53:50.466 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.466 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.466 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "op must wait for map", 2024-02-05T04:53:50.466 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.159147+0000", 2024-02-05T04:53:50.467 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.3206e-05 2024-02-05T04:53:50.467 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.467 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.467 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:50.467 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.161069+0000", 2024-02-05T04:53:50.467 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.001921835 2024-02-05T04:53:50.467 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.467 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.468 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:53:50.468 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.161158+0000", 2024-02-05T04:53:50.468 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.9604000000000005e-05 2024-02-05T04:53:50.468 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.468 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.468 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:50.468 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.167690+0000", 2024-02-05T04:53:50.469 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0065311830000000003 2024-02-05T04:53:50.469 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.469 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.469 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:50.469 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.169420+0000", 2024-02-05T04:53:50.469 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.001730325 2024-02-05T04:53:50.469 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.469 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.469 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "commit_sent", 2024-02-05T04:53:50.470 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.170848+0000", 2024-02-05T04:53:50.470 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.001428393 2024-02-05T04:53:50.470 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.470 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.470 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:50.470 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.170939+0000", 2024-02-05T04:53:50.470 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.1081000000000004e-05 2024-02-05T04:53:50.470 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:50.471 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:50.471 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:50.471 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.471 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.471 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:380 3.1s0 3:86872077:::smithi19639362-21:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-02-05T04:53:50.471 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:34.159043+0000", 2024-02-05T04:53:50.471 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 16.261275791999999, 2024-02-05T04:53:50.471 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.010075702000000001, 2024-02-05T04:53:50.472 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:50.472 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "started", 2024-02-05T04:53:50.472 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:53:50.472 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:53:50.472 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:50.472 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 380 2024-02-05T04:53:50.472 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.472 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:50.472 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.472 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:50.472 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.159043+0000", 2024-02-05T04:53:50.472 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:50.473 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.473 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.473 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:50.473 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.159043+0000", 2024-02-05T04:53:50.473 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:50.473 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.473 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.473 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:50.473 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.159045+0000", 2024-02-05T04:53:50.473 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.66e-06 2024-02-05T04:53:50.473 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.473 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.473 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:50.473 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.159052+0000", 2024-02-05T04:53:50.474 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.5299999999999999e-06 2024-02-05T04:53:50.474 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.474 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.474 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:50.474 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.159075+0000", 2024-02-05T04:53:50.474 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.2799e-05 2024-02-05T04:53:50.474 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.474 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.474 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:50.474 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.159092+0000", 2024-02-05T04:53:50.474 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.7116e-05 2024-02-05T04:53:50.474 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.474 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.474 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:50.475 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.159323+0000", 2024-02-05T04:53:50.475 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00023073399999999999 2024-02-05T04:53:50.475 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.475 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.475 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "waiting_for_map not empty", 2024-02-05T04:53:50.475 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.159333+0000", 2024-02-05T04:53:50.475 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.0363999999999999e-05 2024-02-05T04:53:50.475 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.475 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.475 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:50.476 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.161284+0000", 2024-02-05T04:53:50.476 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0019507019999999999 2024-02-05T04:53:50.476 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.476 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.476 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "waiting for blocked object", 2024-02-05T04:53:50.476 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.161320+0000", 2024-02-05T04:53:50.476 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.6494e-05 2024-02-05T04:53:50.476 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.476 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.476 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:50.476 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.168992+0000", 2024-02-05T04:53:50.476 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0076714549999999998 2024-02-05T04:53:50.476 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.476 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.477 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:53:50.477 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.169053+0000", 2024-02-05T04:53:50.477 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.1193999999999997e-05 2024-02-05T04:53:50.477 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.477 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.477 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:50.477 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.169119+0000", 2024-02-05T04:53:50.477 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.5654000000000005e-05 2024-02-05T04:53:50.478 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:50.478 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:50.478 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:50.478 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.478 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.478 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:398 3.1s0 3:8451ec0c:::smithi19639362-8:head [read 0~737280,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-02-05T04:53:50.478 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:35.183088+0000", 2024-02-05T04:53:50.478 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 15.237230625, 2024-02-05T04:53:50.479 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.011852491, 2024-02-05T04:53:50.479 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:50.479 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "started", 2024-02-05T04:53:50.479 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:53:50.479 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:53:50.479 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:50.479 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 398 2024-02-05T04:53:50.480 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.480 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:50.480 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.480 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:50.480 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.183088+0000", 2024-02-05T04:53:50.480 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:50.481 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.481 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.481 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:50.481 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.183088+0000", 2024-02-05T04:53:50.481 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:50.481 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.482 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.482 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:50.482 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.183090+0000", 2024-02-05T04:53:50.482 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.511e-06 2024-02-05T04:53:50.482 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.482 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.482 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:50.483 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.183096+0000", 2024-02-05T04:53:50.483 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.3500000000000002e-06 2024-02-05T04:53:50.483 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.483 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.483 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:50.483 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.183108+0000", 2024-02-05T04:53:50.483 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.1722999999999999e-05 2024-02-05T04:53:50.483 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.484 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.484 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:50.484 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.183116+0000", 2024-02-05T04:53:50.484 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.1939999999999998e-06 2024-02-05T04:53:50.484 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.484 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.484 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:50.484 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.189528+0000", 2024-02-05T04:53:50.485 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0064124450000000001 2024-02-05T04:53:50.485 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.485 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.485 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "waiting for rw locks", 2024-02-05T04:53:50.485 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.189627+0000", 2024-02-05T04:53:50.485 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.8451999999999999e-05 2024-02-05T04:53:50.485 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.486 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.486 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:50.486 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.190600+0000", 2024-02-05T04:53:50.486 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00097288099999999998 2024-02-05T04:53:50.486 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.486 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.487 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:53:50.487 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.190679+0000", 2024-02-05T04:53:50.487 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.9692999999999997e-05 2024-02-05T04:53:50.487 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.487 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.487 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:50.488 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.194941+0000", 2024-02-05T04:53:50.488 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0042612420000000002 2024-02-05T04:53:50.488 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:50.488 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:50.488 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:50.488 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.488 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.489 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:399 3.1s0 3:8451ec0c:::smithi19639362-8:head [read 0~737280,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-02-05T04:53:50.489 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:35.183130+0000", 2024-02-05T04:53:50.489 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 15.237188928, 2024-02-05T04:53:50.489 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.013111441, 2024-02-05T04:53:50.489 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:50.489 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "started", 2024-02-05T04:53:50.489 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:53:50.489 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:53:50.489 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:50.489 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 399 2024-02-05T04:53:50.489 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.489 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:50.489 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.490 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:50.490 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.183130+0000", 2024-02-05T04:53:50.490 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:50.490 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.490 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.490 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:50.490 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.183130+0000", 2024-02-05T04:53:50.490 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:50.490 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.490 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.490 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:50.490 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.183130+0000", 2024-02-05T04:53:50.490 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.37e-07 2024-02-05T04:53:50.490 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.491 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.491 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:50.491 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.183134+0000", 2024-02-05T04:53:50.491 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.189e-06 2024-02-05T04:53:50.491 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.491 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.491 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:50.491 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.183141+0000", 2024-02-05T04:53:50.491 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.8000000000000001e-06 2024-02-05T04:53:50.491 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.491 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.492 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:50.492 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.183181+0000", 2024-02-05T04:53:50.492 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.9864000000000003e-05 2024-02-05T04:53:50.492 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.492 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.492 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:50.493 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.190123+0000", 2024-02-05T04:53:50.493 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0069416510000000001 2024-02-05T04:53:50.493 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.493 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.493 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "waiting for rw locks", 2024-02-05T04:53:50.493 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.190209+0000", 2024-02-05T04:53:50.494 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.6447999999999996e-05 2024-02-05T04:53:50.494 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.494 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.494 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:50.494 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.191024+0000", 2024-02-05T04:53:50.494 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00081457299999999997 2024-02-05T04:53:50.494 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.494 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.494 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:53:50.494 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.191074+0000", 2024-02-05T04:53:50.495 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.0464999999999999e-05 2024-02-05T04:53:50.495 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.495 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.495 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:50.495 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.196241+0000", 2024-02-05T04:53:50.495 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0051669139999999999 2024-02-05T04:53:50.495 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:50.495 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:50.495 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:50.496 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.496 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.496 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:400 3.1s0 3:8451ec0c:::smithi19639362-8:head [read 0~737280,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-02-05T04:53:50.496 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:35.183194+0000", 2024-02-05T04:53:50.496 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 15.237124695, 2024-02-05T04:53:50.496 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.013579028, 2024-02-05T04:53:50.496 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:50.497 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "started", 2024-02-05T04:53:50.497 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:53:50.497 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:53:50.497 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:50.497 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 400 2024-02-05T04:53:50.497 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.497 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:50.498 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.498 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:50.498 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.183194+0000", 2024-02-05T04:53:50.498 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:50.498 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.498 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.499 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:50.499 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.183194+0000", 2024-02-05T04:53:50.500 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:50.500 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.500 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.500 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:50.500 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.183195+0000", 2024-02-05T04:53:50.500 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.0299999999999999e-07 2024-02-05T04:53:50.500 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.500 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.500 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:50.500 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.183198+0000", 2024-02-05T04:53:50.500 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.1990000000000002e-06 2024-02-05T04:53:50.500 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.501 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.501 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:50.501 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.183205+0000", 2024-02-05T04:53:50.501 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.8789999999999997e-06 2024-02-05T04:53:50.501 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.501 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.501 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:50.501 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.183212+0000", 2024-02-05T04:53:50.501 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.2350000000000002e-06 2024-02-05T04:53:50.501 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.501 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.501 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:50.501 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.190255+0000", 2024-02-05T04:53:50.501 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.007042853 2024-02-05T04:53:50.501 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.502 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.502 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "waiting for rw locks", 2024-02-05T04:53:50.502 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.190345+0000", 2024-02-05T04:53:50.502 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.0160000000000004e-05 2024-02-05T04:53:50.502 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.502 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.502 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:50.502 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.193877+0000", 2024-02-05T04:53:50.502 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0035322410000000002 2024-02-05T04:53:50.502 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.502 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.502 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:53:50.502 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.193962+0000", 2024-02-05T04:53:50.502 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.4635000000000005e-05 2024-02-05T04:53:50.503 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.503 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.503 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:50.503 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.196773+0000", 2024-02-05T04:53:50.503 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0028112229999999998 2024-02-05T04:53:50.503 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:50.503 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:50.503 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:50.503 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.503 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.503 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:425 3.6s0 3:62c1182c:::smithi19639362-26:head [read 0~2572288,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-02-05T04:53:50.503 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:40.950996+0000", 2024-02-05T04:53:50.503 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 9.4693222640000005, 2024-02-05T04:53:50.503 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0081791720000000002, 2024-02-05T04:53:50.503 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:50.504 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "started", 2024-02-05T04:53:50.504 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:53:50.504 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:53:50.504 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:50.504 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 425 2024-02-05T04:53:50.504 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.504 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:50.504 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.504 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:50.504 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.950996+0000", 2024-02-05T04:53:50.504 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:50.505 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.505 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.505 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:50.505 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.950996+0000", 2024-02-05T04:53:50.505 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:50.505 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.505 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.505 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:50.505 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.950998+0000", 2024-02-05T04:53:50.505 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.787e-06 2024-02-05T04:53:50.505 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.505 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.505 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:50.506 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.951005+0000", 2024-02-05T04:53:50.506 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.8979999999999999e-06 2024-02-05T04:53:50.506 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.506 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.506 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:50.506 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.951023+0000", 2024-02-05T04:53:50.506 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.7479e-05 2024-02-05T04:53:50.506 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.506 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.506 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:50.506 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.951042+0000", 2024-02-05T04:53:50.506 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.9071000000000001e-05 2024-02-05T04:53:50.506 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.506 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.506 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:50.507 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.951110+0000", 2024-02-05T04:53:50.507 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.8826999999999997e-05 2024-02-05T04:53:50.507 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.507 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.507 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "op must wait for map", 2024-02-05T04:53:50.507 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.951116+0000", 2024-02-05T04:53:50.507 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.6500000000000001e-06 2024-02-05T04:53:50.507 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.507 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.507 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:50.507 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.952955+0000", 2024-02-05T04:53:50.507 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0018388639999999999 2024-02-05T04:53:50.507 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.507 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.507 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:53:50.507 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.953060+0000", 2024-02-05T04:53:50.508 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.000104999 2024-02-05T04:53:50.508 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.508 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.508 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:50.508 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.959176+0000", 2024-02-05T04:53:50.508 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0061155970000000004 2024-02-05T04:53:50.508 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:50.508 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:50.508 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:50.508 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.508 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.508 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:426 3.6s0 3:62c1182c:::smithi19639362-26:head [read 0~2572288,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-02-05T04:53:50.508 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:40.951069+0000", 2024-02-05T04:53:50.508 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 9.4692493550000005, 2024-02-05T04:53:50.508 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.010366143, 2024-02-05T04:53:50.509 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:50.509 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "started", 2024-02-05T04:53:50.509 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:53:50.509 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:53:50.509 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:50.509 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 426 2024-02-05T04:53:50.509 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.509 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:50.509 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.509 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:50.509 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.951069+0000", 2024-02-05T04:53:50.509 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:50.509 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.509 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.509 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:50.510 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.951069+0000", 2024-02-05T04:53:50.510 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:50.510 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.510 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.510 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:50.510 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.951070+0000", 2024-02-05T04:53:50.510 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.7000000000000004e-07 2024-02-05T04:53:50.510 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.510 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.510 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:50.510 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.951073+0000", 2024-02-05T04:53:50.510 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.467e-06 2024-02-05T04:53:50.510 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.510 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.510 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:50.510 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.951086+0000", 2024-02-05T04:53:50.511 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.2999999999999999e-05 2024-02-05T04:53:50.511 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.511 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.511 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:50.511 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.951094+0000", 2024-02-05T04:53:50.511 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.8979999999999996e-06 2024-02-05T04:53:50.511 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.511 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.511 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:50.511 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.951184+0000", 2024-02-05T04:53:50.511 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.9809999999999996e-05 2024-02-05T04:53:50.511 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.511 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.511 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "waiting_for_map not empty", 2024-02-05T04:53:50.511 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.951188+0000", 2024-02-05T04:53:50.512 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.3189999999999996e-06 2024-02-05T04:53:50.512 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.512 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.512 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:50.512 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.953531+0000", 2024-02-05T04:53:50.512 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0023427140000000001 2024-02-05T04:53:50.512 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.512 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.512 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:53:50.512 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.953591+0000", 2024-02-05T04:53:50.512 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.9707000000000003e-05 2024-02-05T04:53:50.512 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.512 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.512 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:50.512 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.961435+0000", 2024-02-05T04:53:50.513 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0078445579999999997 2024-02-05T04:53:50.513 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:50.513 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:50.513 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:50.513 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.513 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.513 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:427 3.6s0 3:62c1182c:::smithi19639362-26:head [read 0~2572288,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-02-05T04:53:50.513 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:40.951117+0000", 2024-02-05T04:53:50.513 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 9.4692016129999992, 2024-02-05T04:53:50.513 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.012236415, 2024-02-05T04:53:50.513 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:50.513 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "started", 2024-02-05T04:53:50.513 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:53:50.513 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:53:50.513 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:50.514 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 427 2024-02-05T04:53:50.514 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.514 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:50.514 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.514 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:50.514 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.951117+0000", 2024-02-05T04:53:50.514 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:50.514 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.514 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.514 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:50.514 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.951117+0000", 2024-02-05T04:53:50.514 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:50.514 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.514 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.514 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:50.515 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.951118+0000", 2024-02-05T04:53:50.515 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.7199999999999998e-07 2024-02-05T04:53:50.515 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.515 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.515 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:50.515 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.951122+0000", 2024-02-05T04:53:50.515 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.2540000000000003e-06 2024-02-05T04:53:50.515 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.515 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.515 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:50.516 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.951130+0000", 2024-02-05T04:53:50.516 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.3760000000000008e-06 2024-02-05T04:53:50.516 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.516 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.516 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:50.516 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.951137+0000", 2024-02-05T04:53:50.516 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.3430000000000001e-06 2024-02-05T04:53:50.516 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.516 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.516 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:50.516 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.951207+0000", 2024-02-05T04:53:50.516 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.9974999999999993e-05 2024-02-05T04:53:50.516 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.516 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.516 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "waiting_for_map not empty", 2024-02-05T04:53:50.517 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.951210+0000", 2024-02-05T04:53:50.517 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.4589999999999998e-06 2024-02-05T04:53:50.517 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.517 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.517 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:50.517 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.955115+0000", 2024-02-05T04:53:50.517 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.003904687 2024-02-05T04:53:50.517 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.517 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.517 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:53:50.517 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.955169+0000", 2024-02-05T04:53:50.517 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.3823999999999998e-05 2024-02-05T04:53:50.517 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.517 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.518 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:50.518 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.963353+0000", 2024-02-05T04:53:50.518 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0081848249999999997 2024-02-05T04:53:50.518 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:50.518 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:50.518 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:50.518 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.518 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.518 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:428 3.bs0 3:de65491d:::smithi19639362-2:head [append 0~753664] snapc b=[b,a,9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-02-05T04:53:50.518 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:40.962550+0000", 2024-02-05T04:53:50.518 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 9.4577690400000005, 2024-02-05T04:53:50.518 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0091527599999999994, 2024-02-05T04:53:50.518 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:50.518 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:50.518 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:53:50.519 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:53:50.519 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:50.519 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 428 2024-02-05T04:53:50.519 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.519 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:50.519 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.519 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:50.519 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.962550+0000", 2024-02-05T04:53:50.519 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:50.519 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.519 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.519 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:50.519 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.962550+0000", 2024-02-05T04:53:50.519 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:50.519 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.519 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.520 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:50.520 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.962551+0000", 2024-02-05T04:53:50.520 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.533e-06 2024-02-05T04:53:50.520 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.520 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.520 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:50.520 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.964759+0000", 2024-02-05T04:53:50.520 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0022082429999999999 2024-02-05T04:53:50.520 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.520 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.520 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:50.520 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.964774+0000", 2024-02-05T04:53:50.520 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.4729000000000001e-05 2024-02-05T04:53:50.520 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.520 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.521 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:50.521 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.964783+0000", 2024-02-05T04:53:50.521 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.5720000000000003e-06 2024-02-05T04:53:50.521 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.521 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.521 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:50.521 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.964862+0000", 2024-02-05T04:53:50.521 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.9208000000000005e-05 2024-02-05T04:53:50.521 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.521 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.521 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:53:50.521 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.964948+0000", 2024-02-05T04:53:50.521 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.6297000000000004e-05 2024-02-05T04:53:50.521 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.521 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.521 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:50.522 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.966185+0000", 2024-02-05T04:53:50.522 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.001236308 2024-02-05T04:53:50.522 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.522 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.522 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:50.522 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.967870+0000", 2024-02-05T04:53:50.522 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0016859659999999999 2024-02-05T04:53:50.522 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.522 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.522 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "commit_sent", 2024-02-05T04:53:50.522 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.971637+0000", 2024-02-05T04:53:50.522 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0037667590000000002 2024-02-05T04:53:50.522 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.522 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.522 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:50.522 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.971702+0000", 2024-02-05T04:53:50.523 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.5145000000000001e-05 2024-02-05T04:53:50.523 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:50.523 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:50.523 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:50.523 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.523 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.523 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:429 3.bs0 3:de65491d:::smithi19639362-2:head [append 0~720896] snapc b=[b,a,9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-02-05T04:53:50.523 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:40.966654+0000", 2024-02-05T04:53:50.523 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 9.4536646799999993, 2024-02-05T04:53:50.523 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0079481570000000008, 2024-02-05T04:53:50.523 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:50.523 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:50.523 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:53:50.523 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:53:50.523 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:50.524 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 429 2024-02-05T04:53:50.524 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.524 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:50.524 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.524 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:50.524 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.966654+0000", 2024-02-05T04:53:50.524 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:50.524 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.524 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.524 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:50.524 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.966654+0000", 2024-02-05T04:53:50.524 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:50.524 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.524 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.524 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:50.525 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.966656+0000", 2024-02-05T04:53:50.525 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.8929999999999999e-06 2024-02-05T04:53:50.525 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.525 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.525 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:50.525 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.969453+0000", 2024-02-05T04:53:50.525 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0027969929999999998 2024-02-05T04:53:50.525 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.525 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.525 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:50.526 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.969466+0000", 2024-02-05T04:53:50.526 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.2921e-05 2024-02-05T04:53:50.526 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.526 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.526 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:50.526 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.969479+0000", 2024-02-05T04:53:50.526 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.341e-05 2024-02-05T04:53:50.526 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.526 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.526 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:50.526 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.969622+0000", 2024-02-05T04:53:50.526 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00014324800000000001 2024-02-05T04:53:50.526 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.526 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.526 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:53:50.527 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.969742+0000", 2024-02-05T04:53:50.527 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.000119757 2024-02-05T04:53:50.527 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.527 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.527 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:50.527 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.970742+0000", 2024-02-05T04:53:50.527 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.000999785 2024-02-05T04:53:50.527 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.527 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.527 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:50.527 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.973298+0000", 2024-02-05T04:53:50.527 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0025562290000000001 2024-02-05T04:53:50.527 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.527 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.527 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "commit_sent", 2024-02-05T04:53:50.528 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.974555+0000", 2024-02-05T04:53:50.528 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0012567240000000001 2024-02-05T04:53:50.528 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.528 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.528 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:50.528 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.974602+0000", 2024-02-05T04:53:50.528 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.7197000000000002e-05 2024-02-05T04:53:50.528 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:50.528 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:50.528 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:50.528 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.528 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.528 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:431 3.bs0 3:de65491d:::smithi19639362-2:head [setxattr _header (56) in=7b] snapc b=[b,a,9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-02-05T04:53:50.528 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:40.969591+0000", 2024-02-05T04:53:50.528 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 9.4507272980000003, 2024-02-05T04:53:50.529 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.011310982000000001, 2024-02-05T04:53:50.529 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:50.529 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:50.529 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:53:50.529 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:53:50.529 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:50.529 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 431 2024-02-05T04:53:50.529 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.529 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:50.529 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.529 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:50.529 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.969591+0000", 2024-02-05T04:53:50.529 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:50.529 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.529 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.530 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:50.530 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.969591+0000", 2024-02-05T04:53:50.530 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:50.530 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.530 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.530 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:50.530 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.969592+0000", 2024-02-05T04:53:50.530 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.75e-07 2024-02-05T04:53:50.530 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.530 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.530 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:50.530 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.969596+0000", 2024-02-05T04:53:50.530 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.0550000000000001e-06 2024-02-05T04:53:50.530 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.530 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.531 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:50.531 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.969605+0000", 2024-02-05T04:53:50.531 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.0520000000000007e-06 2024-02-05T04:53:50.531 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.531 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.531 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:50.531 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.969611+0000", 2024-02-05T04:53:50.531 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.2990000000000002e-06 2024-02-05T04:53:50.531 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.531 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.531 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:50.531 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.972667+0000", 2024-02-05T04:53:50.531 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0030555030000000002 2024-02-05T04:53:50.531 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.531 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.532 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:53:50.532 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.972758+0000", 2024-02-05T04:53:50.532 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.0587999999999994e-05 2024-02-05T04:53:50.532 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.532 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.532 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:50.532 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.973132+0000", 2024-02-05T04:53:50.532 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00037420900000000002 2024-02-05T04:53:50.532 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.532 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.532 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:50.532 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.980593+0000", 2024-02-05T04:53:50.532 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0074614390000000003 2024-02-05T04:53:50.532 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.532 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.532 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "commit_sent", 2024-02-05T04:53:50.533 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.980636+0000", 2024-02-05T04:53:50.533 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.3025999999999998e-05 2024-02-05T04:53:50.533 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.533 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.533 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:50.533 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.980902+0000", 2024-02-05T04:53:50.533 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00026613599999999999 2024-02-05T04:53:50.533 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:50.533 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:50.533 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:50.533 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.533 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.533 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:432 3.bs0 3:de65491d:::smithi19639362-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-02-05T04:53:50.533 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:40.969620+0000", 2024-02-05T04:53:50.533 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 9.4506984989999996, 2024-02-05T04:53:50.534 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.012177466, 2024-02-05T04:53:50.534 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:50.534 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "started", 2024-02-05T04:53:50.534 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:53:50.534 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:53:50.534 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:50.534 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 432 2024-02-05T04:53:50.534 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.534 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:50.534 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.534 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:50.534 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.969620+0000", 2024-02-05T04:53:50.534 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:50.534 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.534 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.534 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:50.535 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.969620+0000", 2024-02-05T04:53:50.535 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:50.535 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.535 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.535 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:50.535 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.969621+0000", 2024-02-05T04:53:50.535 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.75e-07 2024-02-05T04:53:50.535 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.535 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.535 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:50.535 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.969623+0000", 2024-02-05T04:53:50.535 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.7460000000000001e-06 2024-02-05T04:53:50.535 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.535 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.535 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:50.536 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.969633+0000", 2024-02-05T04:53:50.536 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.1209999999999996e-06 2024-02-05T04:53:50.536 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.536 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.536 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:50.536 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.969639+0000", 2024-02-05T04:53:50.536 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.0000000000000002e-06 2024-02-05T04:53:50.536 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.536 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.536 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:50.536 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.973551+0000", 2024-02-05T04:53:50.536 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0039128829999999998 2024-02-05T04:53:50.536 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.536 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.536 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "waiting for rw locks", 2024-02-05T04:53:50.537 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.973609+0000", 2024-02-05T04:53:50.537 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.7367000000000001e-05 2024-02-05T04:53:50.537 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.537 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.537 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:50.537 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.980931+0000", 2024-02-05T04:53:50.537 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.007322131 2024-02-05T04:53:50.537 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.537 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.537 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:53:50.537 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.980979+0000", 2024-02-05T04:53:50.537 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.8288000000000003e-05 2024-02-05T04:53:50.537 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.537 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.537 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:50.537 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.981798+0000", 2024-02-05T04:53:50.538 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00081845500000000005 2024-02-05T04:53:50.538 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:50.538 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:50.538 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:50.538 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.538 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.538 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:498 3.7s0 3:e54c8e71:::smithi19639362-27:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e52)", 2024-02-05T04:53:50.538 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:43.973924+0000", 2024-02-05T04:53:50.538 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 6.4463942809999999, 2024-02-05T04:53:50.538 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0086285349999999997, 2024-02-05T04:53:50.538 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:50.538 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "started", 2024-02-05T04:53:50.538 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:53:50.538 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:53:50.538 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:50.539 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 498 2024-02-05T04:53:50.539 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.539 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:50.539 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.539 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:50.539 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:43.973924+0000", 2024-02-05T04:53:50.539 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:50.539 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.539 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.539 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:50.539 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:43.973924+0000", 2024-02-05T04:53:50.539 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:50.539 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.539 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.539 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:50.540 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:43.973925+0000", 2024-02-05T04:53:50.540 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.7e-07 2024-02-05T04:53:50.540 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.540 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.540 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:50.540 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:43.973926+0000", 2024-02-05T04:53:50.540 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.3790000000000001e-06 2024-02-05T04:53:50.540 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.540 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.540 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:50.540 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:43.973931+0000", 2024-02-05T04:53:50.540 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.9130000000000002e-06 2024-02-05T04:53:50.540 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.540 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.540 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:50.541 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:43.973934+0000", 2024-02-05T04:53:50.541 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.0879999999999999e-06 2024-02-05T04:53:50.541 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.541 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.541 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:50.541 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:43.976778+0000", 2024-02-05T04:53:50.541 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0028436329999999999 2024-02-05T04:53:50.541 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.541 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.541 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "waiting for rw locks", 2024-02-05T04:53:50.541 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:43.976831+0000", 2024-02-05T04:53:50.541 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.2955999999999999e-05 2024-02-05T04:53:50.541 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.541 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.541 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:50.542 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:43.981578+0000", 2024-02-05T04:53:50.542 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0047468019999999996 2024-02-05T04:53:50.542 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.542 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.542 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:53:50.542 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:43.981665+0000", 2024-02-05T04:53:50.542 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.7497000000000006e-05 2024-02-05T04:53:50.542 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.542 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.542 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:50.542 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:43.982553+0000", 2024-02-05T04:53:50.542 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00088789700000000001 2024-02-05T04:53:50.542 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:50.542 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:50.542 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:50.542 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.543 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.543 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.ds5 58/31 ECSubWrite(tid=56, reqid=client.4634.0:526, at_version=58'35, trim_to=0'0, roll_forward_to=57'34))", 2024-02-05T04:53:50.543 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:49.970090+0000", 2024-02-05T04:53:50.543 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 0.45022911500000001, 2024-02-05T04:53:50.543 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0092720189999999994, 2024-02-05T04:53:50.543 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:50.543 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:50.543 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:50.543 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.543 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:50.543 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:49.970090+0000", 2024-02-05T04:53:50.543 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:50.543 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.543 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.543 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:50.543 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:49.970090+0000", 2024-02-05T04:53:50.544 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:50.544 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.544 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.544 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:50.544 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:49.970090+0000", 2024-02-05T04:53:50.544 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.6199999999999997e-07 2024-02-05T04:53:50.544 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.544 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.544 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:50.544 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:49.973039+0000", 2024-02-05T04:53:50.544 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0029486439999999998 2024-02-05T04:53:50.544 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.544 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.544 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:50.544 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:49.973075+0000", 2024-02-05T04:53:50.545 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.587e-05 2024-02-05T04:53:50.545 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.545 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.545 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:50.545 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:49.973083+0000", 2024-02-05T04:53:50.545 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.5960000000000004e-06 2024-02-05T04:53:50.545 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.545 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.545 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:50.545 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:49.973150+0000", 2024-02-05T04:53:50.545 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.6478999999999998e-05 2024-02-05T04:53:50.545 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.545 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.545 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:50.545 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:49.973159+0000", 2024-02-05T04:53:50.545 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.8820000000000007e-06 2024-02-05T04:53:50.546 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.546 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.546 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:50.546 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:49.979327+0000", 2024-02-05T04:53:50.546 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0061679059999999999 2024-02-05T04:53:50.546 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:50.546 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:50.546 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:50.546 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:49.979362+0000", 2024-02-05T04:53:50.546 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.4980000000000001e-05 2024-02-05T04:53:50.546 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:50.546 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:50.546 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:50.546 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:50.546 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:50.546 INFO:teuthology.orchestra.run.smithi091.stdout:} 2024-02-05T04:53:50.548 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-02-05T04:53:50.549 INFO:tasks.ceph.osd.11.smithi196.stderr:2024-02-05T04:53:50.547+0000 7f85a8d86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 34609) UID: 0 2024-02-05T04:53:50.648 INFO:tasks.ceph.osd.9.smithi091.stderr:2024-02-05T04:53:50.647+0000 7efc18827640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 34279) UID: 0 2024-02-05T04:53:50.665 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-02-05T04:53:50.666 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [], 2024-02-05T04:53:50.666 INFO:teuthology.orchestra.run.smithi150.stdout: "num_ops": 0 2024-02-05T04:53:50.666 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-02-05T04:53:50.675 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-02-05T04:53:50.749 INFO:tasks.ceph.osd.4.smithi062.stderr:2024-02-05T04:53:50.749+0000 7f708c308640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34689) UID: 0 2024-02-05T04:53:50.781 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-02-05T04:53:50.782 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [], 2024-02-05T04:53:50.782 INFO:teuthology.orchestra.run.smithi150.stdout: "complaint_time": 30, 2024-02-05T04:53:50.782 INFO:teuthology.orchestra.run.smithi150.stdout: "num_blocked_ops": 0 2024-02-05T04:53:50.782 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-02-05T04:53:50.790 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-02-05T04:53:50.851 INFO:tasks.ceph.osd.6.smithi150.stderr:2024-02-05T04:53:50.850+0000 7f6bea9f0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34015) UID: 0 2024-02-05T04:53:50.898 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-02-05T04:53:50.898 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 20, 2024-02-05T04:53:50.898 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 600, 2024-02-05T04:53:50.898 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [ 2024-02-05T04:53:50.898 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.898 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.4209.0:85 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'55, mlcod=24'50)", 2024-02-05T04:53:50.898 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:09.697543+0000", 2024-02-05T04:53:50.898 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 41.198543268999998, 2024-02-05T04:53:50.898 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0081615030000000005, 2024-02-05T04:53:50.898 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:50.898 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:50.899 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:50.899 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.899 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:50.899 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.697543+0000", 2024-02-05T04:53:50.899 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:50.899 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.899 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.899 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:50.899 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.697543+0000", 2024-02-05T04:53:50.899 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:50.899 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.899 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.899 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:50.899 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.697544+0000", 2024-02-05T04:53:50.899 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.7899999999999999e-07 2024-02-05T04:53:50.900 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.900 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.900 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:50.900 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.697546+0000", 2024-02-05T04:53:50.900 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.7970000000000001e-06 2024-02-05T04:53:50.900 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.900 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.900 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:50.900 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.697553+0000", 2024-02-05T04:53:50.900 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.2929999999999997e-06 2024-02-05T04:53:50.900 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.900 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.900 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:50.900 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.697556+0000", 2024-02-05T04:53:50.900 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.4309999999999998e-06 2024-02-05T04:53:50.901 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.901 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.901 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:50.901 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.697596+0000", 2024-02-05T04:53:50.901 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.977e-05 2024-02-05T04:53:50.901 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.901 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.901 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:53:50.901 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.697607+0000", 2024-02-05T04:53:50.901 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1459e-05 2024-02-05T04:53:50.901 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.901 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.901 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-02-05T04:53:50.901 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.705669+0000", 2024-02-05T04:53:50.901 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0080620009999999992 2024-02-05T04:53:50.902 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.902 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.902 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:50.902 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.705705+0000", 2024-02-05T04:53:50.902 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.5373000000000002e-05 2024-02-05T04:53:50.902 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:50.902 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:50.902 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:50.902 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.902 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.902 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.4209.0:86 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'56, mlcod=24'52)", 2024-02-05T04:53:50.902 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:09.698014+0000", 2024-02-05T04:53:50.902 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 41.198072169, 2024-02-05T04:53:50.902 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0077071750000000001, 2024-02-05T04:53:50.902 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:50.902 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:50.903 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:50.903 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.903 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:50.903 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698014+0000", 2024-02-05T04:53:50.903 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:50.903 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.903 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.903 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:50.903 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698014+0000", 2024-02-05T04:53:50.903 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:50.903 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.903 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.903 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:50.903 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698015+0000", 2024-02-05T04:53:50.903 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.9900000000000002e-07 2024-02-05T04:53:50.903 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.904 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.904 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:50.904 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698018+0000", 2024-02-05T04:53:50.904 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.0019999999999998e-06 2024-02-05T04:53:50.904 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.904 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.904 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:50.904 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698024+0000", 2024-02-05T04:53:50.904 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.251e-06 2024-02-05T04:53:50.904 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.904 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.904 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:50.904 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698028+0000", 2024-02-05T04:53:50.904 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.76e-06 2024-02-05T04:53:50.904 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.905 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.905 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:50.905 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698066+0000", 2024-02-05T04:53:50.905 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.8556999999999997e-05 2024-02-05T04:53:50.905 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.905 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.905 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:53:50.905 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698078+0000", 2024-02-05T04:53:50.905 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1620999999999999e-05 2024-02-05T04:53:50.905 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.905 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.905 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-02-05T04:53:50.905 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.705706+0000", 2024-02-05T04:53:50.905 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0076284400000000002 2024-02-05T04:53:50.905 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.906 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.906 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:50.906 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.705721+0000", 2024-02-05T04:53:50.906 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.5245000000000001e-05 2024-02-05T04:53:50.906 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:50.906 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:50.906 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:50.906 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.906 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.906 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.4209.0:87 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'57, mlcod=24'52)", 2024-02-05T04:53:50.906 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:09.698307+0000", 2024-02-05T04:53:50.906 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 41.197779240999999, 2024-02-05T04:53:50.906 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0074282530000000001, 2024-02-05T04:53:50.906 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:50.906 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:50.906 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:50.907 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.907 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:50.907 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698307+0000", 2024-02-05T04:53:50.907 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:50.907 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.907 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.907 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:50.907 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698307+0000", 2024-02-05T04:53:50.907 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:50.907 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.907 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.907 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:50.907 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698308+0000", 2024-02-05T04:53:50.907 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.9900000000000001e-07 2024-02-05T04:53:50.907 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.907 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.908 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:50.908 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698358+0000", 2024-02-05T04:53:50.908 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.0649000000000002e-05 2024-02-05T04:53:50.908 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.908 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.908 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:50.908 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698366+0000", 2024-02-05T04:53:50.908 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.9000000000000006e-06 2024-02-05T04:53:50.908 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.908 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.908 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:50.908 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698370+0000", 2024-02-05T04:53:50.908 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.7270000000000001e-06 2024-02-05T04:53:50.908 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.908 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.909 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:50.909 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698410+0000", 2024-02-05T04:53:50.909 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.0117000000000002e-05 2024-02-05T04:53:50.909 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.909 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.909 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:53:50.909 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698423+0000", 2024-02-05T04:53:50.909 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2531000000000001e-05 2024-02-05T04:53:50.909 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.909 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.909 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-02-05T04:53:50.909 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.705722+0000", 2024-02-05T04:53:50.909 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0072995270000000001 2024-02-05T04:53:50.909 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.909 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.909 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:50.910 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.705735+0000", 2024-02-05T04:53:50.910 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.3403e-05 2024-02-05T04:53:50.910 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:50.910 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:50.910 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:50.910 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.910 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.910 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.4209.0:88 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'58, mlcod=24'52)", 2024-02-05T04:53:50.910 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:09.698756+0000", 2024-02-05T04:53:50.910 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 41.197330153000003, 2024-02-05T04:53:50.910 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0072080540000000002, 2024-02-05T04:53:50.910 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:50.910 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:50.910 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:50.910 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.911 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:50.911 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698756+0000", 2024-02-05T04:53:50.911 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:50.911 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.911 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.911 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:50.911 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698756+0000", 2024-02-05T04:53:50.911 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:50.911 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.911 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.911 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:50.911 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698757+0000", 2024-02-05T04:53:50.911 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.6899999999999997e-07 2024-02-05T04:53:50.911 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.911 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.912 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:50.912 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698762+0000", 2024-02-05T04:53:50.912 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.8790000000000002e-06 2024-02-05T04:53:50.912 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.912 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.912 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:50.912 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698773+0000", 2024-02-05T04:53:50.912 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1e-05 2024-02-05T04:53:50.912 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.912 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.912 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:50.912 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698780+0000", 2024-02-05T04:53:50.912 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.8920000000000003e-06 2024-02-05T04:53:50.912 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.912 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.913 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:50.913 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.700009+0000", 2024-02-05T04:53:50.913 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.001229665 2024-02-05T04:53:50.913 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.913 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.913 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:53:50.913 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.700022+0000", 2024-02-05T04:53:50.913 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2699e-05 2024-02-05T04:53:50.913 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.913 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.913 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-02-05T04:53:50.913 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.705943+0000", 2024-02-05T04:53:50.913 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0059209759999999997 2024-02-05T04:53:50.913 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.913 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.913 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:50.914 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.705964+0000", 2024-02-05T04:53:50.914 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.1274e-05 2024-02-05T04:53:50.914 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:50.914 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:50.914 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:50.914 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.914 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.914 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.4209.0:110 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'72, mlcod=24'67)", 2024-02-05T04:53:50.914 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:09.853702+0000", 2024-02-05T04:53:50.914 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 41.042383971, 2024-02-05T04:53:50.914 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0063171329999999999, 2024-02-05T04:53:50.914 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:50.914 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:50.914 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:50.914 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.915 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:50.915 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.853702+0000", 2024-02-05T04:53:50.915 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:50.915 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.915 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.915 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:50.915 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.853702+0000", 2024-02-05T04:53:50.915 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:50.915 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.915 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.915 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:50.915 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.853703+0000", 2024-02-05T04:53:50.915 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.1100000000000002e-07 2024-02-05T04:53:50.915 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.915 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.915 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:50.916 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.853706+0000", 2024-02-05T04:53:50.916 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.3629999999999998e-06 2024-02-05T04:53:50.916 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.916 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.916 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:50.916 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.853713+0000", 2024-02-05T04:53:50.916 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.8410000000000002e-06 2024-02-05T04:53:50.916 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.916 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.916 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:50.916 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.853717+0000", 2024-02-05T04:53:50.916 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.777e-06 2024-02-05T04:53:50.916 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.916 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.916 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:50.917 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.856746+0000", 2024-02-05T04:53:50.917 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0030292800000000001 2024-02-05T04:53:50.917 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.917 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.917 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:53:50.917 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.856762+0000", 2024-02-05T04:53:50.917 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.5651999999999999e-05 2024-02-05T04:53:50.917 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.917 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.917 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-02-05T04:53:50.917 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.859984+0000", 2024-02-05T04:53:50.917 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0032218289999999998 2024-02-05T04:53:50.917 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.917 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.917 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:50.918 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.860020+0000", 2024-02-05T04:53:50.918 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.608e-05 2024-02-05T04:53:50.918 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:50.918 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:50.918 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:50.918 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.918 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.918 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.4209.0:111 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'73, mlcod=24'68)", 2024-02-05T04:53:50.918 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:09.854221+0000", 2024-02-05T04:53:50.918 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 41.041865639999997, 2024-02-05T04:53:50.918 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0058190960000000002, 2024-02-05T04:53:50.918 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:50.918 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:50.918 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:50.918 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.918 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:50.919 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.854221+0000", 2024-02-05T04:53:50.919 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:50.919 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.919 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.919 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:50.919 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.854221+0000", 2024-02-05T04:53:50.919 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:50.919 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.919 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.919 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:50.919 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.854221+0000", 2024-02-05T04:53:50.919 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.72e-07 2024-02-05T04:53:50.919 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.919 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.919 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:50.919 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.854224+0000", 2024-02-05T04:53:50.920 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.1779999999999999e-06 2024-02-05T04:53:50.920 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.920 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.920 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:50.920 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.854231+0000", 2024-02-05T04:53:50.920 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.7440000000000001e-06 2024-02-05T04:53:50.920 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.920 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.920 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:50.920 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.854235+0000", 2024-02-05T04:53:50.920 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.8349999999999997e-06 2024-02-05T04:53:50.920 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.920 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.920 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:50.920 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.856952+0000", 2024-02-05T04:53:50.921 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0027175839999999999 2024-02-05T04:53:50.921 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.921 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.921 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:53:50.921 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.856966+0000", 2024-02-05T04:53:50.921 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.3739e-05 2024-02-05T04:53:50.921 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.921 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.921 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-02-05T04:53:50.921 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.860023+0000", 2024-02-05T04:53:50.921 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0030564979999999999 2024-02-05T04:53:50.921 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.921 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.921 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:50.921 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.860040+0000", 2024-02-05T04:53:50.921 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.7146000000000002e-05 2024-02-05T04:53:50.922 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:50.922 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:50.922 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:50.922 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.922 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.922 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.4209.0:115 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'77, mlcod=24'69)", 2024-02-05T04:53:50.922 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:09.856748+0000", 2024-02-05T04:53:50.922 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 41.039338753999999, 2024-02-05T04:53:50.922 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0058814849999999997, 2024-02-05T04:53:50.922 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:50.922 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:50.922 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:50.922 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.922 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:50.922 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.856748+0000", 2024-02-05T04:53:50.922 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:50.923 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.923 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.923 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:50.923 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.856748+0000", 2024-02-05T04:53:50.923 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:50.923 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.923 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.923 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:50.923 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.856748+0000", 2024-02-05T04:53:50.923 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.4099999999999998e-07 2024-02-05T04:53:50.923 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.923 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.923 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:50.923 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.856781+0000", 2024-02-05T04:53:50.924 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.3056999999999999e-05 2024-02-05T04:53:50.924 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.924 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.924 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:50.924 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.856791+0000", 2024-02-05T04:53:50.924 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.5659999999999996e-06 2024-02-05T04:53:50.924 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.924 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.924 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:50.924 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.856799+0000", 2024-02-05T04:53:50.924 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.8720000000000002e-06 2024-02-05T04:53:50.924 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.924 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.924 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:50.924 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.857805+0000", 2024-02-05T04:53:50.925 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0010061180000000001 2024-02-05T04:53:50.925 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.925 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.925 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:53:50.925 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.857820+0000", 2024-02-05T04:53:50.925 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.4883e-05 2024-02-05T04:53:50.925 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.925 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.925 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-02-05T04:53:50.925 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.862596+0000", 2024-02-05T04:53:50.925 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0047763249999999997 2024-02-05T04:53:50.925 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.925 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.925 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:50.925 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.862629+0000", 2024-02-05T04:53:50.925 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.2923000000000003e-05 2024-02-05T04:53:50.926 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:50.926 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:50.926 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:50.926 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.926 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.926 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.4209.0:117 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'79, mlcod=24'69)", 2024-02-05T04:53:50.926 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:09.857803+0000", 2024-02-05T04:53:50.926 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 41.038283028999999, 2024-02-05T04:53:50.926 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0076301729999999996, 2024-02-05T04:53:50.926 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:50.926 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:50.926 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:50.926 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.926 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:50.926 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.857803+0000", 2024-02-05T04:53:50.927 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:50.927 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.927 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.927 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:50.927 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.857803+0000", 2024-02-05T04:53:50.927 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:50.927 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.927 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.927 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:50.927 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.857804+0000", 2024-02-05T04:53:50.927 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.1999999999999999e-07 2024-02-05T04:53:50.927 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.927 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.927 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:50.927 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.857810+0000", 2024-02-05T04:53:50.928 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.8690000000000001e-06 2024-02-05T04:53:50.928 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.928 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.928 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:50.928 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.857821+0000", 2024-02-05T04:53:50.928 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1059e-05 2024-02-05T04:53:50.928 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.928 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.928 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:50.928 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.857827+0000", 2024-02-05T04:53:50.928 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.1430000000000003e-06 2024-02-05T04:53:50.928 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.928 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.928 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:50.928 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.860138+0000", 2024-02-05T04:53:50.929 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0023111160000000002 2024-02-05T04:53:50.929 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.929 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.929 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:53:50.929 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.860152+0000", 2024-02-05T04:53:50.929 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.3896999999999999e-05 2024-02-05T04:53:50.929 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.929 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.929 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-02-05T04:53:50.929 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.865400+0000", 2024-02-05T04:53:50.930 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0052475040000000001 2024-02-05T04:53:50.930 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.930 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.930 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:50.930 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.865434+0000", 2024-02-05T04:53:50.930 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.3865000000000002e-05 2024-02-05T04:53:50.930 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:50.930 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:50.930 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:50.930 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.930 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.930 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.4634.0:245 3.es0 3:796382fe:::smithi19639362-29:head [rollback 3] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:53:50.930 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:29.186870+0000", 2024-02-05T04:53:50.930 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 21.709216253000001, 2024-02-05T04:53:50.930 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0071777610000000004, 2024-02-05T04:53:50.930 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:50.931 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:50.931 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-02-05T04:53:50.931 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.4634", 2024-02-05T04:53:50.931 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:50.931 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 245 2024-02-05T04:53:50.931 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.931 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:50.931 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.931 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:50.931 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186870+0000", 2024-02-05T04:53:50.931 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:50.931 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.931 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.931 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:50.931 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186870+0000", 2024-02-05T04:53:50.931 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:50.932 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.932 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.932 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:50.932 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186870+0000", 2024-02-05T04:53:50.932 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.0400000000000002e-07 2024-02-05T04:53:50.932 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.932 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.932 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:50.932 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186874+0000", 2024-02-05T04:53:50.932 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.348e-06 2024-02-05T04:53:50.932 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.932 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.932 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:50.932 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186878+0000", 2024-02-05T04:53:50.932 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.481e-06 2024-02-05T04:53:50.933 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.933 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.933 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:50.933 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186881+0000", 2024-02-05T04:53:50.933 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.1879999999999998e-06 2024-02-05T04:53:50.933 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.933 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.933 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:50.933 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.188893+0000", 2024-02-05T04:53:50.933 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.002011231 2024-02-05T04:53:50.933 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.933 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.933 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:53:50.933 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.189032+0000", 2024-02-05T04:53:50.933 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000139467 2024-02-05T04:53:50.933 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.934 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.934 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:50.934 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.189572+0000", 2024-02-05T04:53:50.934 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00053986300000000002 2024-02-05T04:53:50.934 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.934 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.934 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:50.934 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.192597+0000", 2024-02-05T04:53:50.934 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0030245659999999998 2024-02-05T04:53:50.934 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.934 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.934 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-02-05T04:53:50.934 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.193877+0000", 2024-02-05T04:53:50.934 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.001280109 2024-02-05T04:53:50.934 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.934 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.935 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:50.935 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.194048+0000", 2024-02-05T04:53:50.935 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000171204 2024-02-05T04:53:50.935 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:50.935 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:50.935 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:50.935 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.935 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.935 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.4634.0:246 3.es0 3:796382fe:::smithi19639362-29:head [append 0~0] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:53:50.935 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:29.186925+0000", 2024-02-05T04:53:50.935 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 21.709161709, 2024-02-05T04:53:50.935 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0088703129999999995, 2024-02-05T04:53:50.935 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:50.935 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:50.935 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-02-05T04:53:50.935 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.4634", 2024-02-05T04:53:50.936 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:50.936 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 246 2024-02-05T04:53:50.936 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.936 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:50.936 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.936 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:50.936 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186925+0000", 2024-02-05T04:53:50.936 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:50.936 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.936 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.936 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:50.936 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186925+0000", 2024-02-05T04:53:50.936 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:50.936 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.937 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.937 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:50.937 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186925+0000", 2024-02-05T04:53:50.937 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.53e-07 2024-02-05T04:53:50.937 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.937 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.937 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:50.937 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186928+0000", 2024-02-05T04:53:50.937 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.6900000000000001e-06 2024-02-05T04:53:50.937 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.937 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.937 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:50.937 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186933+0000", 2024-02-05T04:53:50.937 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.502e-06 2024-02-05T04:53:50.937 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.937 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.938 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:50.938 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186937+0000", 2024-02-05T04:53:50.938 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.6200000000000001e-06 2024-02-05T04:53:50.938 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.938 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.938 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:50.938 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.190223+0000", 2024-02-05T04:53:50.938 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0032863050000000002 2024-02-05T04:53:50.938 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.938 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.938 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:53:50.938 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.190312+0000", 2024-02-05T04:53:50.938 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.8821999999999998e-05 2024-02-05T04:53:50.938 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.938 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.939 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:50.939 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.190713+0000", 2024-02-05T04:53:50.939 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00040077400000000001 2024-02-05T04:53:50.939 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.939 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.939 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:50.939 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.193148+0000", 2024-02-05T04:53:50.939 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.002435544 2024-02-05T04:53:50.939 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.939 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.939 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-02-05T04:53:50.939 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.195751+0000", 2024-02-05T04:53:50.939 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0026025000000000002 2024-02-05T04:53:50.939 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.939 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.940 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:50.940 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.195795+0000", 2024-02-05T04:53:50.940 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.4202999999999997e-05 2024-02-05T04:53:50.940 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:50.940 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:50.940 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:50.940 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.940 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.940 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.4634.0:249 3.es0 3:70a5b9dd:::smithi19639362-46:head [delete] snapc 3=[3] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:53:50.940 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:29.186943+0000", 2024-02-05T04:53:50.940 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 21.709143748999999, 2024-02-05T04:53:50.940 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.011398484, 2024-02-05T04:53:50.940 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:50.940 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:50.940 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-02-05T04:53:50.941 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.4634", 2024-02-05T04:53:50.941 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:50.941 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 249 2024-02-05T04:53:50.941 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.941 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:50.941 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.941 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:50.941 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186943+0000", 2024-02-05T04:53:50.941 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:50.941 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.941 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.941 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:50.941 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186943+0000", 2024-02-05T04:53:50.941 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:50.941 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.941 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.942 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:50.942 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186943+0000", 2024-02-05T04:53:50.942 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.1899999999999999e-07 2024-02-05T04:53:50.942 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.942 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.942 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:50.942 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186944+0000", 2024-02-05T04:53:50.942 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.207e-06 2024-02-05T04:53:50.942 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.942 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.942 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:50.942 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186948+0000", 2024-02-05T04:53:50.942 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.7129999999999999e-06 2024-02-05T04:53:50.942 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.942 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.943 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:50.943 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186951+0000", 2024-02-05T04:53:50.943 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.9979999999999999e-06 2024-02-05T04:53:50.943 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.943 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.943 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:50.943 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.191572+0000", 2024-02-05T04:53:50.943 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0046207669999999996 2024-02-05T04:53:50.943 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.943 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.943 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:53:50.943 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.191630+0000", 2024-02-05T04:53:50.943 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.8774999999999999e-05 2024-02-05T04:53:50.943 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.943 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.943 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:50.944 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.192049+0000", 2024-02-05T04:53:50.944 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00041888 2024-02-05T04:53:50.944 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.944 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.944 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:50.944 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.194188+0000", 2024-02-05T04:53:50.944 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0021391359999999998 2024-02-05T04:53:50.944 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.944 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.944 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-02-05T04:53:50.944 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.198284+0000", 2024-02-05T04:53:50.944 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0040960509999999999 2024-02-05T04:53:50.944 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.944 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.944 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:50.945 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.198341+0000", 2024-02-05T04:53:50.945 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.6737999999999998e-05 2024-02-05T04:53:50.945 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:50.945 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:50.945 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:50.945 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.945 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.945 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.2s1 37/31 ECSubWrite(tid=19, reqid=client.4634.0:251, at_version=37'11, trim_to=0'0, roll_forward_to=34'8))", 2024-02-05T04:53:50.945 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:29.200609+0000", 2024-02-05T04:53:50.945 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 21.695477644, 2024-02-05T04:53:50.945 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0073060429999999999, 2024-02-05T04:53:50.945 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:50.945 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:53:50.945 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:50.945 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.945 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:50.945 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.200609+0000", 2024-02-05T04:53:50.946 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:50.946 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.946 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.946 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:50.946 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.200609+0000", 2024-02-05T04:53:50.946 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:50.946 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.946 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.946 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:50.946 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.200609+0000", 2024-02-05T04:53:50.946 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.9599999999999999e-07 2024-02-05T04:53:50.946 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.946 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.946 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:50.946 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.200622+0000", 2024-02-05T04:53:50.947 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2833000000000001e-05 2024-02-05T04:53:50.947 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.947 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.947 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:50.947 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.200633+0000", 2024-02-05T04:53:50.947 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0509999999999999e-05 2024-02-05T04:53:50.947 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.947 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.947 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:50.947 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.200640+0000", 2024-02-05T04:53:50.947 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.2150000000000004e-06 2024-02-05T04:53:50.947 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.947 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.947 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:50.947 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.200685+0000", 2024-02-05T04:53:50.947 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.5571000000000001e-05 2024-02-05T04:53:50.948 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.948 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.948 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:50.948 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.200691+0000", 2024-02-05T04:53:50.948 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.5990000000000001e-06 2024-02-05T04:53:50.948 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.948 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.948 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:50.948 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.207875+0000", 2024-02-05T04:53:50.948 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0071843130000000003 2024-02-05T04:53:50.948 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.948 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.948 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:50.948 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.207915+0000", 2024-02-05T04:53:50.948 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.9505999999999997e-05 2024-02-05T04:53:50.949 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:50.949 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:50.949 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:50.949 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.949 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.949 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.2s1 37/31 ECSubWrite(tid=20, reqid=client.4634.0:252, at_version=37'12, trim_to=0'0, roll_forward_to=34'8))", 2024-02-05T04:53:50.949 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:29.201260+0000", 2024-02-05T04:53:50.949 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 21.694826633000002, 2024-02-05T04:53:50.949 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0066905130000000004, 2024-02-05T04:53:50.949 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:50.949 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:53:50.949 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:50.949 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.949 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:50.949 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.201260+0000", 2024-02-05T04:53:50.950 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:50.950 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.950 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.950 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:50.950 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.201260+0000", 2024-02-05T04:53:50.950 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:50.950 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.950 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.950 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:50.950 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.201260+0000", 2024-02-05T04:53:50.950 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.2500000000000005e-07 2024-02-05T04:53:50.950 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.950 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.950 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:50.950 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.201272+0000", 2024-02-05T04:53:50.950 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1105e-05 2024-02-05T04:53:50.951 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.951 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.951 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:50.951 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.201282+0000", 2024-02-05T04:53:50.951 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0572e-05 2024-02-05T04:53:50.951 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.951 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.951 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:50.951 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.201288+0000", 2024-02-05T04:53:50.951 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.9699999999999996e-06 2024-02-05T04:53:50.951 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.951 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.951 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:50.951 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.202134+0000", 2024-02-05T04:53:50.951 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00084635299999999999 2024-02-05T04:53:50.951 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.952 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.952 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:50.952 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.202140+0000", 2024-02-05T04:53:50.952 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.054e-06 2024-02-05T04:53:50.952 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.952 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.952 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:50.952 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.207923+0000", 2024-02-05T04:53:50.952 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0057820989999999997 2024-02-05T04:53:50.952 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.952 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.952 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:50.952 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.207950+0000", 2024-02-05T04:53:50.952 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.7735000000000001e-05 2024-02-05T04:53:50.952 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:50.953 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:50.953 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:50.953 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.953 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.953 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.4634.0:382 3.es0 3:7f0ef6bd:::smithi19639362-19:head [append 0~655360] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-02-05T04:53:50.953 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:34.167434+0000", 2024-02-05T04:53:50.953 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 16.728652459999999, 2024-02-05T04:53:50.953 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0077709989999999998, 2024-02-05T04:53:50.953 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:50.953 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:50.953 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-02-05T04:53:50.953 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.4634", 2024-02-05T04:53:50.953 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:50.953 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 382 2024-02-05T04:53:50.953 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.954 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:50.954 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.954 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:50.954 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.167434+0000", 2024-02-05T04:53:50.954 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:50.954 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.954 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.954 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:50.954 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.167434+0000", 2024-02-05T04:53:50.954 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:50.954 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.954 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.954 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:50.954 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.167436+0000", 2024-02-05T04:53:50.954 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.885e-06 2024-02-05T04:53:50.954 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.955 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.955 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:50.955 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.168595+0000", 2024-02-05T04:53:50.955 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0011590350000000001 2024-02-05T04:53:50.955 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.955 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.955 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:50.955 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.168610+0000", 2024-02-05T04:53:50.955 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.4778e-05 2024-02-05T04:53:50.955 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.955 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.955 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:50.955 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.168700+0000", 2024-02-05T04:53:50.955 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.0462000000000002e-05 2024-02-05T04:53:50.955 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.955 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.956 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:50.956 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.168791+0000", 2024-02-05T04:53:50.956 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.1015999999999997e-05 2024-02-05T04:53:50.956 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.956 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.956 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:53:50.956 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.168911+0000", 2024-02-05T04:53:50.956 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000119864 2024-02-05T04:53:50.956 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.956 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.956 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:50.956 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.170527+0000", 2024-02-05T04:53:50.956 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0016163589999999999 2024-02-05T04:53:50.956 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.956 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.957 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:50.957 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.174753+0000", 2024-02-05T04:53:50.957 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0042254429999999997 2024-02-05T04:53:50.957 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.957 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.957 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-02-05T04:53:50.957 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.175115+0000", 2024-02-05T04:53:50.957 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00036262499999999999 2024-02-05T04:53:50.957 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.957 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.957 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:50.957 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.175205+0000", 2024-02-05T04:53:50.957 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.9531999999999996e-05 2024-02-05T04:53:50.957 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:50.957 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:50.957 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:50.958 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.958 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.958 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.4634.0:385 3.es0 3:7f0ef6bd:::smithi19639362-19:head [setxattr _header (56) in=7b] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-02-05T04:53:50.958 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:34.171897+0000", 2024-02-05T04:53:50.958 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 16.724189682999999, 2024-02-05T04:53:50.958 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.005569114, 2024-02-05T04:53:50.958 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:50.958 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:50.958 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-02-05T04:53:50.958 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.4634", 2024-02-05T04:53:50.958 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:50.958 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 385 2024-02-05T04:53:50.958 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.958 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:50.958 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.958 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:50.959 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.171897+0000", 2024-02-05T04:53:50.959 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:50.959 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.959 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.959 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:50.959 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.171897+0000", 2024-02-05T04:53:50.959 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:50.959 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.959 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.959 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:50.959 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.171898+0000", 2024-02-05T04:53:50.959 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.6899999999999996e-07 2024-02-05T04:53:50.959 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.959 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.959 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:50.959 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.171901+0000", 2024-02-05T04:53:50.960 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.7699999999999999e-06 2024-02-05T04:53:50.960 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.960 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.960 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:50.960 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.171908+0000", 2024-02-05T04:53:50.960 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.1399999999999997e-06 2024-02-05T04:53:50.960 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.960 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.960 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:50.960 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.171913+0000", 2024-02-05T04:53:50.960 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.0509999999999996e-06 2024-02-05T04:53:50.960 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.960 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.960 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:50.960 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.175805+0000", 2024-02-05T04:53:50.961 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.003892674 2024-02-05T04:53:50.961 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.961 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.961 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:53:50.961 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.175908+0000", 2024-02-05T04:53:50.961 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000102518 2024-02-05T04:53:50.961 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.961 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.961 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:50.961 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.176312+0000", 2024-02-05T04:53:50.961 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00040448399999999999 2024-02-05T04:53:50.961 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.961 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.961 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:50.961 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.176961+0000", 2024-02-05T04:53:50.961 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00064888599999999999 2024-02-05T04:53:50.962 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.962 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.962 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-02-05T04:53:50.962 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.177414+0000", 2024-02-05T04:53:50.962 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000453292 2024-02-05T04:53:50.962 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.962 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.962 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:50.962 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.177466+0000", 2024-02-05T04:53:50.962 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.1430000000000001e-05 2024-02-05T04:53:50.962 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:50.962 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:50.962 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:50.962 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.962 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.962 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.4634.0:386 3.es0 3:7f0ef6bd:::smithi19639362-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-02-05T04:53:50.963 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:34.171921+0000", 2024-02-05T04:53:50.963 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 16.724165375999998, 2024-02-05T04:53:50.963 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0068862569999999998, 2024-02-05T04:53:50.963 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:50.963 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "started", 2024-02-05T04:53:50.963 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-02-05T04:53:50.963 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.4634", 2024-02-05T04:53:50.963 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:50.963 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 386 2024-02-05T04:53:50.963 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.963 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:50.963 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.963 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:50.963 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.171921+0000", 2024-02-05T04:53:50.963 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:50.964 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.964 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.964 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:50.964 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.171921+0000", 2024-02-05T04:53:50.964 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:50.964 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.964 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.964 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:50.964 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.171922+0000", 2024-02-05T04:53:50.964 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.7399999999999998e-07 2024-02-05T04:53:50.964 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.964 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.964 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:50.964 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.171924+0000", 2024-02-05T04:53:50.964 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.1900000000000002e-06 2024-02-05T04:53:50.964 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.965 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.965 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:50.965 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.171930+0000", 2024-02-05T04:53:50.965 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.207e-06 2024-02-05T04:53:50.965 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.965 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.965 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:50.965 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.171936+0000", 2024-02-05T04:53:50.965 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.0630000000000002e-06 2024-02-05T04:53:50.965 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.965 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.965 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:50.965 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.176614+0000", 2024-02-05T04:53:50.965 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0046776270000000002 2024-02-05T04:53:50.965 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.966 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.966 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for rw locks", 2024-02-05T04:53:50.966 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.176696+0000", 2024-02-05T04:53:50.966 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.2084999999999998e-05 2024-02-05T04:53:50.966 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.966 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.966 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:50.966 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.177767+0000", 2024-02-05T04:53:50.966 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.001070809 2024-02-05T04:53:50.966 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.966 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.966 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:53:50.966 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.177813+0000", 2024-02-05T04:53:50.966 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.6635999999999999e-05 2024-02-05T04:53:50.966 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.966 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.967 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:50.967 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.178807+0000", 2024-02-05T04:53:50.967 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0009941660000000001 2024-02-05T04:53:50.967 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:50.967 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:50.967 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:50.967 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.967 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.967 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.8s2 43/31 ECSubWrite(tid=64, reqid=osd.4.0:62, at_version=0'0, trim_to=0'0, roll_forward_to=43'19))", 2024-02-05T04:53:50.967 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:35.181070+0000", 2024-02-05T04:53:50.967 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 15.715015978, 2024-02-05T04:53:50.967 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0062151699999999999, 2024-02-05T04:53:50.967 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:50.967 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:53:50.967 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:50.967 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.968 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:50.968 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.181070+0000", 2024-02-05T04:53:50.968 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:50.968 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.968 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.968 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:50.968 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.181070+0000", 2024-02-05T04:53:50.968 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:50.968 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.968 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.968 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:50.968 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.181071+0000", 2024-02-05T04:53:50.968 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.85e-07 2024-02-05T04:53:50.968 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.968 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.969 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:50.969 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.181080+0000", 2024-02-05T04:53:50.969 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.7120000000000006e-06 2024-02-05T04:53:50.969 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.969 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.969 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:50.969 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.181093+0000", 2024-02-05T04:53:50.969 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2938e-05 2024-02-05T04:53:50.969 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.969 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.969 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:50.969 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.181101+0000", 2024-02-05T04:53:50.969 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.1629999999999996e-06 2024-02-05T04:53:50.969 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.969 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.969 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:50.970 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.181211+0000", 2024-02-05T04:53:50.970 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00010961800000000001 2024-02-05T04:53:50.970 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.970 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.970 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:50.970 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.181220+0000", 2024-02-05T04:53:50.970 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.7059999999999999e-06 2024-02-05T04:53:50.970 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.970 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.970 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:50.970 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.187235+0000", 2024-02-05T04:53:50.970 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0060150100000000003 2024-02-05T04:53:50.970 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.970 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.970 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:50.971 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.187286+0000", 2024-02-05T04:53:50.971 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.0238000000000002e-05 2024-02-05T04:53:50.971 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:50.971 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:50.971 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:50.971 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.971 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.971 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.fs1 57/31 ECSubWrite(tid=92, reqid=osd.11.0:92, at_version=57'45, trim_to=0'0, roll_forward_to=54'42))", 2024-02-05T04:53:50.971 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:48.951103+0000", 2024-02-05T04:53:50.971 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 1.944983881, 2024-02-05T04:53:50.971 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0056058180000000003, 2024-02-05T04:53:50.971 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:50.971 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:53:50.971 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:50.971 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.971 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:50.972 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.951103+0000", 2024-02-05T04:53:50.972 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:50.972 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.972 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.972 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:50.972 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.951103+0000", 2024-02-05T04:53:50.972 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:50.972 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.972 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.972 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:50.972 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.951103+0000", 2024-02-05T04:53:50.972 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.6499999999999998e-07 2024-02-05T04:53:50.972 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.972 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.972 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:50.972 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.951130+0000", 2024-02-05T04:53:50.973 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.6749000000000001e-05 2024-02-05T04:53:50.973 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.973 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.973 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:50.973 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.951145+0000", 2024-02-05T04:53:50.973 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.5211000000000001e-05 2024-02-05T04:53:50.973 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.973 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.973 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:50.973 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.951154+0000", 2024-02-05T04:53:50.973 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.1009999999999998e-06 2024-02-05T04:53:50.973 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.973 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.973 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:50.973 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.951210+0000", 2024-02-05T04:53:50.974 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.5497000000000001e-05 2024-02-05T04:53:50.974 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.974 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.974 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:50.974 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.951222+0000", 2024-02-05T04:53:50.974 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1668000000000001e-05 2024-02-05T04:53:50.974 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.974 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.974 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:50.974 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.956619+0000", 2024-02-05T04:53:50.974 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0053975960000000002 2024-02-05T04:53:50.974 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.974 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.974 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:50.974 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.956708+0000", 2024-02-05T04:53:50.974 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.9130999999999998e-05 2024-02-05T04:53:50.975 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:50.975 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:50.975 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:50.975 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.975 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.975 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.3s4 57/31 ECSubWrite(tid=76, reqid=osd.4.0:76, at_version=57'35, trim_to=0'0, roll_forward_to=47'33))", 2024-02-05T04:53:50.975 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:48.952725+0000", 2024-02-05T04:53:50.975 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 1.943361677, 2024-02-05T04:53:50.975 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.007485312, 2024-02-05T04:53:50.975 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:50.975 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:53:50.975 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:50.975 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.975 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:50.975 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.952725+0000", 2024-02-05T04:53:50.975 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:50.976 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.976 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.976 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:50.976 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.952725+0000", 2024-02-05T04:53:50.976 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:50.976 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.976 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.976 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:50.976 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.952726+0000", 2024-02-05T04:53:50.976 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.7100000000000011e-07 2024-02-05T04:53:50.976 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.976 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.976 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:50.976 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.952751+0000", 2024-02-05T04:53:50.976 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.5477e-05 2024-02-05T04:53:50.977 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.977 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.977 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:50.977 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.952765+0000", 2024-02-05T04:53:50.977 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.3502e-05 2024-02-05T04:53:50.977 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.977 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.977 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:50.977 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.952773+0000", 2024-02-05T04:53:50.977 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.1499999999999999e-06 2024-02-05T04:53:50.977 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.977 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.977 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:50.977 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.952832+0000", 2024-02-05T04:53:50.977 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.9095000000000003e-05 2024-02-05T04:53:50.977 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.978 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.978 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:50.978 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.952839+0000", 2024-02-05T04:53:50.978 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.764e-06 2024-02-05T04:53:50.978 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.978 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.978 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:50.978 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.960160+0000", 2024-02-05T04:53:50.978 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0073216820000000004 2024-02-05T04:53:50.978 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.978 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.978 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:50.978 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.960210+0000", 2024-02-05T04:53:50.978 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.9670999999999999e-05 2024-02-05T04:53:50.978 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:50.978 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:50.979 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:50.979 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.979 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.979 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.fs1 58/31 ECSubWrite(tid=104, reqid=client.4634.0:518, at_version=58'51, trim_to=0'0, roll_forward_to=57'49))", 2024-02-05T04:53:50.979 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:49.974036+0000", 2024-02-05T04:53:50.979 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 0.92205051699999996, 2024-02-05T04:53:50.979 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0089249829999999992, 2024-02-05T04:53:50.979 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:53:50.979 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:53:50.979 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:53:50.979 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.979 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:53:50.979 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.974036+0000", 2024-02-05T04:53:50.979 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:50.979 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.979 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.980 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:53:50.980 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.974036+0000", 2024-02-05T04:53:50.980 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:53:50.980 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.980 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.980 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:53:50.980 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.974037+0000", 2024-02-05T04:53:50.980 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.0999999999999997e-07 2024-02-05T04:53:50.980 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.980 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.980 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:53:50.980 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.976765+0000", 2024-02-05T04:53:50.980 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0027280059999999998 2024-02-05T04:53:50.980 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.980 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.981 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:53:50.981 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.976780+0000", 2024-02-05T04:53:50.981 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.5238999999999999e-05 2024-02-05T04:53:50.981 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.981 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.981 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:53:50.981 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.976789+0000", 2024-02-05T04:53:50.981 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.8759999999999994e-06 2024-02-05T04:53:50.981 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.981 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.981 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:53:50.981 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.976857+0000", 2024-02-05T04:53:50.981 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.7984999999999994e-05 2024-02-05T04:53:50.981 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.981 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.981 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:53:50.982 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.976869+0000", 2024-02-05T04:53:50.982 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1579999999999999e-05 2024-02-05T04:53:50.982 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.982 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.982 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:53:50.982 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.982917+0000", 2024-02-05T04:53:50.982 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0060485069999999998 2024-02-05T04:53:50.982 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:53:50.982 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:53:50.982 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:53:50.982 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.982961+0000", 2024-02-05T04:53:50.982 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.388e-05 2024-02-05T04:53:50.982 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:50.982 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:50.982 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:50.982 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:53:50.983 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:53:50.983 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-02-05T04:53:50.983 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 12, 13, 14, 15] out_osds: [8] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-02-05T04:53:50.983 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T04:53:50.983 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2024-02-05T04:53:50.984 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-05T04:53:50.985 INFO:tasks.rados.rados.0.smithi196.stdout:204: done (4 left) 2024-02-05T04:53:50.985 INFO:tasks.rados.rados.0.smithi196.stdout:205: done (3 left) 2024-02-05T04:53:50.985 INFO:tasks.rados.rados.0.smithi196.stdout:208: done (2 left) 2024-02-05T04:53:50.985 INFO:tasks.rados.rados.0.smithi196.stdout:209: done (1 left) 2024-02-05T04:53:50.985 INFO:tasks.rados.rados.0.smithi196.stdout:212: done (0 left) 2024-02-05T04:53:50.985 INFO:tasks.rados.rados.0.smithi196.stdout:213: snap_create 2024-02-05T04:53:50.986 INFO:tasks.ceph.osd.15.smithi196.stderr:2024-02-05T04:53:50.983+0000 7f1facc45640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 34611) UID: 0 2024-02-05T04:53:50.986 INFO:teuthology.orchestra.run.smithi062.stderr:noscrub is unset 2024-02-05T04:53:50.987 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-02-05T04:53:51.085 INFO:tasks.ceph.osd.0.smithi062.stderr:2024-02-05T04:53:51.084+0000 7fece8e03640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34688) UID: 0 2024-02-05T04:53:51.113 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T04:53:51.133 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-02-05T04:53:51.134 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [], 2024-02-05T04:53:51.134 INFO:teuthology.orchestra.run.smithi150.stdout: "num_ops": 0 2024-02-05T04:53:51.134 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-02-05T04:53:51.142 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-02-05T04:53:51.187 INFO:tasks.ceph.osd.14.smithi150.stderr:2024-02-05T04:53:51.186+0000 7f5f81083640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 34020) UID: 0 2024-02-05T04:53:51.240 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0: {} 2024-02-05T04:53:51.241 INFO:teuthology.orchestra.run.smithi062.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T04:53:51.248 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-02-05T04:53:51.248 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [], 2024-02-05T04:53:51.248 INFO:teuthology.orchestra.run.smithi150.stdout: "complaint_time": 30, 2024-02-05T04:53:51.248 INFO:teuthology.orchestra.run.smithi150.stdout: "num_blocked_ops": 0 2024-02-05T04:53:51.248 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-02-05T04:53:51.257 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-02-05T04:53:51.260 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1: {} 2024-02-05T04:53:51.260 INFO:teuthology.orchestra.run.smithi062.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T04:53:51.277 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2: {} 2024-02-05T04:53:51.278 INFO:teuthology.orchestra.run.smithi062.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T04:53:51.296 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3: {} 2024-02-05T04:53:51.296 INFO:teuthology.orchestra.run.smithi062.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T04:53:51.298 INFO:tasks.ceph.osd.10.smithi150.stderr:2024-02-05T04:53:51.297+0000 7f3075a4d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 34018) UID: 0 2024-02-05T04:53:51.314 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4: {} 2024-02-05T04:53:51.315 INFO:teuthology.orchestra.run.smithi062.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T04:53:51.332 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5: {} 2024-02-05T04:53:51.332 INFO:teuthology.orchestra.run.smithi062.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T04:53:51.350 INFO:teuthology.orchestra.run.smithi062.stdout:osd.6: {} 2024-02-05T04:53:51.350 INFO:teuthology.orchestra.run.smithi062.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T04:53:51.353 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:53:51.353 INFO:teuthology.orchestra.run.smithi062.stderr:dumped all 2024-02-05T04:53:51.364 INFO:teuthology.orchestra.run.smithi062.stdout:{"pg_ready":true,"pg_map":{"version":68,"stamp":"2024-02-05T04:53:50.943782+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":68256288,"num_objects":71,"num_object_clones":24,"num_object_copies":487,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":4,"num_read":395,"num_read_kb":49766,"num_write":609,"num_write_kb":77300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":10179104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":649,"ondisk_log_size":649,"up":130,"acting":129,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":124,"num_osds":15,"num_per_pool_osds":15,"num_per_pool_omap_osds":15,"kb":1415577600,"kb_used":550968,"kb_used_data":128984,"kb_used_omap":23,"kb_used_meta":421736,"kb_avail":1415026632,"statfs":{"total":1449551462400,"available":1448987271168,"internally_reserved":0,"allocated":132079616,"data_stored":152878684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23925,"internal_metadata":431858315},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3358720,"num_objects":4,"num_object_clones":4,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":2,"num_read":2,"num_read_kb":1,"num_write":30,"num_write_kb":3280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":10179104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.986737"},"pg_stats":[{"pgid":"3.f","version":"54'42","reported_seq":149,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-05T04:53:46.965185+0000","last_change":"2024-02-05T04:53:43.962429+0000","last_active":"2024-02-05T04:53:46.965185+0000","last_peered":"2024-02-05T04:53:46.965185+0000","last_clean":"2024-02-05T04:53:46.965185+0000","last_became_active":"2024-02-05T04:53:24.723119+0000","last_became_peered":"2024-02-05T04:53:24.723119+0000","last_unstale":"2024-02-05T04:53:46.965185+0000","last_undegraded":"2024-02-05T04:53:46.965185+0000","last_fullsized":"2024-02-05T04:53:46.965185+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:23.066897+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:33.772536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0095848140000000005,"stat_sum":{"num_bytes":7536640,"num_objects":10,"num_object_clones":3,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":36,"num_read_kb":5806,"num_write":30,"num_write_kb":7856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,14,4,0,10,1],"acting":[11,2,14,4,0,10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"4","length":"3"},{"start":"a","length":"1"},{"start":"c","length":"1"}]},{"pgid":"3.e","version":"52'48","reported_seq":163,"reported_epoch":56,"state":"active+undersized+remapped","last_fresh":"2024-02-05T04:53:48.017209+0000","last_change":"2024-02-05T04:53:48.017560+0000","last_active":"2024-02-05T04:53:48.017209+0000","last_peered":"2024-02-05T04:53:48.017209+0000","last_clean":"2024-02-05T04:53:44.969493+0000","last_became_active":"2024-02-05T04:53:47.968460+0000","last_became_peered":"2024-02-05T04:53:47.968460+0000","last_unstale":"2024-02-05T04:53:48.017209+0000","last_undegraded":"2024-02-05T04:53:48.017209+0000","last_fullsized":"2024-02-05T04:53:47.959165+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:23.066897+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0062682679999999996,"stat_sum":{"num_bytes":3457024,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":11,"num_read_kb":6,"num_write":46,"num_write_kb":5520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":6209536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7,11,13,6,5],"acting":[2,4,2147483647,11,13,6,5],"avail_no_missing":["2(0)","4(1)","5(6)","6(5)","7(2)","8(2)","11(3)","13(4)"],"object_location_counts":[{"shards":"2(0),4(1),5(6),6(5),11(3),13(4)","objects":4}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"4","length":"3"},{"start":"a","length":"1"},{"start":"c","length":"1"}]},{"pgid":"3.d","version":"56'30","reported_seq":131,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-05T04:53:47.957424+0000","last_change":"2024-02-05T04:53:43.951019+0000","last_active":"2024-02-05T04:53:47.957424+0000","last_peered":"2024-02-05T04:53:47.957424+0000","last_clean":"2024-02-05T04:53:47.957424+0000","last_became_active":"2024-02-05T04:53:24.723128+0000","last_became_peered":"2024-02-05T04:53:24.723128+0000","last_unstale":"2024-02-05T04:53:47.957424+0000","last_undegraded":"2024-02-05T04:53:47.957424+0000","last_fullsized":"2024-02-05T04:53:47.957424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:23.066897+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:52.836844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042206399999999999,"stat_sum":{"num_bytes":6062080,"num_objects":6,"num_object_clones":2,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":20,"num_read_kb":3753,"num_write":40,"num_write_kb":6320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,14,10,9,13,5],"acting":[7,3,14,10,9,13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"4","length":"3"},{"start":"a","length":"1"},{"start":"c","length":"1"}]},{"pgid":"3.c","version":"52'11","reported_seq":104,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-05T04:53:46.965039+0000","last_change":"2024-02-05T04:53:43.959798+0000","last_active":"2024-02-05T04:53:46.965039+0000","last_peered":"2024-02-05T04:53:46.965039+0000","last_clean":"2024-02-05T04:53:46.965039+0000","last_became_active":"2024-02-05T04:53:24.723258+0000","last_became_peered":"2024-02-05T04:53:24.723258+0000","last_unstale":"2024-02-05T04:53:46.965039+0000","last_undegraded":"2024-02-05T04:53:46.965039+0000","last_fullsized":"2024-02-05T04:53:46.965039+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:23.066897+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:50.013185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0070675490000000002,"stat_sum":{"num_bytes":1736704,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":195,"num_write":6,"num_write_kb":1696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,13,5,10,6,1,9],"acting":[11,13,5,10,6,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"4","length":"3"},{"start":"a","length":"1"},{"start":"c","length":"1"}]},{"pgid":"3.b","version":"52'50","reported_seq":160,"reported_epoch":54,"state":"remapped+peering","last_fresh":"2024-02-05T04:53:45.958015+0000","last_change":"2024-02-05T04:53:46.961951+0000","last_active":"2024-02-05T04:53:45.958015+0000","last_peered":"2024-02-05T04:53:44.972625+0000","last_clean":"2024-02-05T04:53:44.972625+0000","last_became_active":"2024-02-05T04:53:24.486317+0000","last_became_peered":"2024-02-05T04:53:24.486317+0000","last_unstale":"2024-02-05T04:53:45.958015+0000","last_undegraded":"2024-02-05T04:53:45.958015+0000","last_fullsized":"2024-02-05T04:53:45.958015+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:23.066897+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0076553280000000003,"stat_sum":{"num_bytes":6848512,"num_objects":7,"num_object_clones":2,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":8849,"num_write":31,"num_write_kb":9072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,1,10,0,6,7,12],"acting":[13,1,2147483647,0,6,7,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,6,7,8,10,12],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"4","length":"3"},{"start":"a","length":"1"},{"start":"c","length":"1"}]},{"pgid":"3.a","version":"0'0","reported_seq":84,"reported_epoch":53,"state":"peering","last_fresh":"2024-02-05T04:53:45.266982+0000","last_change":"2024-02-05T04:53:45.958080+0000","last_active":"2024-02-05T04:53:45.958080+0000","last_peered":"2024-02-05T04:53:45.266982+0000","last_clean":"2024-02-05T04:53:45.266982+0000","last_became_active":"2024-02-05T04:53:24.488548+0000","last_became_peered":"2024-02-05T04:53:24.488548+0000","last_unstale":"2024-02-05T04:53:45.266982+0000","last_undegraded":"2024-02-05T04:53:45.266982+0000","last_fullsized":"2024-02-05T04:53:45.266982+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:23.066897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051381799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,10,9,14,6,11],"acting":[3,7,10,9,14,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6,7,8,9,10,11,14],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"4","length":"3"},{"start":"a","length":"1"},{"start":"c","length":"1"}]},{"pgid":"3.9","version":"52'19","reported_seq":118,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-05T04:53:46.963709+0000","last_change":"2024-02-05T04:53:43.962732+0000","last_active":"2024-02-05T04:53:46.963709+0000","last_peered":"2024-02-05T04:53:46.963709+0000","last_clean":"2024-02-05T04:53:46.963709+0000","last_became_active":"2024-02-05T04:53:24.722954+0000","last_became_peered":"2024-02-05T04:53:24.722954+0000","last_unstale":"2024-02-05T04:53:46.963709+0000","last_undegraded":"2024-02-05T04:53:46.963709+0000","last_fullsized":"2024-02-05T04:53:46.963709+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:23.066897+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:47.850959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0067263540000000004,"stat_sum":{"num_bytes":2424832,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":7286,"num_write":13,"num_write_kb":3152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0,13,3,15,9,10],"acting":[14,0,13,3,15,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"4","length":"3"},{"start":"a","length":"1"},{"start":"c","length":"1"}]},{"pgid":"3.8","version":"45'23","reported_seq":112,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-02-05T04:53:45.959881+0000","last_change":"2024-02-05T04:53:43.952986+0000","last_active":"2024-02-05T04:53:45.959881+0000","last_peered":"2024-02-05T04:53:45.959881+0000","last_clean":"2024-02-05T04:53:45.959881+0000","last_became_active":"2024-02-05T04:53:24.486354+0000","last_became_peered":"2024-02-05T04:53:24.486354+0000","last_unstale":"2024-02-05T04:53:45.959881+0000","last_undegraded":"2024-02-05T04:53:45.959881+0000","last_fullsized":"2024-02-05T04:53:45.959881+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:23.066897+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:32.400061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041662200000000001,"stat_sum":{"num_bytes":3211264,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":10,"num_read_kb":244,"num_write":11,"num_write_kb":3136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,0,14,15,6],"acting":[4,1,2,0,14,15,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"4","length":"3"},{"start":"a","length":"1"},{"start":"c","length":"1"}]},{"pgid":"1.0","version":"26'157","reported_seq":222,"reported_epoch":56,"state":"active+undersized+remapped","last_fresh":"2024-02-05T04:53:47.979843+0000","last_change":"2024-02-05T04:53:47.979843+0000","last_active":"2024-02-05T04:53:47.979843+0000","last_peered":"2024-02-05T04:53:47.979843+0000","last_clean":"2024-02-05T04:53:11.949771+0000","last_became_active":"2024-02-05T04:53:47.968369+0000","last_became_peered":"2024-02-05T04:53:47.968369+0000","last_unstale":"2024-02-05T04:53:47.979843+0000","last_undegraded":"2024-02-05T04:53:47.979843+0000","last_fullsized":"2024-02-05T04:53:47.959276+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:07.900741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:07.900741+0000","last_clean_scrub_stamp":"2024-02-05T04:53:07.900741+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2],"avail_no_missing":["2","3","8"],"object_location_counts":[{"shards":"2","objects":2}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":35,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-05T04:53:44.970365+0000","last_change":"2024-02-05T04:53:12.028466+0000","last_active":"2024-02-05T04:53:44.970365+0000","last_peered":"2024-02-05T04:53:44.970365+0000","last_clean":"2024-02-05T04:53:44.970365+0000","last_became_active":"2024-02-05T04:53:12.028138+0000","last_became_peered":"2024-02-05T04:53:12.028138+0000","last_unstale":"2024-02-05T04:53:44.970365+0000","last_undegraded":"2024-02-05T04:53:44.970365+0000","last_fullsized":"2024-02-05T04:53:44.970365+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_clean_scrub_stamp":"2024-02-05T04:53:11.017155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:38.427375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,15],"acting":[4,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"56'46","reported_seq":169,"reported_epoch":56,"state":"active+undersized+remapped","last_fresh":"2024-02-05T04:53:48.050494+0000","last_change":"2024-02-05T04:53:48.050895+0000","last_active":"2024-02-05T04:53:48.050494+0000","last_peered":"2024-02-05T04:53:48.050494+0000","last_clean":"2024-02-05T04:53:45.266460+0000","last_became_active":"2024-02-05T04:53:47.975370+0000","last_became_peered":"2024-02-05T04:53:47.975370+0000","last_unstale":"2024-02-05T04:53:48.050494+0000","last_undegraded":"2024-02-05T04:53:48.050494+0000","last_fullsized":"2024-02-05T04:53:47.959878+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:23.066897+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050862900000000003,"stat_sum":{"num_bytes":3440640,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":23,"num_read_kb":4936,"num_write":41,"num_write_kb":3408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":3440640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,2,3,6,15,10,11],"acting":[12,2,3,6,15,2147483647,11],"avail_no_missing":["12(0)","2(1)","3(2)","6(3)","8(5)","10(5)","11(6)","15(4)"],"object_location_counts":[{"shards":"2(1),3(2),6(3),11(6),12(0),15(4)","objects":4}],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"4","length":"3"},{"start":"a","length":"1"},{"start":"c","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":35,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-05T04:53:45.266811+0000","last_change":"2024-02-05T04:53:12.040500+0000","last_active":"2024-02-05T04:53:45.266811+0000","last_peered":"2024-02-05T04:53:45.266811+0000","last_clean":"2024-02-05T04:53:45.266811+0000","last_became_active":"2024-02-05T04:53:12.040297+0000","last_became_peered":"2024-02-05T04:53:12.040297+0000","last_unstale":"2024-02-05T04:53:45.266811+0000","last_undegraded":"2024-02-05T04:53:45.266811+0000","last_fullsized":"2024-02-05T04:53:45.266811+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_clean_scrub_stamp":"2024-02-05T04:53:11.017155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:20.053075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"52'45","reported_seq":133,"reported_epoch":54,"state":"remapped+peering","last_fresh":"2024-02-05T04:53:45.957149+0000","last_change":"2024-02-05T04:53:46.960997+0000","last_active":"2024-02-05T04:53:45.957149+0000","last_peered":"2024-02-05T04:53:44.972403+0000","last_clean":"2024-02-05T04:53:44.972403+0000","last_became_active":"2024-02-05T04:53:24.723508+0000","last_became_peered":"2024-02-05T04:53:24.723508+0000","last_unstale":"2024-02-05T04:53:45.957149+0000","last_undegraded":"2024-02-05T04:53:45.957149+0000","last_fullsized":"2024-02-05T04:53:45.957149+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:23.066897+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.017274099000000001,"stat_sum":{"num_bytes":3391488,"num_objects":5,"num_object_clones":2,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":2166,"num_write":21,"num_write_kb":3664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,11,4,10,15,2,14],"acting":[13,11,4,2147483647,2147483647,2,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,4,8,10,11,14,15],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"4","length":"3"},{"start":"a","length":"1"},{"start":"c","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":3,"reported_epoch":54,"state":"peering","last_fresh":"2024-02-05T04:53:45.960501+0000","last_change":"2024-02-05T04:53:45.960501+0000","last_active":"2024-02-05T04:53:11.017155+0000","last_peered":"2024-02-05T04:53:11.017155+0000","last_clean":"2024-02-05T04:53:11.017155+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-05T04:53:45.960501+0000","last_undegraded":"2024-02-05T04:53:45.960501+0000","last_fullsized":"2024-02-05T04:53:45.960501+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_clean_scrub_stamp":"2024-02-05T04:53:11.017155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"34'5","reported_seq":93,"reported_epoch":54,"state":"remapped+peering","last_fresh":"2024-02-05T04:53:45.956967+0000","last_change":"2024-02-05T04:53:46.963500+0000","last_active":"2024-02-05T04:53:45.956967+0000","last_peered":"2024-02-05T04:53:45.266392+0000","last_clean":"2024-02-05T04:53:45.266392+0000","last_became_active":"2024-02-05T04:53:24.485082+0000","last_became_peered":"2024-02-05T04:53:24.485082+0000","last_unstale":"2024-02-05T04:53:45.956967+0000","last_undegraded":"2024-02-05T04:53:45.956967+0000","last_fullsized":"2024-02-05T04:53:45.956967+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:23.066897+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001232346,"stat_sum":{"num_bytes":114688,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":5,"num_write_kb":432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,2,15,0,12,6,9],"acting":[14,2,15,0,12,2147483647,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,2,6,8,9,12,15],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"4","length":"3"},{"start":"a","length":"1"},{"start":"c","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":35,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-05T04:53:45.267068+0000","last_change":"2024-02-05T04:53:12.035901+0000","last_active":"2024-02-05T04:53:45.267068+0000","last_peered":"2024-02-05T04:53:45.267068+0000","last_clean":"2024-02-05T04:53:45.267068+0000","last_became_active":"2024-02-05T04:53:12.035739+0000","last_became_peered":"2024-02-05T04:53:12.035739+0000","last_unstale":"2024-02-05T04:53:45.267068+0000","last_undegraded":"2024-02-05T04:53:45.267068+0000","last_fullsized":"2024-02-05T04:53:45.267068+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_clean_scrub_stamp":"2024-02-05T04:53:11.017155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:14.062467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9],"acting":[3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"47'33","reported_seq":122,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-02-05T04:53:45.960477+0000","last_change":"2024-02-05T04:53:43.953126+0000","last_active":"2024-02-05T04:53:45.960477+0000","last_peered":"2024-02-05T04:53:45.960477+0000","last_clean":"2024-02-05T04:53:45.960477+0000","last_became_active":"2024-02-05T04:53:24.641280+0000","last_became_peered":"2024-02-05T04:53:24.641280+0000","last_unstale":"2024-02-05T04:53:45.960477+0000","last_undegraded":"2024-02-05T04:53:45.960477+0000","last_fullsized":"2024-02-05T04:53:45.960477+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:23.066897+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:44.315916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047976399999999998,"stat_sum":{"num_bytes":3260416,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":13,"num_read_kb":1878,"num_write":36,"num_write_kb":3424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1,15,2,5,3],"acting":[4,7,1,15,2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"4","length":"3"},{"start":"a","length":"1"},{"start":"c","length":"1"}]},{"pgid":"3.5","version":"52'32","reported_seq":122,"reported_epoch":54,"state":"remapped+peering","last_fresh":"2024-02-05T04:53:45.958357+0000","last_change":"2024-02-05T04:53:46.960573+0000","last_active":"2024-02-05T04:53:45.958357+0000","last_peered":"2024-02-05T04:53:45.267230+0000","last_clean":"2024-02-05T04:53:45.267230+0000","last_became_active":"2024-02-05T04:53:24.643048+0000","last_became_peered":"2024-02-05T04:53:24.643048+0000","last_unstale":"2024-02-05T04:53:45.958357+0000","last_undegraded":"2024-02-05T04:53:45.958357+0000","last_fullsized":"2024-02-05T04:53:45.958357+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:23.066897+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0085434910000000003,"stat_sum":{"num_bytes":5767168,"num_objects":5,"num_object_clones":2,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":14,"num_read_kb":1831,"num_write":18,"num_write_kb":5632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13,5,3,7,1,6],"acting":[15,13,5,2147483647,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,3,5,6,7,8,13],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"4","length":"3"},{"start":"a","length":"1"},{"start":"c","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":37,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-02-05T04:53:45.962076+0000","last_change":"2024-02-05T04:53:12.032853+0000","last_active":"2024-02-05T04:53:45.962076+0000","last_peered":"2024-02-05T04:53:45.962076+0000","last_clean":"2024-02-05T04:53:45.962076+0000","last_became_active":"2024-02-05T04:53:12.032664+0000","last_became_peered":"2024-02-05T04:53:12.032664+0000","last_unstale":"2024-02-05T04:53:45.962076+0000","last_undegraded":"2024-02-05T04:53:45.962076+0000","last_fullsized":"2024-02-05T04:53:45.962076+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_clean_scrub_stamp":"2024-02-05T04:53:11.017155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:14.638056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"53'57","reported_seq":156,"reported_epoch":54,"state":"remapped+peering","last_fresh":"2024-02-05T04:53:45.960701+0000","last_change":"2024-02-05T04:53:46.959022+0000","last_active":"2024-02-05T04:53:45.960701+0000","last_peered":"2024-02-05T04:53:44.980296+0000","last_clean":"2024-02-05T04:53:44.980296+0000","last_became_active":"2024-02-05T04:53:24.643568+0000","last_became_peered":"2024-02-05T04:53:24.643568+0000","last_unstale":"2024-02-05T04:53:45.960701+0000","last_undegraded":"2024-02-05T04:53:45.960701+0000","last_fullsized":"2024-02-05T04:53:45.960701+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:23.066897+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00472471,"stat_sum":{"num_bytes":13090816,"num_objects":9,"num_object_clones":4,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":25,"num_read_kb":4622,"num_write":59,"num_write_kb":12784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5,0,9,4,13],"acting":[1,2,5,2147483647,9,4,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,2,4,5,8,9,13],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"4","length":"3"},{"start":"a","length":"1"},{"start":"c","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":36,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-05T04:53:44.952103+0000","last_change":"2024-02-05T04:53:12.753233+0000","last_active":"2024-02-05T04:53:44.952103+0000","last_peered":"2024-02-05T04:53:44.952103+0000","last_clean":"2024-02-05T04:53:44.952103+0000","last_became_active":"2024-02-05T04:53:12.752915+0000","last_became_peered":"2024-02-05T04:53:12.752915+0000","last_unstale":"2024-02-05T04:53:44.952103+0000","last_undegraded":"2024-02-05T04:53:44.952103+0000","last_fullsized":"2024-02-05T04:53:44.952103+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_clean_scrub_stamp":"2024-02-05T04:53:11.017155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:18.391150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"52'25","reported_seq":113,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-05T04:53:46.961836+0000","last_change":"2024-02-05T04:53:43.962548+0000","last_active":"2024-02-05T04:53:46.961836+0000","last_peered":"2024-02-05T04:53:46.961836+0000","last_clean":"2024-02-05T04:53:46.961836+0000","last_became_active":"2024-02-05T04:53:24.641453+0000","last_became_peered":"2024-02-05T04:53:24.641453+0000","last_unstale":"2024-02-05T04:53:46.961836+0000","last_undegraded":"2024-02-05T04:53:46.961836+0000","last_fullsized":"2024-02-05T04:53:46.961836+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:23.066897+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:26.186961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0098146129999999998,"stat_sum":{"num_bytes":4128768,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":6,"num_read_kb":4,"num_write":33,"num_write_kb":4032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,6,12,1,4,5,11],"acting":[13,6,12,1,4,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"4","length":"3"},{"start":"a","length":"1"},{"start":"c","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":37,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-02-05T04:53:45.960289+0000","last_change":"2024-02-05T04:53:12.037950+0000","last_active":"2024-02-05T04:53:45.960289+0000","last_peered":"2024-02-05T04:53:45.960289+0000","last_clean":"2024-02-05T04:53:45.960289+0000","last_became_active":"2024-02-05T04:53:12.037750+0000","last_became_peered":"2024-02-05T04:53:12.037750+0000","last_unstale":"2024-02-05T04:53:45.960289+0000","last_undegraded":"2024-02-05T04:53:45.960289+0000","last_fullsized":"2024-02-05T04:53:45.960289+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_clean_scrub_stamp":"2024-02-05T04:53:11.017155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:26.129079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.6","version":"44'26","reported_seq":120,"reported_epoch":54,"state":"remapped+peering","last_fresh":"2024-02-05T04:53:45.957099+0000","last_change":"2024-02-05T04:53:46.961207+0000","last_active":"2024-02-05T04:53:45.957099+0000","last_peered":"2024-02-05T04:53:44.972520+0000","last_clean":"2024-02-05T04:53:44.972520+0000","last_became_active":"2024-02-05T04:53:24.485438+0000","last_became_peered":"2024-02-05T04:53:24.485438+0000","last_unstale":"2024-02-05T04:53:45.957099+0000","last_undegraded":"2024-02-05T04:53:45.957099+0000","last_fullsized":"2024-02-05T04:53:45.957099+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:23.066897+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010167838,"stat_sum":{"num_bytes":3194880,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":8070,"num_write":16,"num_write_kb":3712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,12,4,15,1,7,10],"acting":[13,12,4,15,1,7,2147483647],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,4,7,8,10,12,15],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"4","length":"3"},{"start":"a","length":"1"},{"start":"c","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":36,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-02-05T04:53:45.961244+0000","last_change":"2024-02-05T04:53:12.781269+0000","last_active":"2024-02-05T04:53:45.961244+0000","last_peered":"2024-02-05T04:53:45.961244+0000","last_clean":"2024-02-05T04:53:45.961244+0000","last_became_active":"2024-02-05T04:53:12.780970+0000","last_became_peered":"2024-02-05T04:53:12.780970+0000","last_unstale":"2024-02-05T04:53:45.961244+0000","last_undegraded":"2024-02-05T04:53:45.961244+0000","last_fullsized":"2024-02-05T04:53:45.961244+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_clean_scrub_stamp":"2024-02-05T04:53:11.017155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:36.119048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,0],"acting":[15,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":67665920,"num_objects":69,"num_object_clones":24,"num_object_copies":483,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":4,"num_read":259,"num_read_kb":49648,"num_write":406,"num_write_kb":73840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":9650176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":130297856,"data_stored":159232000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":492,"ondisk_log_size":492,"up":112,"acting":112,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":12,"up_from":23,"seq":98784247819,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32092,"kb_used_data":5064,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339748,"statfs":{"total":96636764160,"available":96603901952,"internally_reserved":0,"allocated":5185536,"data_stored":5406714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,5,6,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28899999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45200000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52900000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.498}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34200000000000003}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54300000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60099999999999998}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56799999999999995}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69199999999999995}]}]},{"osd":11,"up_from":23,"seq":98784247821,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33856,"kb_used_data":6832,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337984,"statfs":{"total":96636764160,"available":96602095616,"internally_reserved":0,"allocated":6995968,"data_stored":7942138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,4,5,6,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.501}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61699999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41699999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59399999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34899999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64700000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46400000000000002}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39400000000000002}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68000000000000005}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51600000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49299999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69599999999999995}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57499999999999996}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88800000000000001}]}]},{"osd":10,"up_from":23,"seq":98784247821,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37984,"kb_used_data":10960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94333856,"statfs":{"total":96636764160,"available":96597868544,"internally_reserved":0,"allocated":11223040,"data_stored":12255226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.437}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47299999999999998}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56000000000000005}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73599999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83099999999999996}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75700000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76900000000000002}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70299999999999996}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81699999999999995}]}]},{"osd":3,"up_from":21,"seq":90194313228,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31436,"kb_used_data":4412,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340404,"statfs":{"total":96636764160,"available":96604573696,"internally_reserved":0,"allocated":4517888,"data_stored":4709460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,4,6,7,8,9,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60399999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35199999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50800000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32600000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.437}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56499999999999995}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58299999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64500000000000002}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52500000000000002}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69799999999999995}]}]},{"osd":4,"up_from":21,"seq":90194313227,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37800,"kb_used_data":10780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94334040,"statfs":{"total":96636764160,"available":96598056960,"internally_reserved":0,"allocated":11038720,"data_stored":13552724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,9,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.373}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46400000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49299999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67000000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45000000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53800000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33500000000000002}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58299999999999996}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60999999999999999}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69799999999999995}]}]},{"osd":5,"up_from":21,"seq":90194313227,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41044,"kb_used_data":9860,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94330796,"statfs":{"total":96636764160,"available":96594735104,"internally_reserved":0,"allocated":10096640,"data_stored":12536916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,6,7,9,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45200000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52700000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69899999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93600000000000005}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78600000000000003}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73599999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80600000000000005}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83999999999999997}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.997}]}]},{"osd":6,"up_from":21,"seq":90194313228,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38868,"kb_used_data":7684,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94332972,"statfs":{"total":96636764160,"available":96596963328,"internally_reserved":0,"allocated":7868416,"data_stored":9210964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,3,4,5,7,8,9,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32000000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52500000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50600000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34999999999999998}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56799999999999995}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.443}]}]},{"osd":1,"up_from":22,"seq":94489280524,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39652,"kb_used_data":12628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94332188,"statfs":{"total":96636764160,"available":96596160512,"internally_reserved":0,"allocated":12931072,"data_stored":15447983,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,4,5,8,9,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45200000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.497}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.308}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.626}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55500000000000005}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.378}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.433}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35599999999999998}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53000000000000003}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67800000000000005}]}]},{"osd":14,"up_from":22,"seq":94489280523,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33272,"kb_used_data":6256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338568,"statfs":{"total":96636764160,"available":96602693632,"internally_reserved":0,"allocated":6406144,"data_stored":6773844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38400000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63300000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.318}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39000000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68200000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34100000000000003}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40000000000000002}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.497}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49299999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57599999999999996}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60599999999999998}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68400000000000005}]}]},{"osd":9,"up_from":22,"seq":94489280523,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33460,"kb_used_data":6440,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338380,"statfs":{"total":96636764160,"available":96602501120,"internally_reserved":0,"allocated":6594560,"data_stored":7698351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,3,4,5,6,8,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36099999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55200000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.495}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70599999999999996}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73699999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59299999999999997}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75700000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]}]},{"osd":13,"up_from":22,"seq":94489280522,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44136,"kb_used_data":12952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94327704,"statfs":{"total":96636764160,"available":96591568896,"internally_reserved":0,"allocated":13262848,"data_stored":16230319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,4,5,6,7,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65500000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36499999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51300000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64200000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80500000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.434}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48699999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41399999999999998}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63500000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.505}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47299999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89900000000000002}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83699999999999997}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86799999999999999}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78600000000000003}]}]},{"osd":0,"up_from":23,"seq":98784247820,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37084,"kb_used_data":10056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94334756,"statfs":{"total":96636764160,"available":96598790144,"internally_reserved":0,"allocated":10297344,"data_stored":10928122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54800000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35599999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48799999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59399999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47199999999999998}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40600000000000003}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68600000000000005}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66300000000000003}]}]},{"osd":15,"up_from":22,"seq":94489280524,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32968,"kb_used_data":5948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338872,"statfs":{"total":96636764160,"available":96603004928,"internally_reserved":0,"allocated":6090752,"data_stored":6698927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,4,5,6,7,8,9,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55700000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.497}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69399999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58899999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57199999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64900000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71499999999999997}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63400000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78800000000000003}]}]},{"osd":2,"up_from":23,"seq":98784247821,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41904,"kb_used_data":10724,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329936,"statfs":{"total":96636764160,"available":96593854464,"internally_reserved":0,"allocated":10981376,"data_stored":13369882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,11,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55100000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.433}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61599999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52500000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41199999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45500000000000002}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56499999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50900000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]}]},{"osd":7,"up_from":23,"seq":98784247821,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35412,"kb_used_data":8388,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94336428,"statfs":{"total":96636764160,"available":96600502272,"internally_reserved":0,"allocated":8589312,"data_stored":10117114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,2,3,5,6,8,9,10,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33900000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39900000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35899999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49299999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51700000000000002}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53200000000000003}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60899999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85199999999999998}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67600000000000005}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58899999999999997}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65000000000000002}]}]},{"osd":8,"up_from":0,"seq":98784247818,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":9543680,"data_stored":10522624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12193792,"data_stored":15052800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9695232,"data_stored":12435456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3796992,"data_stored":4325376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":10317824,"data_stored":13168640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":9375744,"data_stored":12152832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":7147520,"data_stored":8826880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":7835648,"data_stored":9711616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":9826304,"data_stored":12816384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":5857280,"data_stored":7303168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":10469376,"data_stored":11849728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":6242304,"data_stored":7536640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4431872,"data_stored":5001216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":12525568,"data_stored":15835136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":5685248,"data_stored":6389760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":5353472,"data_stored":6303744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T04:53:51.368 INFO:tasks.ceph.ceph_manager.ceph:increase pool size by 10 2024-02-05T04:53:51.369 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pg_num 26 2024-02-05T04:53:51.369 INFO:teuthology.orchestra.run.smithi062.stdout:osd.7: {} 2024-02-05T04:53:51.370 INFO:teuthology.orchestra.run.smithi062.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T04:53:51.386 INFO:teuthology.orchestra.run.smithi062.stdout:osd.8: {} 2024-02-05T04:53:51.386 INFO:teuthology.orchestra.run.smithi062.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-05T04:53:51.387 INFO:tasks.ceph.osd.14.smithi150.stderr:2024-02-05T04:53:51.386+0000 7f5f81083640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 34020) UID: 0 2024-02-05T04:53:51.403 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:51.403 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:51.404 INFO:teuthology.orchestra.run.smithi062.stdout:osd.9: {} 2024-02-05T04:53:51.404 INFO:teuthology.orchestra.run.smithi062.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-05T04:53:51.412 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-02-05T04:53:51.422 INFO:teuthology.orchestra.run.smithi062.stdout:osd.10: {} 2024-02-05T04:53:51.422 INFO:teuthology.orchestra.run.smithi062.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-05T04:53:51.440 INFO:teuthology.orchestra.run.smithi062.stdout:osd.11: {} 2024-02-05T04:53:51.441 INFO:teuthology.orchestra.run.smithi062.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-05T04:53:51.457 INFO:teuthology.orchestra.run.smithi062.stdout:osd.12: {} 2024-02-05T04:53:51.458 INFO:teuthology.orchestra.run.smithi062.stderr:osd.12: osd_enable_op_tracker = '' 2024-02-05T04:53:51.476 INFO:teuthology.orchestra.run.smithi062.stdout:osd.13: {} 2024-02-05T04:53:51.476 INFO:teuthology.orchestra.run.smithi062.stderr:osd.13: osd_enable_op_tracker = '' 2024-02-05T04:53:51.495 INFO:teuthology.orchestra.run.smithi062.stdout:osd.14: {} 2024-02-05T04:53:51.496 INFO:teuthology.orchestra.run.smithi062.stderr:osd.14: osd_enable_op_tracker = '' 2024-02-05T04:53:51.496 INFO:tasks.ceph.osd.6.smithi150.stderr:2024-02-05T04:53:51.495+0000 7f6bea9f0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34015) UID: 0 2024-02-05T04:53:51.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-02-05T04:53:51.495+0000 7feba26f2640 0 -- 172.21.15.62:0/1982755322 >> [v2:172.21.15.196:6826/3187639683,v1:172.21.15.196:6827/3187639683] conn(0x7feb9c2220c0 msgr2=0x7feb9c2244b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-02-05T04:53:51.559 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:51.559 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:51.568 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-02-05T04:53:51.609 INFO:tasks.ceph.osd.6.smithi150.stderr:2024-02-05T04:53:51.608+0000 7f6bea9f0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34015) UID: 0 2024-02-05T04:53:51.690 INFO:tasks.ceph.osd.6.smithi150.stderr:2024-02-05T04:53:51.689+0000 7f6bea9f0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34015) UID: 0 2024-02-05T04:53:51.714 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:51.714 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:51.715 INFO:teuthology.orchestra.run.smithi062.stdout:osd.15: {} 2024-02-05T04:53:51.715 INFO:teuthology.orchestra.run.smithi062.stderr:osd.15: osd_enable_op_tracker = '' 2024-02-05T04:53:51.723 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-02-05T04:53:51.791 INFO:tasks.ceph.osd.0.smithi062.stderr:2024-02-05T04:53:51.790+0000 7fece8e03640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34688) UID: 0 2024-02-05T04:53:51.870 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:51.870 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:51.879 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-02-05T04:53:51.892 INFO:tasks.ceph.osd.8.smithi062.stderr:2024-02-05T04:53:51.891+0000 7f9385bf2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 34693) UID: 0 2024-02-05T04:53:51.953 INFO:teuthology.orchestra.run.smithi062.stderr:set pool 3 pg_num to 26 2024-02-05T04:53:51.958 INFO:tasks.rados.rados.0.smithi196.stdout:213: done (0 left) 2024-02-05T04:53:51.958 INFO:tasks.rados.rados.0.smithi196.stdout:214: rollback oid 34 current snap is 17 2024-02-05T04:53:51.958 INFO:tasks.rados.rados.0.smithi196.stdout:rollback oid 34 to 13 2024-02-05T04:53:51.958 INFO:tasks.rados.rados.0.smithi196.stdout:215: snap_remove snap 14 2024-02-05T04:53:51.987 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:51.988 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:51.993 INFO:tasks.ceph.osd.13.smithi091.stderr:2024-02-05T04:53:51.992+0000 7f526fd4d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 34281) UID: 0 2024-02-05T04:53:51.996 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-02-05T04:53:52.094 INFO:tasks.ceph.osd.8.smithi062.stderr:2024-02-05T04:53:52.093+0000 7f9385bf2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 34693) UID: 0 2024-02-05T04:53:52.101 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:52.102 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:52.110 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-02-05T04:53:52.194 INFO:tasks.ceph.osd.7.smithi196.stderr:2024-02-05T04:53:52.193+0000 7fb281419640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34606) UID: 0 2024-02-05T04:53:52.216 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:52.216 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:52.225 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-02-05T04:53:52.296 INFO:tasks.ceph.osd.11.smithi196.stderr:2024-02-05T04:53:52.294+0000 7f85a8d86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 34609) UID: 0 2024-02-05T04:53:52.342 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:52.342 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:52.352 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-02-05T04:53:52.436 INFO:tasks.ceph.osd.15.smithi196.stderr:2024-02-05T04:53:52.435+0000 7f1facc45640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 34611) UID: 0 2024-02-05T04:53:52.498 INFO:tasks.ceph.osd.12.smithi062.stderr:2024-02-05T04:53:52.497+0000 7f5e5847e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 34695) UID: 0 2024-02-05T04:53:52.498 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:52.499 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:52.507 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-02-05T04:53:52.599 INFO:tasks.ceph.osd.10.smithi150.stderr:2024-02-05T04:53:52.598+0000 7f3075a4d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 34018) UID: 0 2024-02-05T04:53:52.653 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:52.653 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:52.661 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-02-05T04:53:52.699 INFO:tasks.ceph.osd.6.smithi150.stderr:2024-02-05T04:53:52.698+0000 7f6bea9f0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34015) UID: 0 2024-02-05T04:53:52.767 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:52.767 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:52.776 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-02-05T04:53:52.801 INFO:tasks.ceph.osd.8.smithi062.stderr:2024-02-05T04:53:52.800+0000 7f9385bf2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 34693) UID: 0 2024-02-05T04:53:52.882 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:52.882 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:52.891 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-02-05T04:53:52.936 INFO:tasks.ceph.osd.15.smithi196.stderr:2024-02-05T04:53:52.935+0000 7f1facc45640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 34611) UID: 0 2024-02-05T04:53:52.958 INFO:tasks.rados.rados.0.smithi196.stdout:215: done (1 left) 2024-02-05T04:53:52.958 INFO:tasks.rados.rados.0.smithi196.stdout:216: delete oid 41 current snap is 17 2024-02-05T04:53:52.958 INFO:tasks.rados.rados.0.smithi196.stdout:214: finishing rollback tid 0 to smithi19639362-34 2024-02-05T04:53:52.958 INFO:tasks.rados.rados.0.smithi196.stdout:214: finishing rollback tid 1 to smithi19639362-34 2024-02-05T04:53:52.958 INFO:tasks.rados.rados.0.smithi196.stdout:214: finishing rollback tid 2 to smithi19639362-34 2024-02-05T04:53:52.958 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 34 v 50 (ObjNum 161 snap 0 seq_num 1) dirty exists 2024-02-05T04:53:52.963 INFO:tasks.rados.rados.0.smithi196.stdout:214: done (1 left) 2024-02-05T04:53:52.964 INFO:tasks.rados.rados.0.smithi196.stdout:216: done (0 left) 2024-02-05T04:53:52.964 INFO:tasks.rados.rados.0.smithi196.stdout:217: rollback oid 44 current snap is 17 2024-02-05T04:53:52.964 INFO:tasks.rados.rados.0.smithi196.stdout:rollback oid 44 to 11 2024-02-05T04:53:52.964 INFO:tasks.rados.rados.0.smithi196.stdout:218: read oid 11 snap -1 2024-02-05T04:53:52.964 INFO:tasks.rados.rados.0.smithi196.stdout:218: expect (ObjNum 37 snap 0 seq_num 37) 2024-02-05T04:53:52.964 INFO:tasks.rados.rados.0.smithi196.stdout:219: delete oid 43 current snap is 17 2024-02-05T04:53:52.972 INFO:tasks.rados.rados.0.smithi196.stdout:217: finishing rollback tid 1 to smithi19639362-44 2024-02-05T04:53:52.972 INFO:tasks.rados.rados.0.smithi196.stdout:217: finishing rollback tid 2 to smithi19639362-44 2024-02-05T04:53:52.972 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 44 v 37 (ObjNum 68 snap 2 seq_num 68) dirty exists 2024-02-05T04:53:52.973 INFO:tasks.rados.rados.0.smithi196.stdout:218: expect (ObjNum 37 snap 0 seq_num 37) 2024-02-05T04:53:52.988 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-02-05T04:53:52.991 INFO:tasks.rados.rados.0.smithi196.stdout:217: done (2 left) 2024-02-05T04:53:52.991 INFO:tasks.rados.rados.0.smithi196.stdout:218: done (1 left) 2024-02-05T04:53:52.991 INFO:tasks.rados.rados.0.smithi196.stdout:219: done (0 left) 2024-02-05T04:53:52.991 INFO:tasks.rados.rados.0.smithi196.stdout:220: rmattr oid 28 current snap is 17 2024-02-05T04:53:52.991 INFO:tasks.rados.rados.0.smithi196.stdout:220: done (0 left) 2024-02-05T04:53:52.991 INFO:tasks.rados.rados.0.smithi196.stdout:append oid 29 current snap is 17 2024-02-05T04:53:52.991 INFO:tasks.rados.rados.0.smithi196.stdout:221: seq_num 108 ranges {2752512=655360,3407872=606208,4014080=475136,4489216=81920} 2024-02-05T04:53:53.002 INFO:tasks.ceph.osd.13.smithi091.stderr:2024-02-05T04:53:53.002+0000 7f526fd4d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 34281) UID: 0 2024-02-05T04:53:53.004 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:53.004 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:53.009 INFO:tasks.rados.rados.0.smithi196.stdout:221: writing smithi19639362-29 from 2752512 to 3407872 tid 1 2024-02-05T04:53:53.012 INFO:tasks.rados.rados.0.smithi196.stdout:221: writing smithi19639362-29 from 3407872 to 4014080 tid 2 2024-02-05T04:53:53.014 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-02-05T04:53:53.016 INFO:tasks.rados.rados.0.smithi196.stdout:221: writing smithi19639362-29 from 4014080 to 4489216 tid 3 2024-02-05T04:53:53.018 INFO:tasks.rados.rados.0.smithi196.stdout:221: writing smithi19639362-29 from 4489216 to 4571136 tid 4 2024-02-05T04:53:53.018 INFO:tasks.rados.rados.0.smithi196.stdout:222: read oid 41 snap 5 2024-02-05T04:53:53.018 INFO:tasks.rados.rados.0.smithi196.stdout:222: expect (ObjNum 40 snap 0 seq_num 40) 2024-02-05T04:53:53.018 INFO:tasks.rados.rados.0.smithi196.stdout:append oid 35 current snap is 17 2024-02-05T04:53:53.018 INFO:tasks.rados.rados.0.smithi196.stdout:221: finishing write tid 1 to smithi19639362-29 2024-02-05T04:53:53.018 INFO:tasks.rados.rados.0.smithi196.stdout:223: seq_num 109 ranges {4947968=671744,5619712=753664,6373376=409600} 2024-02-05T04:53:53.047 INFO:tasks.rados.rados.0.smithi196.stdout:223: writing smithi19639362-35 from 4947968 to 5619712 tid 1 2024-02-05T04:53:53.051 INFO:tasks.rados.rados.0.smithi196.stdout:223: writing smithi19639362-35 from 5619712 to 6373376 tid 2 2024-02-05T04:53:53.057 INFO:tasks.rados.rados.0.smithi196.stdout:223: writing smithi19639362-35 from 6373376 to 6782976 tid 3 2024-02-05T04:53:53.057 INFO:tasks.rados.rados.0.smithi196.stdout:224: setattr oid 8 current snap is 17 2024-02-05T04:53:53.057 INFO:tasks.rados.rados.0.smithi196.stdout:221: finishing write tid 2 to smithi19639362-29 2024-02-05T04:53:53.057 INFO:tasks.rados.rados.0.smithi196.stdout:222: expect (ObjNum 40 snap 0 seq_num 40) 2024-02-05T04:53:53.072 INFO:tasks.rados.rados.0.smithi196.stdout:221: finishing write tid 3 to smithi19639362-29 2024-02-05T04:53:53.073 INFO:tasks.rados.rados.0.smithi196.stdout:221: finishing write tid 4 to smithi19639362-29 2024-02-05T04:53:53.073 INFO:tasks.rados.rados.0.smithi196.stdout:221: finishing write tid 5 to smithi19639362-29 2024-02-05T04:53:53.073 INFO:tasks.rados.rados.0.smithi196.stdout:221: finishing write tid 6 to smithi19639362-29 2024-02-05T04:53:53.073 INFO:tasks.rados.rados.0.smithi196.stdout:221: oid 29 updating version 0 to 56 2024-02-05T04:53:53.073 INFO:tasks.rados.rados.0.smithi196.stdout:221: oid 29 version 56 is already newer than 55 2024-02-05T04:53:53.073 INFO:tasks.rados.rados.0.smithi196.stdout:221: oid 29 version 56 is already newer than 53 2024-02-05T04:53:53.073 INFO:tasks.rados.rados.0.smithi196.stdout:221: oid 29 version 56 is already newer than 52 2024-02-05T04:53:53.074 INFO:tasks.rados.rados.0.smithi196.stdout:221: oid 29 version 56 is already newer than 54 2024-02-05T04:53:53.074 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 29 v 56 (ObjNum 108 snap 17 seq_num 108) dirty exists 2024-02-05T04:53:53.074 INFO:tasks.rados.rados.0.smithi196.stdout:221: left oid 29 (ObjNum 108 snap 17 seq_num 108) 2024-02-05T04:53:53.074 INFO:tasks.rados.rados.0.smithi196.stdout:223: finishing write tid 1 to smithi19639362-35 2024-02-05T04:53:53.074 INFO:tasks.rados.rados.0.smithi196.stdout:223: finishing write tid 2 to smithi19639362-35 2024-02-05T04:53:53.074 INFO:tasks.rados.rados.0.smithi196.stdout:223: finishing write tid 3 to smithi19639362-35 2024-02-05T04:53:53.074 INFO:tasks.rados.rados.0.smithi196.stdout:223: finishing write tid 4 to smithi19639362-35 2024-02-05T04:53:53.074 INFO:tasks.rados.rados.0.smithi196.stdout:223: finishing write tid 5 to smithi19639362-35 2024-02-05T04:53:53.074 INFO:tasks.rados.rados.0.smithi196.stdout:223: oid 35 updating version 0 to 64 2024-02-05T04:53:53.074 INFO:tasks.rados.rados.0.smithi196.stdout:223: oid 35 updating version 64 to 65 2024-02-05T04:53:53.074 INFO:tasks.rados.rados.0.smithi196.stdout:223: oid 35 updating version 65 to 66 2024-02-05T04:53:53.074 INFO:tasks.rados.rados.0.smithi196.stdout:223: oid 35 version 66 is already newer than 63 2024-02-05T04:53:53.074 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 35 v 66 (ObjNum 109 snap 17 seq_num 109) dirty exists 2024-02-05T04:53:53.075 INFO:tasks.rados.rados.0.smithi196.stdout:223: left oid 35 (ObjNum 109 snap 17 seq_num 109) 2024-02-05T04:53:53.075 INFO:tasks.rados.rados.0.smithi196.stdout:221: done (3 left) 2024-02-05T04:53:53.075 INFO:tasks.rados.rados.0.smithi196.stdout:222: done (2 left) 2024-02-05T04:53:53.075 INFO:tasks.rados.rados.0.smithi196.stdout:223: done (1 left) 2024-02-05T04:53:53.075 INFO:tasks.rados.rados.0.smithi196.stdout:225: delete oid 16 current snap is 17 2024-02-05T04:53:53.077 INFO:tasks.rados.rados.0.smithi196.stdout:225: done (1 left) 2024-02-05T04:53:53.078 INFO:tasks.rados.rados.0.smithi196.stdout:append oid 9 current snap is 17 2024-02-05T04:53:53.078 INFO:tasks.rados.rados.0.smithi196.stdout:226: seq_num 111 ranges {2621440=704512,3325952=655360,3981312=507904} 2024-02-05T04:53:53.095 INFO:tasks.rados.rados.0.smithi196.stdout:226: writing smithi19639362-9 from 2621440 to 3325952 tid 1 2024-02-05T04:53:53.099 INFO:tasks.rados.rados.0.smithi196.stdout:226: writing smithi19639362-9 from 3325952 to 3981312 tid 2 2024-02-05T04:53:53.102 INFO:tasks.rados.rados.0.smithi196.stdout:226: writing smithi19639362-9 from 3981312 to 4489216 tid 3 2024-02-05T04:53:53.102 INFO:tasks.rados.rados.0.smithi196.stdout:227: read oid 29 snap 13 2024-02-05T04:53:53.102 INFO:tasks.rados.rados.0.smithi196.stdout:227: expect (ObjNum 78 snap 4 seq_num 78) 2024-02-05T04:53:53.102 INFO:tasks.rados.rados.0.smithi196.stdout:228: setattr oid 3 current snap is 17 2024-02-05T04:53:53.102 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 8 v 51 (ObjNum 60 snap 2 seq_num 60) dirty exists 2024-02-05T04:53:53.104 INFO:tasks.rados.rados.0.smithi196.stdout:226: finishing write tid 1 to smithi19639362-9 2024-02-05T04:53:53.104 INFO:tasks.rados.rados.0.smithi196.stdout:224: done (3 left) 2024-02-05T04:53:53.104 INFO:tasks.rados.rados.0.smithi196.stdout:229: snap_create 2024-02-05T04:53:53.104 INFO:tasks.ceph.osd.0.smithi062.stderr:2024-02-05T04:53:53.102+0000 7fece8e03640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34688) UID: 0 2024-02-05T04:53:53.105 INFO:tasks.rados.rados.0.smithi196.stdout:226: finishing write tid 2 to smithi19639362-9 2024-02-05T04:53:53.105 INFO:tasks.rados.rados.0.smithi196.stdout:226: finishing write tid 3 to smithi19639362-9 2024-02-05T04:53:53.106 INFO:tasks.rados.rados.0.smithi196.stdout:226: finishing write tid 4 to smithi19639362-9 2024-02-05T04:53:53.107 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 3 v 53 (ObjNum 33 snap 0 seq_num 3191348304) dirty exists 2024-02-05T04:53:53.108 INFO:tasks.rados.rados.0.smithi196.stdout:226: finishing write tid 5 to smithi19639362-9 2024-02-05T04:53:53.108 INFO:tasks.rados.rados.0.smithi196.stdout:226: oid 9 updating version 0 to 71 2024-02-05T04:53:53.108 INFO:tasks.rados.rados.0.smithi196.stdout:226: oid 9 version 71 is already newer than 70 2024-02-05T04:53:53.109 INFO:tasks.rados.rados.0.smithi196.stdout:226: oid 9 version 71 is already newer than 68 2024-02-05T04:53:53.109 INFO:tasks.rados.rados.0.smithi196.stdout:226: oid 9 version 71 is already newer than 69 2024-02-05T04:53:53.109 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 9 v 71 (ObjNum 111 snap 17 seq_num 111) dirty exists 2024-02-05T04:53:53.110 INFO:tasks.rados.rados.0.smithi196.stdout:226: left oid 9 (ObjNum 111 snap 17 seq_num 111) 2024-02-05T04:53:53.117 INFO:tasks.rados.rados.0.smithi196.stdout:227: expect (ObjNum 78 snap 4 seq_num 78) 2024-02-05T04:53:53.131 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:53.132 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:53.141 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-02-05T04:53:53.225 INFO:tasks.ceph.osd.15.smithi196.stderr:2024-02-05T04:53:53.224+0000 7f1facc45640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 34611) UID: 0 2024-02-05T04:53:53.287 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:53.287 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:53.296 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-02-05T04:53:53.304 INFO:tasks.ceph.osd.13.smithi091.stderr:2024-02-05T04:53:53.304+0000 7f526fd4d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 34281) UID: 0 2024-02-05T04:53:53.406 INFO:tasks.ceph.osd.14.smithi150.stderr:2024-02-05T04:53:53.405+0000 7f5f81083640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 34020) UID: 0 2024-02-05T04:53:53.442 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:53.442 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:53.451 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-02-05T04:53:53.506 INFO:tasks.ceph.osd.1.smithi091.stderr:2024-02-05T04:53:53.506+0000 7ff9dc283640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34275) UID: 0 2024-02-05T04:53:53.556 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:53.556 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:53.566 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-02-05T04:53:53.608 INFO:tasks.ceph.osd.6.smithi150.stderr:2024-02-05T04:53:53.607+0000 7f6bea9f0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34015) UID: 0 2024-02-05T04:53:53.671 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:53.671 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:53.680 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-02-05T04:53:53.708 INFO:tasks.ceph.osd.10.smithi150.stderr:2024-02-05T04:53:53.707+0000 7f3075a4d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 34018) UID: 0 2024-02-05T04:53:53.727 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T04:53:53.786 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:53.787 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:53.795 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-02-05T04:53:53.809 INFO:tasks.ceph.osd.1.smithi091.stderr:2024-02-05T04:53:53.809+0000 7ff9dc283640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34275) UID: 0 2024-02-05T04:53:53.876 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0: {} 2024-02-05T04:53:53.876 INFO:teuthology.orchestra.run.smithi062.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T04:53:53.896 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1: {} 2024-02-05T04:53:53.896 INFO:teuthology.orchestra.run.smithi062.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T04:53:53.910 INFO:tasks.ceph.osd.12.smithi062.stderr:2024-02-05T04:53:53.909+0000 7f5e5847e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 34695) UID: 0 2024-02-05T04:53:53.913 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2: {} 2024-02-05T04:53:53.914 INFO:teuthology.orchestra.run.smithi062.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T04:53:53.921 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-02-05T04:53:53.921 INFO:teuthology.orchestra.run.smithi062.stdout: "ops": [], 2024-02-05T04:53:53.921 INFO:teuthology.orchestra.run.smithi062.stdout: "num_ops": 0 2024-02-05T04:53:53.921 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-02-05T04:53:53.930 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-02-05T04:53:53.932 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3: {} 2024-02-05T04:53:53.932 INFO:teuthology.orchestra.run.smithi062.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T04:53:53.950 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4: {} 2024-02-05T04:53:53.951 INFO:teuthology.orchestra.run.smithi062.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T04:53:53.965 INFO:teuthology.orchestra.run.smithi062.stderr:nodeep-scrub is unset 2024-02-05T04:53:53.971 INFO:tasks.rados.rados.0.smithi196.stdout:226: done (3 left) 2024-02-05T04:53:53.971 INFO:tasks.rados.rados.0.smithi196.stdout:227: done (2 left) 2024-02-05T04:53:53.971 INFO:tasks.rados.rados.0.smithi196.stdout:228: done (1 left) 2024-02-05T04:53:53.971 INFO:tasks.rados.rados.0.smithi196.stdout:229: done (0 left) 2024-02-05T04:53:53.972 INFO:tasks.rados.rados.0.smithi196.stdout:append oid 20 current snap is 18 2024-02-05T04:53:53.972 INFO:tasks.rados.rados.0.smithi196.stdout:230: seq_num 112 ranges {0=720896,720896=802816,1523712=360448} 2024-02-05T04:53:53.974 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5: {} 2024-02-05T04:53:53.974 INFO:teuthology.orchestra.run.smithi062.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T04:53:53.976 INFO:tasks.rados.rados.0.smithi196.stdout:230: writing smithi19639362-20 from 0 to 720896 tid 1 2024-02-05T04:53:53.981 INFO:tasks.rados.rados.0.smithi196.stdout:230: writing smithi19639362-20 from 720896 to 1523712 tid 2 2024-02-05T04:53:53.983 INFO:tasks.rados.rados.0.smithi196.stdout:230: writing smithi19639362-20 from 1523712 to 1884160 tid 3 2024-02-05T04:53:53.983 INFO:tasks.rados.rados.0.smithi196.stdout:231: copy_from oid 24 from oid 28 current snap is 18 2024-02-05T04:53:53.983 INFO:tasks.rados.rados.0.smithi196.stdout:append oid 4 current snap is 18 2024-02-05T04:53:53.983 INFO:tasks.rados.rados.0.smithi196.stdout:232: seq_num 113 ranges {2310144=737280,3047424=557056,3604480=606208} 2024-02-05T04:53:53.999 INFO:tasks.rados.rados.0.smithi196.stdout:232: writing smithi19639362-4 from 2310144 to 3047424 tid 1 2024-02-05T04:53:54.002 INFO:tasks.rados.rados.0.smithi196.stdout:232: writing smithi19639362-4 from 3047424 to 3604480 tid 2 2024-02-05T04:53:54.007 INFO:tasks.rados.rados.0.smithi196.stdout:232: writing smithi19639362-4 from 3604480 to 4210688 tid 3 2024-02-05T04:53:54.007 INFO:tasks.rados.rados.0.smithi196.stdout:233: read oid 1 snap -1 2024-02-05T04:53:54.007 INFO:tasks.rados.rados.0.smithi196.stdout:233: expect (ObjNum 96 snap 11 seq_num 96) 2024-02-05T04:53:54.007 INFO:tasks.rados.rados.0.smithi196.stdout:234: rollback oid 30 current snap is 18 2024-02-05T04:53:54.007 INFO:tasks.rados.rados.0.smithi196.stdout:rollback oid 30 to 6 2024-02-05T04:53:54.007 INFO:tasks.rados.rados.0.smithi196.stdout:232: finishing write tid 1 to smithi19639362-4 2024-02-05T04:53:54.007 INFO:tasks.rados.rados.0.smithi196.stdout:232: finishing write tid 2 to smithi19639362-4 2024-02-05T04:53:54.008 INFO:tasks.rados.rados.0.smithi196.stdout:append oid 21 current snap is 18 2024-02-05T04:53:54.008 INFO:tasks.rados.rados.0.smithi196.stdout:235: seq_num 114 ranges {458752=753664,1212416=688128,1900544=475136} 2024-02-05T04:53:54.011 INFO:teuthology.orchestra.run.smithi062.stdout:osd.6: {} 2024-02-05T04:53:54.011 INFO:teuthology.orchestra.run.smithi062.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T04:53:54.011 INFO:tasks.ceph.osd.12.smithi062.stderr:2024-02-05T04:53:54.010+0000 7f5e5847e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 34695) UID: 0 2024-02-05T04:53:54.014 INFO:tasks.rados.rados.0.smithi196.stdout:235: writing smithi19639362-21 from 458752 to 1212416 tid 1 2024-02-05T04:53:54.019 INFO:tasks.rados.rados.0.smithi196.stdout:235: writing smithi19639362-21 from 1212416 to 1900544 tid 2 2024-02-05T04:53:54.022 INFO:tasks.rados.rados.0.smithi196.stdout:235: writing smithi19639362-21 from 1900544 to 2375680 tid 3 2024-02-05T04:53:54.022 INFO:tasks.rados.rados.0.smithi196.stdout:236: snap_remove snap 12 2024-02-05T04:53:54.048 INFO:teuthology.orchestra.run.smithi062.stdout:osd.7: {} 2024-02-05T04:53:54.048 INFO:teuthology.orchestra.run.smithi062.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T04:53:54.065 INFO:teuthology.orchestra.run.smithi062.stdout:osd.8: {} 2024-02-05T04:53:54.065 INFO:teuthology.orchestra.run.smithi062.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-05T04:53:54.070 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-02-05T04:53:54.070 INFO:teuthology.orchestra.run.smithi062.stdout: "ops": [], 2024-02-05T04:53:54.070 INFO:teuthology.orchestra.run.smithi062.stdout: "complaint_time": 30, 2024-02-05T04:53:54.071 INFO:teuthology.orchestra.run.smithi062.stdout: "num_blocked_ops": 0 2024-02-05T04:53:54.071 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-02-05T04:53:54.080 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-02-05T04:53:54.084 INFO:teuthology.orchestra.run.smithi062.stdout:osd.9: {} 2024-02-05T04:53:54.084 INFO:teuthology.orchestra.run.smithi062.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-05T04:53:54.103 INFO:teuthology.orchestra.run.smithi062.stdout:osd.10: {} 2024-02-05T04:53:54.103 INFO:teuthology.orchestra.run.smithi062.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-05T04:53:54.112 INFO:tasks.ceph.osd.11.smithi196.stderr:2024-02-05T04:53:54.111+0000 7f85a8d86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 34609) UID: 0 2024-02-05T04:53:54.120 INFO:teuthology.orchestra.run.smithi062.stdout:osd.11: {} 2024-02-05T04:53:54.121 INFO:teuthology.orchestra.run.smithi062.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-05T04:53:54.138 INFO:teuthology.orchestra.run.smithi062.stdout:osd.12: {} 2024-02-05T04:53:54.138 INFO:teuthology.orchestra.run.smithi062.stderr:osd.12: osd_enable_op_tracker = '' 2024-02-05T04:53:54.155 INFO:teuthology.orchestra.run.smithi062.stdout:osd.13: {} 2024-02-05T04:53:54.156 INFO:teuthology.orchestra.run.smithi062.stderr:osd.13: osd_enable_op_tracker = '' 2024-02-05T04:53:54.174 INFO:teuthology.orchestra.run.smithi062.stdout:osd.14: {} 2024-02-05T04:53:54.174 INFO:teuthology.orchestra.run.smithi062.stderr:osd.14: osd_enable_op_tracker = '' 2024-02-05T04:53:54.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-02-05T04:53:54.173+0000 7f4443fff640 0 -- 172.21.15.62:0/4274005359 >> [v2:172.21.15.196:6826/3187639683,v1:172.21.15.196:6827/3187639683] conn(0x7f44442213e0 msgr2=0x7f44442237d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-02-05T04:53:54.193 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-02-05T04:53:54.193 INFO:teuthology.orchestra.run.smithi062.stdout: "size": 20, 2024-02-05T04:53:54.193 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 600, 2024-02-05T04:53:54.193 INFO:teuthology.orchestra.run.smithi062.stdout: "ops": [ 2024-02-05T04:53:54.193 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.193 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.bs3 43/31 ECSubWrite(tid=139, reqid=osd.13.0:137, at_version=0'0, trim_to=0'0, roll_forward_to=43'28))", 2024-02-05T04:53:54.193 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:35.180865+0000", 2024-02-05T04:53:54.194 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 19.010606428999999, 2024-02-05T04:53:54.194 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0037505139999999999, 2024-02-05T04:53:54.194 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:54.194 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:54.194 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:54.194 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.194 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:54.194 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.180865+0000", 2024-02-05T04:53:54.194 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.194 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.194 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.194 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:54.194 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.180865+0000", 2024-02-05T04:53:54.194 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.195 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.195 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.195 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:54.195 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.180865+0000", 2024-02-05T04:53:54.195 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.0200000000000001e-07 2024-02-05T04:53:54.195 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.195 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.195 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:54.195 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.180875+0000", 2024-02-05T04:53:54.195 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.8959999999999998e-06 2024-02-05T04:53:54.195 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.195 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.195 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:54.195 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.180892+0000", 2024-02-05T04:53:54.196 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.6531000000000001e-05 2024-02-05T04:53:54.196 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.196 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.196 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:54.196 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.180901+0000", 2024-02-05T04:53:54.196 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.2890000000000003e-06 2024-02-05T04:53:54.196 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.196 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.196 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:54.196 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.180961+0000", 2024-02-05T04:53:54.196 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.0012000000000001e-05 2024-02-05T04:53:54.196 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.196 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.196 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:54.197 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.180971+0000", 2024-02-05T04:53:54.197 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0190999999999999e-05 2024-02-05T04:53:54.197 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.197 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.197 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:54.197 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.184573+0000", 2024-02-05T04:53:54.197 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00360181 2024-02-05T04:53:54.197 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.197 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.197 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:54.197 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.184615+0000", 2024-02-05T04:53:54.197 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.1983000000000001e-05 2024-02-05T04:53:54.197 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.197 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:54.197 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.198 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.198 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.198 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.8s3 43/31 ECSubWrite(tid=64, reqid=osd.4.0:62, at_version=0'0, trim_to=0'0, roll_forward_to=43'19))", 2024-02-05T04:53:54.198 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:35.181173+0000", 2024-02-05T04:53:54.198 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 19.010298110000001, 2024-02-05T04:53:54.198 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0033436339999999998, 2024-02-05T04:53:54.198 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:54.198 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:54.198 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:54.198 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.198 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:54.198 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.181173+0000", 2024-02-05T04:53:54.198 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.198 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.199 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.199 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:54.199 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.181173+0000", 2024-02-05T04:53:54.199 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.199 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.199 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.199 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:54.199 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.181174+0000", 2024-02-05T04:53:54.199 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.85e-07 2024-02-05T04:53:54.199 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.199 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.199 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:54.199 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.181183+0000", 2024-02-05T04:53:54.199 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.8019999999999998e-06 2024-02-05T04:53:54.199 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.200 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.200 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:54.200 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.181195+0000", 2024-02-05T04:53:54.200 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.223e-05 2024-02-05T04:53:54.200 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.200 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.200 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:54.200 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.181205+0000", 2024-02-05T04:53:54.200 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.7569999999999999e-06 2024-02-05T04:53:54.200 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.200 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.200 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:54.200 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.181269+0000", 2024-02-05T04:53:54.200 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.4759999999999997e-05 2024-02-05T04:53:54.201 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.201 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.201 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:54.201 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.181278+0000", 2024-02-05T04:53:54.201 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.9500000000000007e-06 2024-02-05T04:53:54.201 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.201 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.201 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:54.201 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.184483+0000", 2024-02-05T04:53:54.201 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0032044019999999999 2024-02-05T04:53:54.201 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.201 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.201 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:54.201 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.184517+0000", 2024-02-05T04:53:54.201 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.3847999999999998e-05 2024-02-05T04:53:54.202 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.202 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:54.202 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.202 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.202 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.202 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.bs3 52/31 ECSubWrite(tid=200, reqid=osd.13.0:200, at_version=52'50, trim_to=0'0, roll_forward_to=51'48))", 2024-02-05T04:53:54.202 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:43.953702+0000", 2024-02-05T04:53:54.202 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 10.237769418999999, 2024-02-05T04:53:54.202 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0050156649999999999, 2024-02-05T04:53:54.202 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:54.202 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:54.202 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:54.202 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.202 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:54.202 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.953702+0000", 2024-02-05T04:53:54.203 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.203 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.203 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.203 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:54.203 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.953702+0000", 2024-02-05T04:53:54.203 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.203 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.203 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.203 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:54.203 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.953702+0000", 2024-02-05T04:53:54.203 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.11e-07 2024-02-05T04:53:54.203 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.203 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.203 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:54.203 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.953721+0000", 2024-02-05T04:53:54.203 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.8932000000000001e-05 2024-02-05T04:53:54.204 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.204 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.204 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:54.204 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.953732+0000", 2024-02-05T04:53:54.204 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1564e-05 2024-02-05T04:53:54.204 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.204 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.204 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:54.204 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.953741+0000", 2024-02-05T04:53:54.204 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.2050000000000008e-06 2024-02-05T04:53:54.204 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.204 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.204 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:54.204 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.953801+0000", 2024-02-05T04:53:54.205 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.9344e-05 2024-02-05T04:53:54.205 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.205 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.205 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:54.205 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.953812+0000", 2024-02-05T04:53:54.205 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0845e-05 2024-02-05T04:53:54.205 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.205 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.205 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:54.205 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.958673+0000", 2024-02-05T04:53:54.205 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0048608640000000003 2024-02-05T04:53:54.205 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.205 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.205 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:54.205 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.958717+0000", 2024-02-05T04:53:54.206 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.4700000000000002e-05 2024-02-05T04:53:54.206 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.206 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:54.206 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.206 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.206 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.206 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.fs4 52/31 ECSubWrite(tid=84, reqid=osd.11.0:84, at_version=52'38, trim_to=0'0, roll_forward_to=51'36))", 2024-02-05T04:53:54.206 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:43.953859+0000", 2024-02-05T04:53:54.206 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 10.237611953, 2024-02-05T04:53:54.206 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0027209690000000002, 2024-02-05T04:53:54.206 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:54.206 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:54.206 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:54.206 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.206 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:54.207 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.953859+0000", 2024-02-05T04:53:54.207 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.207 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.207 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.207 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:54.207 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.953859+0000", 2024-02-05T04:53:54.207 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.207 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.207 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.207 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:54.207 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.953859+0000", 2024-02-05T04:53:54.207 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.1100000000000001e-07 2024-02-05T04:53:54.207 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.207 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.207 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:54.208 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.953876+0000", 2024-02-05T04:53:54.208 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.6997e-05 2024-02-05T04:53:54.208 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.208 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.208 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:54.208 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.953985+0000", 2024-02-05T04:53:54.208 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00010902499999999999 2024-02-05T04:53:54.208 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.208 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.208 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:54.208 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.953994+0000", 2024-02-05T04:53:54.208 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.2379999999999997e-06 2024-02-05T04:53:54.208 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.208 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.208 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:54.209 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.954058+0000", 2024-02-05T04:53:54.209 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.4493999999999996e-05 2024-02-05T04:53:54.209 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.209 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.209 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:54.209 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.954073+0000", 2024-02-05T04:53:54.209 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.4929999999999999e-05 2024-02-05T04:53:54.209 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.209 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.209 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:54.209 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.956549+0000", 2024-02-05T04:53:54.209 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0024758720000000001 2024-02-05T04:53:54.209 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.209 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.209 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:54.210 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.956580+0000", 2024-02-05T04:53:54.210 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.1001999999999999e-05 2024-02-05T04:53:54.210 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.210 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:54.210 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.210 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.210 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.210 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.fs4 52/31 ECSubWrite(tid=85, reqid=osd.11.0:85, at_version=52'40, trim_to=0'0, roll_forward_to=51'36))", 2024-02-05T04:53:54.210 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:43.954702+0000", 2024-02-05T04:53:54.210 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 10.236769268, 2024-02-05T04:53:54.210 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.005090982, 2024-02-05T04:53:54.210 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:54.210 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:54.210 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:54.211 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.211 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:54.211 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.954702+0000", 2024-02-05T04:53:54.211 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.211 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.211 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.211 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:54.211 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.954702+0000", 2024-02-05T04:53:54.211 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.211 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.211 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.211 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:54.211 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.954702+0000", 2024-02-05T04:53:54.211 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.9999999999999997e-07 2024-02-05T04:53:54.211 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.212 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.212 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:54.212 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.954717+0000", 2024-02-05T04:53:54.212 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.4905e-05 2024-02-05T04:53:54.212 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.212 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.212 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:54.212 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.954737+0000", 2024-02-05T04:53:54.212 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.9578000000000001e-05 2024-02-05T04:53:54.212 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.212 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.212 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:54.212 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.954745+0000", 2024-02-05T04:53:54.212 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.8299999999999996e-06 2024-02-05T04:53:54.213 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.213 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.213 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:54.213 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.954820+0000", 2024-02-05T04:53:54.213 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.4832000000000005e-05 2024-02-05T04:53:54.213 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.213 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.213 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:54.213 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.954827+0000", 2024-02-05T04:53:54.213 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.9349999999999997e-06 2024-02-05T04:53:54.213 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.213 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.213 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:54.213 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.959760+0000", 2024-02-05T04:53:54.213 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0049331669999999996 2024-02-05T04:53:54.214 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.214 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.214 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:54.214 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.959793+0000", 2024-02-05T04:53:54.214 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.3034999999999998e-05 2024-02-05T04:53:54.214 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.214 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:54.214 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.214 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.214 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.214 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.9s1 52/31 ECSubWrite(tid=44, reqid=osd.14.0:44, at_version=52'19, trim_to=0'0, roll_forward_to=51'17))", 2024-02-05T04:53:54.214 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:43.957858+0000", 2024-02-05T04:53:54.214 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 10.233612774999999, 2024-02-05T04:53:54.214 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0041238409999999996, 2024-02-05T04:53:54.214 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:54.215 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:54.215 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:54.215 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.215 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:54.215 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.957858+0000", 2024-02-05T04:53:54.215 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.215 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.215 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.215 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:54.215 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.957858+0000", 2024-02-05T04:53:54.215 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.215 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.215 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.215 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:54.215 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.957859+0000", 2024-02-05T04:53:54.215 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.1800000000000002e-07 2024-02-05T04:53:54.216 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.216 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.216 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:54.216 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.957886+0000", 2024-02-05T04:53:54.216 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.7733e-05 2024-02-05T04:53:54.216 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.216 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.216 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:54.216 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.957896+0000", 2024-02-05T04:53:54.216 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.5780000000000005e-06 2024-02-05T04:53:54.216 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.216 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.216 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:54.216 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.957901+0000", 2024-02-05T04:53:54.217 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.5949999999999998e-06 2024-02-05T04:53:54.217 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.217 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.217 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:54.217 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.957958+0000", 2024-02-05T04:53:54.217 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.6323999999999998e-05 2024-02-05T04:53:54.217 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.217 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.217 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:54.217 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.957968+0000", 2024-02-05T04:53:54.217 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0093e-05 2024-02-05T04:53:54.217 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.217 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.217 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:54.218 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.961950+0000", 2024-02-05T04:53:54.218 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0039823269999999999 2024-02-05T04:53:54.218 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.218 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.218 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:54.218 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.961982+0000", 2024-02-05T04:53:54.218 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.1872999999999998e-05 2024-02-05T04:53:54.218 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.218 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:54.218 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.218 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.218 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.218 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.bs3 52/31 ECSubWrite(tid=202, reqid=osd.13.0:200, at_version=0'0, trim_to=0'0, roll_forward_to=52'50))", 2024-02-05T04:53:54.218 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:43.959201+0000", 2024-02-05T04:53:54.218 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 10.232269695999999, 2024-02-05T04:53:54.219 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0055030230000000001, 2024-02-05T04:53:54.219 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:54.219 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:54.219 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:54.219 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.219 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:54.219 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.959201+0000", 2024-02-05T04:53:54.219 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.219 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.219 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.219 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:54.219 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.959201+0000", 2024-02-05T04:53:54.219 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.219 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.219 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.220 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:54.220 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.959202+0000", 2024-02-05T04:53:54.220 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.4899999999999995e-07 2024-02-05T04:53:54.220 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.220 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.220 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:54.220 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.959211+0000", 2024-02-05T04:53:54.220 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.4350000000000003e-06 2024-02-05T04:53:54.220 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.220 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.220 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:54.220 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.959221+0000", 2024-02-05T04:53:54.220 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.8700000000000004e-06 2024-02-05T04:53:54.220 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.221 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.221 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:54.221 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.959227+0000", 2024-02-05T04:53:54.221 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.1349999999999997e-06 2024-02-05T04:53:54.221 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.221 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.221 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:54.221 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.959613+0000", 2024-02-05T04:53:54.221 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00038529800000000001 2024-02-05T04:53:54.221 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.221 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.221 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:54.221 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.959622+0000", 2024-02-05T04:53:54.221 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.4339999999999998e-06 2024-02-05T04:53:54.221 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.222 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.222 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:54.222 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.964670+0000", 2024-02-05T04:53:54.222 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.005047719 2024-02-05T04:53:54.222 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.222 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.222 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:54.222 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.964704+0000", 2024-02-05T04:53:54.222 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.4483000000000002e-05 2024-02-05T04:53:54.222 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.222 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:54.222 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.222 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.222 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.222 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.fs4 52/31 ECSubWrite(tid=87, reqid=osd.11.0:85, at_version=0'0, trim_to=0'0, roll_forward_to=52'40))", 2024-02-05T04:53:54.223 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:43.961973+0000", 2024-02-05T04:53:54.223 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 10.229498080000001, 2024-02-05T04:53:54.223 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0052598189999999998, 2024-02-05T04:53:54.223 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:54.223 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:54.223 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:54.223 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.223 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:54.223 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.961973+0000", 2024-02-05T04:53:54.223 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.223 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.223 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.223 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:54.223 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.961973+0000", 2024-02-05T04:53:54.224 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.224 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.224 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.224 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:54.224 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.961974+0000", 2024-02-05T04:53:54.224 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.9599999999999999e-07 2024-02-05T04:53:54.224 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.224 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.224 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:54.224 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.961983+0000", 2024-02-05T04:53:54.224 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.7270000000000002e-06 2024-02-05T04:53:54.224 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.224 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.224 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:54.225 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.961994+0000", 2024-02-05T04:53:54.225 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0778000000000001e-05 2024-02-05T04:53:54.225 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.225 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.225 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:54.225 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.962002+0000", 2024-02-05T04:53:54.225 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.5299999999999999e-06 2024-02-05T04:53:54.225 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.225 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.225 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:54.225 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.962052+0000", 2024-02-05T04:53:54.225 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.0080000000000002e-05 2024-02-05T04:53:54.225 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.225 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.225 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:54.225 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.962061+0000", 2024-02-05T04:53:54.226 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.8100000000000004e-06 2024-02-05T04:53:54.226 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.226 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.226 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:54.226 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.967197+0000", 2024-02-05T04:53:54.226 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0051366930000000003 2024-02-05T04:53:54.226 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.226 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.226 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:54.226 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.967233+0000", 2024-02-05T04:53:54.226 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.5505e-05 2024-02-05T04:53:54.226 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.226 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:54.226 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.226 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.227 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.227 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.9s1 52/31 ECSubWrite(tid=45, reqid=osd.14.0:44, at_version=0'0, trim_to=0'0, roll_forward_to=52'19))", 2024-02-05T04:53:54.227 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:43.962534+0000", 2024-02-05T04:53:54.227 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 10.228937052999999, 2024-02-05T04:53:54.227 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0047221670000000002, 2024-02-05T04:53:54.227 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:54.227 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:54.227 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:54.227 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.227 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:54.227 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.962534+0000", 2024-02-05T04:53:54.227 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.227 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.227 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.227 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:54.228 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.962534+0000", 2024-02-05T04:53:54.228 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.228 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.228 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.228 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:54.228 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.962534+0000", 2024-02-05T04:53:54.228 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.0999999999999999e-07 2024-02-05T04:53:54.228 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.228 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.228 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:54.228 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.962543+0000", 2024-02-05T04:53:54.228 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.4889999999999995e-06 2024-02-05T04:53:54.228 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.229 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.229 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:54.229 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.962554+0000", 2024-02-05T04:53:54.229 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0576e-05 2024-02-05T04:53:54.229 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.229 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.229 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:54.229 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.962562+0000", 2024-02-05T04:53:54.229 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.9150000000000002e-06 2024-02-05T04:53:54.229 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.229 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.229 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:54.229 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.962614+0000", 2024-02-05T04:53:54.229 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.1167e-05 2024-02-05T04:53:54.229 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.230 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.230 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:54.230 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.962623+0000", 2024-02-05T04:53:54.230 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.1670000000000005e-06 2024-02-05T04:53:54.230 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.230 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.230 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:54.230 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.967222+0000", 2024-02-05T04:53:54.230 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.004598787 2024-02-05T04:53:54.230 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.230 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.230 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:54.230 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.967256+0000", 2024-02-05T04:53:54.230 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.4555999999999999e-05 2024-02-05T04:53:54.230 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.230 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:54.231 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.231 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.231 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.231 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.fs4 54/31 ECSubWrite(tid=88, reqid=client.4634.0:507, at_version=54'42, trim_to=0'0, roll_forward_to=52'40))", 2024-02-05T04:53:54.231 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:45.962031+0000", 2024-02-05T04:53:54.231 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 8.2294395080000005, 2024-02-05T04:53:54.231 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0046418570000000001, 2024-02-05T04:53:54.231 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:54.231 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:54.231 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:54.231 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.231 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:54.231 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:45.962031+0000", 2024-02-05T04:53:54.231 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.231 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.232 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.232 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:54.232 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:45.962031+0000", 2024-02-05T04:53:54.232 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.232 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.232 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.232 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:54.232 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:45.962032+0000", 2024-02-05T04:53:54.232 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.99e-07 2024-02-05T04:53:54.232 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.232 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.232 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:54.232 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:45.962058+0000", 2024-02-05T04:53:54.232 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.6069999999999999e-05 2024-02-05T04:53:54.232 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.233 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.233 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:54.233 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:45.962072+0000", 2024-02-05T04:53:54.233 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.4423000000000001e-05 2024-02-05T04:53:54.233 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.233 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.233 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:54.233 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:45.962084+0000", 2024-02-05T04:53:54.233 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.2150000000000001e-05 2024-02-05T04:53:54.233 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.233 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.233 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:54.233 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:45.962145+0000", 2024-02-05T04:53:54.234 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.0795999999999999e-05 2024-02-05T04:53:54.234 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.234 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.234 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:54.234 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:45.962158+0000", 2024-02-05T04:53:54.234 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.2801e-05 2024-02-05T04:53:54.234 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.234 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.234 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:54.234 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:45.966640+0000", 2024-02-05T04:53:54.234 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0044822799999999999 2024-02-05T04:53:54.234 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.234 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.234 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:54.234 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:45.966673+0000", 2024-02-05T04:53:54.235 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.3138e-05 2024-02-05T04:53:54.235 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.235 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:54.235 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.235 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.235 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.235 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpRead(3.bs3 58/57 ECSubRead(tid=242, to_read={3:df6528f1:::smithi19639362-33:head=0,532480,0}, subchunks={3:df6528f1:::smithi19639362-33:head=[0,1]}, attrs_to_read=))", 2024-02-05T04:53:54.235 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:49.955701+0000", 2024-02-05T04:53:54.235 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 4.2357696120000004, 2024-02-05T04:53:54.235 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0063665620000000001, 2024-02-05T04:53:54.235 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:54.235 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:54.235 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:54.235 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.235 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:54.236 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.955701+0000", 2024-02-05T04:53:54.236 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.236 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.236 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.236 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:54.236 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.955701+0000", 2024-02-05T04:53:54.236 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.236 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.236 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.236 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:54.236 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.955702+0000", 2024-02-05T04:53:54.236 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.0900000000000001e-07 2024-02-05T04:53:54.236 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.237 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.237 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:54.237 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.955710+0000", 2024-02-05T04:53:54.237 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.2220000000000002e-06 2024-02-05T04:53:54.237 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.237 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.237 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:54.237 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.955732+0000", 2024-02-05T04:53:54.237 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.2018000000000002e-05 2024-02-05T04:53:54.237 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.237 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.237 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:54.237 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.955745+0000", 2024-02-05T04:53:54.237 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.3025e-05 2024-02-05T04:53:54.238 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.238 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.238 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:54.238 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.955862+0000", 2024-02-05T04:53:54.238 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00011653400000000001 2024-02-05T04:53:54.238 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.238 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.238 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:54.238 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.962068+0000", 2024-02-05T04:53:54.238 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0062060539999999999 2024-02-05T04:53:54.238 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.238 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:54.238 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.238 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.239 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.239 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.4s3 58/57 ECSubWrite(tid=107, reqid=osd.1.0:107, at_version=58'59, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:53:54.239 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:49.956038+0000", 2024-02-05T04:53:54.239 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 4.2354327539999996, 2024-02-05T04:53:54.239 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.005678163, 2024-02-05T04:53:54.239 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:54.239 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:54.239 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:54.239 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.239 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:54.239 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956038+0000", 2024-02-05T04:53:54.239 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.239 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.239 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.240 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:54.240 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956038+0000", 2024-02-05T04:53:54.240 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.240 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.240 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.240 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:54.240 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956039+0000", 2024-02-05T04:53:54.240 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.2200000000000003e-07 2024-02-05T04:53:54.240 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.240 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.240 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:54.240 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956068+0000", 2024-02-05T04:53:54.240 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.9013000000000002e-05 2024-02-05T04:53:54.240 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.241 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.241 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:54.241 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956082+0000", 2024-02-05T04:53:54.241 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.4066999999999999e-05 2024-02-05T04:53:54.241 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.241 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.241 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:54.241 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956091+0000", 2024-02-05T04:53:54.241 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.9539999999999993e-06 2024-02-05T04:53:54.241 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.241 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.241 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:54.241 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956177+0000", 2024-02-05T04:53:54.242 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.5548000000000002e-05 2024-02-05T04:53:54.242 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.242 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.242 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:54.242 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956187+0000", 2024-02-05T04:53:54.242 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0352e-05 2024-02-05T04:53:54.242 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.242 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.242 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:54.242 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.961666+0000", 2024-02-05T04:53:54.242 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0054792979999999996 2024-02-05T04:53:54.242 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.242 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.242 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:54.242 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.961716+0000", 2024-02-05T04:53:54.243 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.0108999999999997e-05 2024-02-05T04:53:54.243 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.243 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:54.243 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.243 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.243 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.243 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.4s3 58/57 ECSubWrite(tid=108, reqid=osd.1.0:108, at_version=58'61, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:53:54.243 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:49.956139+0000", 2024-02-05T04:53:54.243 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 4.2353321790000003, 2024-02-05T04:53:54.243 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0079487310000000005, 2024-02-05T04:53:54.243 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:54.243 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:54.243 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:54.244 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.244 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:54.244 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956139+0000", 2024-02-05T04:53:54.244 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.244 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.244 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.244 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:54.244 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956139+0000", 2024-02-05T04:53:54.244 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.244 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.244 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.244 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:54.244 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956139+0000", 2024-02-05T04:53:54.244 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.0999999999999999e-07 2024-02-05T04:53:54.244 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.245 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.245 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:54.245 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956165+0000", 2024-02-05T04:53:54.245 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.5202999999999999e-05 2024-02-05T04:53:54.245 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.245 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.245 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:54.245 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956176+0000", 2024-02-05T04:53:54.245 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1523e-05 2024-02-05T04:53:54.245 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.245 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.245 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:54.245 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956184+0000", 2024-02-05T04:53:54.245 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.0940000000000008e-06 2024-02-05T04:53:54.246 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.246 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.246 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:54.246 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956668+0000", 2024-02-05T04:53:54.246 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000483535 2024-02-05T04:53:54.246 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.246 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.246 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:54.246 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956681+0000", 2024-02-05T04:53:54.246 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.2958e-05 2024-02-05T04:53:54.246 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.246 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.246 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:54.246 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.964043+0000", 2024-02-05T04:53:54.247 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0073623769999999998 2024-02-05T04:53:54.247 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.247 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.247 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:54.247 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.964088+0000", 2024-02-05T04:53:54.247 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.4530999999999998e-05 2024-02-05T04:53:54.247 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.247 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:54.247 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.247 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.247 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.247 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpRead(3.bs3 58/57 ECSubRead(tid=243, to_read={3:db93bf34:::smithi19639362-24:head=0,98304,0}, subchunks={3:db93bf34:::smithi19639362-24:head=[0,1]}, attrs_to_read=))", 2024-02-05T04:53:54.247 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:49.956639+0000", 2024-02-05T04:53:54.247 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 4.2348315149999998, 2024-02-05T04:53:54.248 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0056018429999999996, 2024-02-05T04:53:54.248 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:54.248 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:54.248 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:54.248 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.248 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:54.248 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956639+0000", 2024-02-05T04:53:54.248 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.248 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.248 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.248 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:54.248 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956639+0000", 2024-02-05T04:53:54.248 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.248 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.248 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.249 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:54.249 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956640+0000", 2024-02-05T04:53:54.249 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.6800000000000009e-07 2024-02-05T04:53:54.249 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.249 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.249 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:54.249 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956650+0000", 2024-02-05T04:53:54.249 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.9739999999999997e-06 2024-02-05T04:53:54.249 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.249 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.249 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:54.249 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956681+0000", 2024-02-05T04:53:54.249 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.0655999999999999e-05 2024-02-05T04:53:54.249 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.249 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.250 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:54.250 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956707+0000", 2024-02-05T04:53:54.250 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.5469999999999998e-05 2024-02-05T04:53:54.250 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.250 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.250 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:54.250 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.962076+0000", 2024-02-05T04:53:54.250 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0053699300000000002 2024-02-05T04:53:54.250 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.250 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.250 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:54.250 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.962241+0000", 2024-02-05T04:53:54.250 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000164845 2024-02-05T04:53:54.250 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.250 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:54.251 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.251 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.251 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.251 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpRead(3.bs3 58/57 ECSubRead(tid=245, to_read={3:db93bf34:::smithi19639362-24:head=0,98304,0}, subchunks={3:db93bf34:::smithi19639362-24:head=[0,1]}, attrs_to_read=))", 2024-02-05T04:53:54.251 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:49.958849+0000", 2024-02-05T04:53:54.251 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 4.2326215270000001, 2024-02-05T04:53:54.251 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0035797170000000001, 2024-02-05T04:53:54.251 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:54.251 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:54.251 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:54.251 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.251 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:54.251 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.958849+0000", 2024-02-05T04:53:54.251 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.252 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.252 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.252 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:54.252 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.958849+0000", 2024-02-05T04:53:54.252 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.252 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.252 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.252 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:54.252 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.958850+0000", 2024-02-05T04:53:54.252 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.6499999999999998e-07 2024-02-05T04:53:54.252 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.252 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.252 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:54.253 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.958868+0000", 2024-02-05T04:53:54.253 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.7274000000000001e-05 2024-02-05T04:53:54.253 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.253 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.253 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:54.253 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.958885+0000", 2024-02-05T04:53:54.253 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.7087000000000001e-05 2024-02-05T04:53:54.253 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.253 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.253 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:54.253 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.958897+0000", 2024-02-05T04:53:54.253 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1810999999999999e-05 2024-02-05T04:53:54.253 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.253 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.253 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:54.254 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.962298+0000", 2024-02-05T04:53:54.254 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0034016189999999998 2024-02-05T04:53:54.254 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.254 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.254 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:54.254 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.962429+0000", 2024-02-05T04:53:54.254 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000131061 2024-02-05T04:53:54.254 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.254 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:54.254 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.254 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.254 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.254 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpRead(3.bs3 58/57 ECSubRead(tid=246, to_read={3:db93bf34:::smithi19639362-24:head=0,98304,0}, subchunks={3:db93bf34:::smithi19639362-24:head=[0,1]}, attrs_to_read=))", 2024-02-05T04:53:54.254 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:49.959544+0000", 2024-02-05T04:53:54.254 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 4.2319268799999996, 2024-02-05T04:53:54.255 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0030492060000000001, 2024-02-05T04:53:54.255 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:54.255 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:54.255 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:54.255 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.255 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:54.255 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.959544+0000", 2024-02-05T04:53:54.255 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.255 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.255 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.255 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:54.255 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.959544+0000", 2024-02-05T04:53:54.255 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.255 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.255 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.256 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:54.256 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.959545+0000", 2024-02-05T04:53:54.256 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.4300000000000003e-07 2024-02-05T04:53:54.256 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.256 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.256 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:54.256 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.959553+0000", 2024-02-05T04:53:54.256 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.9650000000000008e-06 2024-02-05T04:53:54.256 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.256 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.256 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:54.256 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.959568+0000", 2024-02-05T04:53:54.256 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.5211999999999999e-05 2024-02-05T04:53:54.256 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.257 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.257 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:54.257 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.959582+0000", 2024-02-05T04:53:54.257 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.3769e-05 2024-02-05T04:53:54.257 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.257 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.257 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:54.257 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.962471+0000", 2024-02-05T04:53:54.257 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0028894099999999998 2024-02-05T04:53:54.257 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.257 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.257 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:54.257 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.962593+0000", 2024-02-05T04:53:54.257 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00012220699999999999 2024-02-05T04:53:54.257 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.258 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:54.258 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.258 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.258 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.258 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.4s3 58/57 ECSubWrite(tid=109, reqid=osd.1.0:108, at_version=0'0, trim_to=0'0, roll_forward_to=58'61))", 2024-02-05T04:53:54.258 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:49.964582+0000", 2024-02-05T04:53:54.258 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 4.2268888259999997, 2024-02-05T04:53:54.258 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0024986119999999999, 2024-02-05T04:53:54.258 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:54.258 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:54.258 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:54.258 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.258 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:54.259 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.964582+0000", 2024-02-05T04:53:54.259 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.259 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.259 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.259 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:54.259 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.964582+0000", 2024-02-05T04:53:54.259 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.259 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.259 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.259 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:54.259 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.964583+0000", 2024-02-05T04:53:54.259 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.7000000000000004e-07 2024-02-05T04:53:54.259 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.259 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.259 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:54.260 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.964594+0000", 2024-02-05T04:53:54.260 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1317e-05 2024-02-05T04:53:54.260 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.260 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.260 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:54.260 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.964608+0000", 2024-02-05T04:53:54.260 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.4064e-05 2024-02-05T04:53:54.260 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.260 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.260 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:54.260 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.964617+0000", 2024-02-05T04:53:54.260 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.5799999999999992e-06 2024-02-05T04:53:54.260 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.260 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.261 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:54.261 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.964715+0000", 2024-02-05T04:53:54.261 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.7742e-05 2024-02-05T04:53:54.261 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.261 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.261 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:54.261 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.964726+0000", 2024-02-05T04:53:54.261 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1715999999999999e-05 2024-02-05T04:53:54.261 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.261 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.261 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:54.261 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.967039+0000", 2024-02-05T04:53:54.261 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.002312761 2024-02-05T04:53:54.261 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.261 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.262 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:54.262 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.967081+0000", 2024-02-05T04:53:54.262 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.1662000000000002e-05 2024-02-05T04:53:54.262 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.262 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:54.262 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.262 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.262 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.262 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.bs3 58/57 ECSubWrite(tid=251, reqid=osd.13.0:249, at_version=0'0, trim_to=0'0, roll_forward_to=58'52))", 2024-02-05T04:53:54.262 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:49.968190+0000", 2024-02-05T04:53:54.262 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 4.2232810609999998, 2024-02-05T04:53:54.262 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0043444590000000002, 2024-02-05T04:53:54.262 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:54.262 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:54.263 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:54.263 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.263 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:54.263 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.968190+0000", 2024-02-05T04:53:54.263 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.263 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.263 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.263 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:54.263 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.968190+0000", 2024-02-05T04:53:54.263 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.263 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.263 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.263 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:54.263 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.968191+0000", 2024-02-05T04:53:54.263 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.4899999999999995e-07 2024-02-05T04:53:54.264 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.264 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.264 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:54.264 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.968199+0000", 2024-02-05T04:53:54.264 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.3480000000000004e-06 2024-02-05T04:53:54.264 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.264 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.264 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:54.264 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.968211+0000", 2024-02-05T04:53:54.264 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1812e-05 2024-02-05T04:53:54.264 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.264 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.264 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:54.264 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.968217+0000", 2024-02-05T04:53:54.265 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.3210000000000001e-06 2024-02-05T04:53:54.265 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.265 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.265 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:54.265 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.968273+0000", 2024-02-05T04:53:54.265 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.5856999999999997e-05 2024-02-05T04:53:54.265 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.265 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.265 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:54.265 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.968284+0000", 2024-02-05T04:53:54.265 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1112e-05 2024-02-05T04:53:54.265 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.265 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.265 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:54.266 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.972492+0000", 2024-02-05T04:53:54.266 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0042084260000000004 2024-02-05T04:53:54.266 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.266 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.266 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:54.266 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.972534+0000", 2024-02-05T04:53:54.266 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.1934000000000002e-05 2024-02-05T04:53:54.266 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.266 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:54.266 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.266 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.266 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.266 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.bs3 58/57 ECSubWrite(tid=252, reqid=osd.13.0:252, at_version=58'54, trim_to=0'0, roll_forward_to=58'52))", 2024-02-05T04:53:54.266 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:49.968817+0000", 2024-02-05T04:53:54.266 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 4.2226544439999998, 2024-02-05T04:53:54.267 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0037491209999999998, 2024-02-05T04:53:54.267 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:54.267 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:54.267 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:54.267 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.267 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:54.267 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.968817+0000", 2024-02-05T04:53:54.267 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.267 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.267 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.267 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:54.267 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.968817+0000", 2024-02-05T04:53:54.267 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.267 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.268 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.268 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:54.268 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.968817+0000", 2024-02-05T04:53:54.268 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.5000000000000002e-07 2024-02-05T04:53:54.268 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.268 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.268 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:54.268 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.968835+0000", 2024-02-05T04:53:54.268 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.7833999999999999e-05 2024-02-05T04:53:54.268 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.268 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.268 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:54.268 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.968846+0000", 2024-02-05T04:53:54.268 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1093e-05 2024-02-05T04:53:54.268 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.269 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.269 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:54.269 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.968859+0000", 2024-02-05T04:53:54.269 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.2690999999999999e-05 2024-02-05T04:53:54.269 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.269 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.269 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:54.269 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.968921+0000", 2024-02-05T04:53:54.269 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.2593000000000002e-05 2024-02-05T04:53:54.269 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.269 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.269 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:54.269 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.968933+0000", 2024-02-05T04:53:54.269 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1212e-05 2024-02-05T04:53:54.270 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.270 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.270 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:54.270 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.972538+0000", 2024-02-05T04:53:54.270 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.003605811 2024-02-05T04:53:54.270 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.270 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.270 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:54.270 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.972566+0000", 2024-02-05T04:53:54.270 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.7237e-05 2024-02-05T04:53:54.270 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.270 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:54.270 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.270 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.270 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.271 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.fs4 58/31 ECSubWrite(tid=104, reqid=client.4634.0:518, at_version=58'51, trim_to=0'0, roll_forward_to=57'49))", 2024-02-05T04:53:54.271 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:49.973886+0000", 2024-02-05T04:53:54.271 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 4.217585401, 2024-02-05T04:53:54.271 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.004459373, 2024-02-05T04:53:54.271 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:54.271 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:54.271 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:54.271 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.271 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:54.271 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.973886+0000", 2024-02-05T04:53:54.271 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.271 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.271 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.271 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:54.271 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.973886+0000", 2024-02-05T04:53:54.272 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.272 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.272 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.272 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:54.272 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.973886+0000", 2024-02-05T04:53:54.272 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.8100000000000001e-07 2024-02-05T04:53:54.272 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.272 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.272 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:54.272 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.975693+0000", 2024-02-05T04:53:54.272 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0018069480000000001 2024-02-05T04:53:54.272 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.272 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.272 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:54.272 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.975711+0000", 2024-02-05T04:53:54.273 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.7669e-05 2024-02-05T04:53:54.273 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.273 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.273 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:54.273 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.975721+0000", 2024-02-05T04:53:54.273 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.8649999999999996e-06 2024-02-05T04:53:54.273 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.273 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.273 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:54.273 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.975801+0000", 2024-02-05T04:53:54.273 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.0357000000000003e-05 2024-02-05T04:53:54.273 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.273 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.273 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:54.274 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.975813+0000", 2024-02-05T04:53:54.274 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1590000000000001e-05 2024-02-05T04:53:54.274 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.274 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.274 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:54.274 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.978304+0000", 2024-02-05T04:53:54.274 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0024906350000000002 2024-02-05T04:53:54.274 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.274 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.274 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:54.274 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.978345+0000", 2024-02-05T04:53:54.274 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.1427999999999997e-05 2024-02-05T04:53:54.274 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.274 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:54.274 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.275 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.275 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:54.275 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-02-05T04:53:54.279 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-02-05T04:53:54.280 INFO:tasks.ceph.osd.10.smithi150.stderr:2024-02-05T04:53:54.275+0000 7f3075a4d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 34018) UID: 0 2024-02-05T04:53:54.377 INFO:tasks.ceph.osd.11.smithi196.stderr:2024-02-05T04:53:54.376+0000 7f85a8d86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 34609) UID: 0 2024-02-05T04:53:54.393 INFO:teuthology.orchestra.run.smithi062.stdout:osd.15: {} 2024-02-05T04:53:54.393 INFO:teuthology.orchestra.run.smithi062.stderr:osd.15: osd_enable_op_tracker = '' 2024-02-05T04:53:54.426 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-02-05T04:53:54.427 INFO:teuthology.orchestra.run.smithi062.stdout: "ops": [ 2024-02-05T04:53:54.427 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.427 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:591 3.8s0 3.4d1e10e8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-02-05T04:53:54.427 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:54.007323+0000", 2024-02-05T04:53:54.427 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 0.41906446600000002, 2024-02-05T04:53:54.427 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.41908612499999998, 2024-02-05T04:53:54.427 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:54.427 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "waiting for peered", 2024-02-05T04:53:54.427 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:53:54.427 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:53:54.427 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:54.427 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 591 2024-02-05T04:53:54.428 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.428 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:54.428 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.428 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:54.428 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.007323+0000", 2024-02-05T04:53:54.428 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.428 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.428 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.428 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:54.428 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.007323+0000", 2024-02-05T04:53:54.428 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.428 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.428 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.429 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:54.429 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.007324+0000", 2024-02-05T04:53:54.429 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.8339999999999999e-06 2024-02-05T04:53:54.429 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.429 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.429 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:54.429 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.007333+0000", 2024-02-05T04:53:54.429 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.5790000000000004e-06 2024-02-05T04:53:54.429 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.429 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.429 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:54.429 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.007353+0000", 2024-02-05T04:53:54.429 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.9568999999999998e-05 2024-02-05T04:53:54.429 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.429 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.430 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:54.430 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.007363+0000", 2024-02-05T04:53:54.430 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0319000000000001e-05 2024-02-05T04:53:54.430 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.430 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.430 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:54.430 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.007446+0000", 2024-02-05T04:53:54.430 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.2924999999999996e-05 2024-02-05T04:53:54.430 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.430 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.430 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for peered", 2024-02-05T04:53:54.430 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.007450+0000", 2024-02-05T04:53:54.430 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.5909999999999998e-06 2024-02-05T04:53:54.430 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.431 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:54.431 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.431 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.431 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.431 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:592 3.8s0 3.4d1e10e8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-02-05T04:53:54.431 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:54.007387+0000", 2024-02-05T04:53:54.431 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 0.41900005699999998, 2024-02-05T04:53:54.431 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.41906416400000002, 2024-02-05T04:53:54.431 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:54.431 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "waiting for peered", 2024-02-05T04:53:54.431 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:53:54.431 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:53:54.431 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:54.431 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 592 2024-02-05T04:53:54.431 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.432 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:54.432 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.432 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:54.432 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.007387+0000", 2024-02-05T04:53:54.432 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.432 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.432 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.432 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:54.432 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.007387+0000", 2024-02-05T04:53:54.432 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.432 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.432 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.432 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:54.432 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.007387+0000", 2024-02-05T04:53:54.432 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.3799999999999998e-07 2024-02-05T04:53:54.433 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.433 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.433 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:54.433 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.007390+0000", 2024-02-05T04:53:54.433 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.4030000000000001e-06 2024-02-05T04:53:54.433 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.433 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.433 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:54.433 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.007397+0000", 2024-02-05T04:53:54.433 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.272e-06 2024-02-05T04:53:54.433 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.433 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.433 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:54.433 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.007402+0000", 2024-02-05T04:53:54.433 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.7400000000000004e-06 2024-02-05T04:53:54.433 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.434 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.434 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:54.434 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.007491+0000", 2024-02-05T04:53:54.434 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.9462999999999994e-05 2024-02-05T04:53:54.434 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.434 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.434 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for peered", 2024-02-05T04:53:54.434 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.007493+0000", 2024-02-05T04:53:54.434 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.7540000000000001e-06 2024-02-05T04:53:54.434 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.434 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:54.434 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.434 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.434 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.434 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:593 3.8s0 3.4d1e10e8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-02-05T04:53:54.435 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:54.007407+0000", 2024-02-05T04:53:54.435 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 0.41897957000000002, 2024-02-05T04:53:54.435 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.41907878799999998, 2024-02-05T04:53:54.435 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:54.435 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "waiting for peered", 2024-02-05T04:53:54.435 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:53:54.435 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:53:54.435 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:54.435 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 593 2024-02-05T04:53:54.435 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.435 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:54.436 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.436 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:54.436 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.007407+0000", 2024-02-05T04:53:54.436 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.436 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.436 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.436 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:54.436 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.007407+0000", 2024-02-05T04:53:54.436 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.436 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.437 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.437 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:54.437 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.007408+0000", 2024-02-05T04:53:54.437 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.0400000000000002e-07 2024-02-05T04:53:54.437 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.437 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.437 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:54.437 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.007413+0000", 2024-02-05T04:53:54.437 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.9790000000000001e-06 2024-02-05T04:53:54.437 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.437 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.437 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:54.437 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.007418+0000", 2024-02-05T04:53:54.437 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.8180000000000003e-06 2024-02-05T04:53:54.438 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.438 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.438 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:54.438 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.007422+0000", 2024-02-05T04:53:54.438 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.1930000000000003e-06 2024-02-05T04:53:54.438 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.438 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.438 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:54.438 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.007524+0000", 2024-02-05T04:53:54.438 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000101808 2024-02-05T04:53:54.438 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.438 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.438 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for peered", 2024-02-05T04:53:54.438 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.007525+0000", 2024-02-05T04:53:54.438 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.2330000000000001e-06 2024-02-05T04:53:54.439 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.439 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:54.439 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.439 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.439 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-02-05T04:53:54.439 INFO:teuthology.orchestra.run.smithi062.stdout: "num_ops": 3 2024-02-05T04:53:54.439 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-02-05T04:53:54.440 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-02-05T04:53:54.478 INFO:tasks.ceph.osd.6.smithi150.stderr:2024-02-05T04:53:54.477+0000 7f6bea9f0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34015) UID: 0 2024-02-05T04:53:54.579 INFO:tasks.ceph.osd.8.smithi062.stderr:2024-02-05T04:53:54.578+0000 7f9385bf2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 34693) UID: 0 2024-02-05T04:53:54.590 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-02-05T04:53:54.590 INFO:teuthology.orchestra.run.smithi062.stdout: "ops": [], 2024-02-05T04:53:54.590 INFO:teuthology.orchestra.run.smithi062.stdout: "complaint_time": 30, 2024-02-05T04:53:54.590 INFO:teuthology.orchestra.run.smithi062.stdout: "num_blocked_ops": 0 2024-02-05T04:53:54.590 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-02-05T04:53:54.598 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-02-05T04:53:54.683 INFO:tasks.ceph.osd.4.smithi062.stderr:2024-02-05T04:53:54.682+0000 7f708c308640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34689) UID: 0 2024-02-05T04:53:54.749 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-02-05T04:53:54.749 INFO:teuthology.orchestra.run.smithi062.stdout: "size": 20, 2024-02-05T04:53:54.749 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 600, 2024-02-05T04:53:54.749 INFO:teuthology.orchestra.run.smithi062.stdout: "ops": [ 2024-02-05T04:53:54.749 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.749 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:218 3.3s0 3:ce48de55:::smithi19639362-16:head [append 0~638976] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:53:54.749 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:29.132294+0000", 2024-02-05T04:53:54.749 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 25.614555745000001, 2024-02-05T04:53:54.749 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0081869909999999994, 2024-02-05T04:53:54.750 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:54.750 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:54.750 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:53:54.750 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:53:54.750 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:54.750 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 218 2024-02-05T04:53:54.750 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.750 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:54.750 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.750 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:54.750 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.132294+0000", 2024-02-05T04:53:54.750 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.750 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.750 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.751 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:54.751 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.132294+0000", 2024-02-05T04:53:54.751 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.751 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.751 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.751 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:54.751 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.132296+0000", 2024-02-05T04:53:54.751 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.886e-06 2024-02-05T04:53:54.751 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.751 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.751 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:54.751 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.133151+0000", 2024-02-05T04:53:54.751 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00085535699999999995 2024-02-05T04:53:54.751 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.751 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.752 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:54.752 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.133165+0000", 2024-02-05T04:53:54.752 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.396e-05 2024-02-05T04:53:54.752 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.752 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.752 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:54.752 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.133182+0000", 2024-02-05T04:53:54.752 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.6787e-05 2024-02-05T04:53:54.752 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.752 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.752 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:54.752 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.133229+0000", 2024-02-05T04:53:54.752 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.7509999999999997e-05 2024-02-05T04:53:54.752 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.753 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.753 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:53:54.753 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.133324+0000", 2024-02-05T04:53:54.753 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.4950000000000004e-05 2024-02-05T04:53:54.753 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.753 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.753 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:54.753 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.134392+0000", 2024-02-05T04:53:54.753 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0010674219999999999 2024-02-05T04:53:54.753 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.753 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.753 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:54.753 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.138240+0000", 2024-02-05T04:53:54.753 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0038487930000000001 2024-02-05T04:53:54.753 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.753 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.754 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:53:54.754 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.140400+0000", 2024-02-05T04:53:54.754 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0021592690000000001 2024-02-05T04:53:54.754 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.754 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.754 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:54.754 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.140481+0000", 2024-02-05T04:53:54.754 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.1057000000000007e-05 2024-02-05T04:53:54.754 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.754 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:54.754 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.754 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.754 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.754 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:219 3.3s0 3:ce48de55:::smithi19639362-16:head [append 0~376832] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:53:54.754 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:29.134822+0000", 2024-02-05T04:53:54.755 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 25.612027069, 2024-02-05T04:53:54.755 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0088602019999999993, 2024-02-05T04:53:54.755 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:54.755 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:54.755 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:53:54.755 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:53:54.755 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:54.755 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 219 2024-02-05T04:53:54.755 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.755 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:54.755 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.755 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:54.755 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.134822+0000", 2024-02-05T04:53:54.755 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.755 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.756 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.756 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:54.756 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.134822+0000", 2024-02-05T04:53:54.756 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.756 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.756 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.756 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:54.756 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.134825+0000", 2024-02-05T04:53:54.756 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.1229999999999998e-06 2024-02-05T04:53:54.756 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.756 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.756 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:54.756 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.135417+0000", 2024-02-05T04:53:54.756 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00059254900000000005 2024-02-05T04:53:54.757 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.757 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.757 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:54.757 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.135430+0000", 2024-02-05T04:53:54.757 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.2663000000000001e-05 2024-02-05T04:53:54.757 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.757 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.757 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:54.757 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.135436+0000", 2024-02-05T04:53:54.757 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.6959999999999999e-06 2024-02-05T04:53:54.757 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.757 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.757 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:54.757 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.135834+0000", 2024-02-05T04:53:54.757 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000397382 2024-02-05T04:53:54.758 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.758 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.758 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:53:54.758 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.135967+0000", 2024-02-05T04:53:54.758 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00013333 2024-02-05T04:53:54.758 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.758 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.758 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:54.758 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.136753+0000", 2024-02-05T04:53:54.758 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00078585700000000005 2024-02-05T04:53:54.758 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.758 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.758 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:54.758 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.138286+0000", 2024-02-05T04:53:54.758 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0015333390000000001 2024-02-05T04:53:54.759 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.759 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.759 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:53:54.759 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.143611+0000", 2024-02-05T04:53:54.759 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0053249639999999997 2024-02-05T04:53:54.759 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.759 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.759 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:54.759 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.143683+0000", 2024-02-05T04:53:54.759 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.1298999999999996e-05 2024-02-05T04:53:54.759 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.759 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:54.759 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.759 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.759 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.760 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:220 3.3s0 3:ce48de55:::smithi19639362-16:head [setxattr _header (56) in=7b] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:53:54.760 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:29.135450+0000", 2024-02-05T04:53:54.760 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 25.611399147, 2024-02-05T04:53:54.760 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0084937420000000003, 2024-02-05T04:53:54.760 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:54.760 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:54.760 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:53:54.760 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:53:54.760 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:54.760 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 220 2024-02-05T04:53:54.760 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.760 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:54.760 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.760 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:54.761 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.135450+0000", 2024-02-05T04:53:54.761 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.761 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.761 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.761 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:54.761 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.135450+0000", 2024-02-05T04:53:54.761 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.761 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.761 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.761 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:54.761 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.135451+0000", 2024-02-05T04:53:54.761 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.4799999999999997e-07 2024-02-05T04:53:54.761 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.761 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.762 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:54.762 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.135454+0000", 2024-02-05T04:53:54.762 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.3539999999999998e-06 2024-02-05T04:53:54.762 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.762 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.762 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:54.762 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.135458+0000", 2024-02-05T04:53:54.762 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.1509999999999997e-06 2024-02-05T04:53:54.762 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.762 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.762 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:54.762 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.135461+0000", 2024-02-05T04:53:54.762 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.1650000000000002e-06 2024-02-05T04:53:54.762 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.762 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.763 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:54.763 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.137469+0000", 2024-02-05T04:53:54.763 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0020076410000000001 2024-02-05T04:53:54.763 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.763 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.763 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:53:54.763 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.137586+0000", 2024-02-05T04:53:54.763 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000117599 2024-02-05T04:53:54.763 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.763 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.763 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:54.763 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.138018+0000", 2024-02-05T04:53:54.763 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00043188200000000002 2024-02-05T04:53:54.763 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.764 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.764 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:54.764 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.140234+0000", 2024-02-05T04:53:54.764 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0022155159999999998 2024-02-05T04:53:54.764 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.764 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.764 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:53:54.764 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.143857+0000", 2024-02-05T04:53:54.764 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0036236749999999998 2024-02-05T04:53:54.764 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.764 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.764 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:54.764 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.143944+0000", 2024-02-05T04:53:54.764 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.6910999999999995e-05 2024-02-05T04:53:54.764 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.765 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:54.765 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.765 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.765 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.765 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:221 3.3s0 3:ce48de55:::smithi19639362-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:53:54.765 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:29.135468+0000", 2024-02-05T04:53:54.765 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 25.611382009, 2024-02-05T04:53:54.765 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.010019444000000001, 2024-02-05T04:53:54.765 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:54.765 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "started", 2024-02-05T04:53:54.765 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:53:54.765 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:53:54.765 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:54.765 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 221 2024-02-05T04:53:54.765 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.766 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:54.766 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.766 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:54.766 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.135468+0000", 2024-02-05T04:53:54.766 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.766 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.766 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.766 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:54.766 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.135468+0000", 2024-02-05T04:53:54.766 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.766 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.766 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.766 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:54.766 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.135468+0000", 2024-02-05T04:53:54.766 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.4299999999999999e-07 2024-02-05T04:53:54.767 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.767 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.767 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:54.767 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.135469+0000", 2024-02-05T04:53:54.767 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.457e-06 2024-02-05T04:53:54.767 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.767 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.767 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:54.767 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.135475+0000", 2024-02-05T04:53:54.767 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.9939999999999997e-06 2024-02-05T04:53:54.767 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.767 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.767 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:54.767 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.135480+0000", 2024-02-05T04:53:54.767 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.0989999999999998e-06 2024-02-05T04:53:54.767 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.768 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.768 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:54.768 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.138320+0000", 2024-02-05T04:53:54.768 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0028394739999999998 2024-02-05T04:53:54.768 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.768 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.768 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for rw locks", 2024-02-05T04:53:54.768 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.138401+0000", 2024-02-05T04:53:54.768 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.0966999999999999e-05 2024-02-05T04:53:54.768 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.768 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.768 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:54.768 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.144381+0000", 2024-02-05T04:53:54.768 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0059798389999999998 2024-02-05T04:53:54.769 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.769 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.769 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:53:54.769 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.144442+0000", 2024-02-05T04:53:54.769 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.1311999999999998e-05 2024-02-05T04:53:54.769 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.769 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.769 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:54.769 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.145487+0000", 2024-02-05T04:53:54.769 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0010450590000000001 2024-02-05T04:53:54.769 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.769 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:54.770 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.770 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.770 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.770 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(osd.7.23:1 3.3s0 3:c5eac4f6:::smithi19639362-39:head [assert-version v6,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:53:54.770 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:29.137174+0000", 2024-02-05T04:53:54.770 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 25.609675783, 2024-02-05T04:53:54.770 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.006190615, 2024-02-05T04:53:54.770 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:54.770 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "started", 2024-02-05T04:53:54.770 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:54.770 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.770 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:54.770 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.137174+0000", 2024-02-05T04:53:54.770 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.770 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.771 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.771 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:54.771 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.137174+0000", 2024-02-05T04:53:54.771 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.771 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.771 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.771 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:54.771 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.137175+0000", 2024-02-05T04:53:54.771 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.6700000000000001e-06 2024-02-05T04:53:54.771 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.771 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.771 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:54.771 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.137183+0000", 2024-02-05T04:53:54.771 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.4599999999999997e-06 2024-02-05T04:53:54.771 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.772 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.772 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:54.772 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.137196+0000", 2024-02-05T04:53:54.772 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.3139000000000001e-05 2024-02-05T04:53:54.772 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.772 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.772 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:54.772 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.137205+0000", 2024-02-05T04:53:54.772 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.6389999999999998e-06 2024-02-05T04:53:54.772 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.772 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.772 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:54.772 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.138803+0000", 2024-02-05T04:53:54.772 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0015985280000000001 2024-02-05T04:53:54.772 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.773 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.773 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:53:54.773 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.138876+0000", 2024-02-05T04:53:54.773 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.2908000000000001e-05 2024-02-05T04:53:54.773 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.773 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.773 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:54.773 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.143364+0000", 2024-02-05T04:53:54.773 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0044882710000000003 2024-02-05T04:53:54.773 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.773 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:54.773 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.773 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.773 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.773 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.7s4 37/31 ECSubWrite(tid=85, reqid=client.4634.0:243, at_version=0'0, trim_to=0'0, roll_forward_to=37'6))", 2024-02-05T04:53:54.774 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:29.191315+0000", 2024-02-05T04:53:54.774 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 25.555534684000001, 2024-02-05T04:53:54.774 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0057709909999999996, 2024-02-05T04:53:54.774 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:54.774 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:54.774 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:54.774 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.774 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:54.774 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.191315+0000", 2024-02-05T04:53:54.774 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.774 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.774 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.774 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:54.774 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.191315+0000", 2024-02-05T04:53:54.774 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.775 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.775 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.775 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:54.775 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.191316+0000", 2024-02-05T04:53:54.775 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.8899999999999998e-07 2024-02-05T04:53:54.775 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.775 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.775 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:54.775 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.191338+0000", 2024-02-05T04:53:54.775 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.1903e-05 2024-02-05T04:53:54.775 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.775 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.775 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:54.775 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.191352+0000", 2024-02-05T04:53:54.776 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.4817999999999999e-05 2024-02-05T04:53:54.776 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.776 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.776 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:54.776 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.191361+0000", 2024-02-05T04:53:54.776 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.4090000000000003e-06 2024-02-05T04:53:54.776 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.776 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.776 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:54.776 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.191440+0000", 2024-02-05T04:53:54.776 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.9468000000000006e-05 2024-02-05T04:53:54.776 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.776 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.776 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:54.776 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.191449+0000", 2024-02-05T04:53:54.778 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.9050000000000003e-06 2024-02-05T04:53:54.778 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.778 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.778 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:54.778 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.197039+0000", 2024-02-05T04:53:54.778 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0055895720000000001 2024-02-05T04:53:54.778 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.778 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.779 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:54.779 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.197086+0000", 2024-02-05T04:53:54.779 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.7027000000000002e-05 2024-02-05T04:53:54.779 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.779 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:54.779 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.779 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.779 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.779 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.1s2 37/31 ECSubWrite(tid=86, reqid=client.4634.0:247, at_version=37'15, trim_to=0'0, roll_forward_to=35'13))", 2024-02-05T04:53:54.779 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:29.192941+0000", 2024-02-05T04:53:54.779 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 25.553908453999998, 2024-02-05T04:53:54.779 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0055093850000000003, 2024-02-05T04:53:54.779 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:54.779 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:54.780 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:54.780 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.780 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:54.780 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.192941+0000", 2024-02-05T04:53:54.780 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.780 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.780 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.780 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:54.780 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.192941+0000", 2024-02-05T04:53:54.780 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.780 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.780 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.780 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:54.780 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.192948+0000", 2024-02-05T04:53:54.781 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.3000000000000004e-06 2024-02-05T04:53:54.781 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.781 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.781 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:54.781 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.193266+0000", 2024-02-05T04:53:54.781 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00031747399999999999 2024-02-05T04:53:54.781 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.781 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.781 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:54.781 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.193282+0000", 2024-02-05T04:53:54.781 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.5716e-05 2024-02-05T04:53:54.781 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.781 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.781 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:54.781 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.193291+0000", 2024-02-05T04:53:54.782 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.1090000000000004e-06 2024-02-05T04:53:54.782 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.782 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.782 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:54.782 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.193347+0000", 2024-02-05T04:53:54.782 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.6628000000000001e-05 2024-02-05T04:53:54.782 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.782 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.782 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:54.782 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.193353+0000", 2024-02-05T04:53:54.782 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.9220000000000002e-06 2024-02-05T04:53:54.782 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.782 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.782 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:54.782 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.198406+0000", 2024-02-05T04:53:54.782 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00505303 2024-02-05T04:53:54.783 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.783 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.783 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:54.783 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.198450+0000", 2024-02-05T04:53:54.783 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.4206000000000003e-05 2024-02-05T04:53:54.783 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.783 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:54.783 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.783 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.783 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.783 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:261 3.3s0 3:c5eac4f6:::smithi19639362-39:head [read 0~638976,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-02-05T04:53:54.783 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:30.132041+0000", 2024-02-05T04:53:54.783 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 24.614808193999998, 2024-02-05T04:53:54.783 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0057721969999999997, 2024-02-05T04:53:54.784 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:54.784 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "started", 2024-02-05T04:53:54.784 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:53:54.784 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:53:54.784 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:54.784 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 261 2024-02-05T04:53:54.784 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.784 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:54.784 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.784 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:54.784 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.132041+0000", 2024-02-05T04:53:54.784 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.784 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.784 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.784 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:54.785 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.132041+0000", 2024-02-05T04:53:54.785 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.785 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.785 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.785 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:54.785 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.132042+0000", 2024-02-05T04:53:54.785 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.5000000000000002e-07 2024-02-05T04:53:54.785 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.785 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.785 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:54.785 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.132046+0000", 2024-02-05T04:53:54.785 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.4680000000000003e-06 2024-02-05T04:53:54.785 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.785 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.785 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:54.786 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.132054+0000", 2024-02-05T04:53:54.786 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.8609999999999998e-06 2024-02-05T04:53:54.786 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.786 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.786 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:54.786 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.132060+0000", 2024-02-05T04:53:54.786 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.711e-06 2024-02-05T04:53:54.786 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.786 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.786 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:54.786 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.133280+0000", 2024-02-05T04:53:54.786 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.001219574 2024-02-05T04:53:54.786 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.786 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.786 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:53:54.786 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.133394+0000", 2024-02-05T04:53:54.787 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000114666 2024-02-05T04:53:54.787 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.787 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.787 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:54.787 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.137814+0000", 2024-02-05T04:53:54.787 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0044192670000000002 2024-02-05T04:53:54.787 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.787 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:54.787 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.787 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.787 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.787 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.4s5 38/31 ECSubWrite(tid=28, reqid=osd.1.0:27, at_version=0'0, trim_to=0'0, roll_forward_to=38'16))", 2024-02-05T04:53:54.787 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:30.139653+0000", 2024-02-05T04:53:54.787 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 24.607196842, 2024-02-05T04:53:54.787 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0060967790000000001, 2024-02-05T04:53:54.787 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:54.788 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:54.788 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:54.788 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.788 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:54.788 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.139653+0000", 2024-02-05T04:53:54.788 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.788 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.788 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.788 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:54.788 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.139653+0000", 2024-02-05T04:53:54.788 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.788 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.788 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.788 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:54.788 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.139653+0000", 2024-02-05T04:53:54.789 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.3999999999999997e-07 2024-02-05T04:53:54.789 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.789 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.789 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:54.789 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.139661+0000", 2024-02-05T04:53:54.789 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.2679999999999994e-06 2024-02-05T04:53:54.789 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.789 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.789 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:54.789 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.139670+0000", 2024-02-05T04:53:54.789 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.7779999999999996e-06 2024-02-05T04:53:54.789 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.789 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.789 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:54.789 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.139679+0000", 2024-02-05T04:53:54.790 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.6610000000000006e-06 2024-02-05T04:53:54.790 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.790 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.790 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:54.790 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.139801+0000", 2024-02-05T04:53:54.790 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000122149 2024-02-05T04:53:54.790 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.790 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.790 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:54.790 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.139812+0000", 2024-02-05T04:53:54.790 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0902e-05 2024-02-05T04:53:54.790 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.790 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.790 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:54.791 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.145702+0000", 2024-02-05T04:53:54.791 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.005890658 2024-02-05T04:53:54.791 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.791 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.791 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:54.791 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.145749+0000", 2024-02-05T04:53:54.791 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.7023e-05 2024-02-05T04:53:54.791 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.791 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:54.791 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.791 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.791 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.791 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.1s2 38/31 ECSubWrite(tid=89, reqid=osd.13.0:89, at_version=38'17, trim_to=0'0, roll_forward_to=37'15))", 2024-02-05T04:53:54.791 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:30.140160+0000", 2024-02-05T04:53:54.791 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 24.60668944, 2024-02-05T04:53:54.791 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0056285220000000004, 2024-02-05T04:53:54.792 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:54.792 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:54.792 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:54.792 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.792 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:54.792 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.140160+0000", 2024-02-05T04:53:54.792 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.792 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.792 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.792 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:54.792 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.140160+0000", 2024-02-05T04:53:54.792 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.792 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.792 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.792 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:54.793 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.140160+0000", 2024-02-05T04:53:54.793 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.0100000000000001e-07 2024-02-05T04:53:54.793 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.793 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.793 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:54.793 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.140183+0000", 2024-02-05T04:53:54.793 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.2708999999999999e-05 2024-02-05T04:53:54.793 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.793 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.793 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:54.793 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.140197+0000", 2024-02-05T04:53:54.793 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.3867e-05 2024-02-05T04:53:54.793 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.793 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.793 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:54.794 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.140206+0000", 2024-02-05T04:53:54.794 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.1900000000000001e-06 2024-02-05T04:53:54.794 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.794 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.794 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:54.794 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.140303+0000", 2024-02-05T04:53:54.794 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.6751999999999998e-05 2024-02-05T04:53:54.794 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.794 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.794 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:54.794 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.140314+0000", 2024-02-05T04:53:54.794 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1260000000000001e-05 2024-02-05T04:53:54.794 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.794 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.794 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:54.794 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.145762+0000", 2024-02-05T04:53:54.795 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0054476330000000003 2024-02-05T04:53:54.795 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.795 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.795 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:54.795 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.145789+0000", 2024-02-05T04:53:54.795 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.6910000000000002e-05 2024-02-05T04:53:54.795 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.795 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:54.795 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.795 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.795 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.795 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.6s2 38/31 ECSubWrite(tid=88, reqid=client.4634.0:262, at_version=38'9, trim_to=0'0, roll_forward_to=37'8))", 2024-02-05T04:53:54.795 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:30.140280+0000", 2024-02-05T04:53:54.795 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 24.606569288999999, 2024-02-05T04:53:54.795 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0054517059999999997, 2024-02-05T04:53:54.796 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:54.796 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:54.796 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:54.796 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.796 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:54.796 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.140280+0000", 2024-02-05T04:53:54.796 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.796 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.796 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.796 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:54.796 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.140280+0000", 2024-02-05T04:53:54.796 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.796 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.796 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.796 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:54.797 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.140281+0000", 2024-02-05T04:53:54.797 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.7299999999999996e-07 2024-02-05T04:53:54.797 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.797 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.797 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:54.797 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.140304+0000", 2024-02-05T04:53:54.797 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.3595e-05 2024-02-05T04:53:54.797 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.797 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.797 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:54.797 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.140357+0000", 2024-02-05T04:53:54.798 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.2485999999999999e-05 2024-02-05T04:53:54.798 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.798 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.798 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:54.798 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.140366+0000", 2024-02-05T04:53:54.798 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.6150000000000003e-06 2024-02-05T04:53:54.798 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.798 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.798 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:54.798 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.140552+0000", 2024-02-05T04:53:54.798 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00018572099999999999 2024-02-05T04:53:54.798 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.798 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.798 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:54.798 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.140560+0000", 2024-02-05T04:53:54.798 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.2239999999999995e-06 2024-02-05T04:53:54.799 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.799 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.799 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:54.799 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.145684+0000", 2024-02-05T04:53:54.799 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0051238170000000001 2024-02-05T04:53:54.799 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.799 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.799 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:54.799 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.145732+0000", 2024-02-05T04:53:54.799 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.7675e-05 2024-02-05T04:53:54.799 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.799 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:54.799 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.799 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.799 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.800 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:390 3.8s0 3:170878b2:::smithi19639362-30:head [copy-from ver 20] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-02-05T04:53:54.800 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:34.171935+0000", 2024-02-05T04:53:54.800 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 20.574914049, 2024-02-05T04:53:54.800 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.014076916, 2024-02-05T04:53:54.800 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:54.800 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:54.800 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:53:54.800 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:53:54.800 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:54.800 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 390 2024-02-05T04:53:54.800 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.800 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:54.800 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.800 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:54.800 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.171935+0000", 2024-02-05T04:53:54.801 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.801 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.801 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.801 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:54.801 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.171935+0000", 2024-02-05T04:53:54.801 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.801 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.801 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.801 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:54.801 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.171937+0000", 2024-02-05T04:53:54.801 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.8959999999999999e-06 2024-02-05T04:53:54.801 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.801 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.801 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:54.801 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.171944+0000", 2024-02-05T04:53:54.801 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.5729999999999996e-06 2024-02-05T04:53:54.802 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.802 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.802 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:54.802 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.171956+0000", 2024-02-05T04:53:54.802 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.2499e-05 2024-02-05T04:53:54.802 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.802 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.802 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:54.802 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.171964+0000", 2024-02-05T04:53:54.802 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.6450000000000005e-06 2024-02-05T04:53:54.802 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.802 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.802 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:54.802 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.172025+0000", 2024-02-05T04:53:54.802 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.0713e-05 2024-02-05T04:53:54.803 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.803 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.803 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:53:54.803 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.172112+0000", 2024-02-05T04:53:54.803 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.7421000000000002e-05 2024-02-05T04:53:54.803 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.803 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.803 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:54.803 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.180774+0000", 2024-02-05T04:53:54.803 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.008661986 2024-02-05T04:53:54.803 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.803 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.803 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:54.803 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.182980+0000", 2024-02-05T04:53:54.803 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0022060389999999999 2024-02-05T04:53:54.804 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.804 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.804 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:53:54.804 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.185921+0000", 2024-02-05T04:53:54.804 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0029412179999999998 2024-02-05T04:53:54.804 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.804 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.804 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:54.804 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.186012+0000", 2024-02-05T04:53:54.804 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.0926000000000003e-05 2024-02-05T04:53:54.804 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.804 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:54.804 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.804 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.804 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.805 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:391 3.8s0 3:170878b2:::smithi19639362-30:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-02-05T04:53:54.805 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:34.172014+0000", 2024-02-05T04:53:54.805 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 20.574835566000001, 2024-02-05T04:53:54.805 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.010857311999999999, 2024-02-05T04:53:54.805 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:54.805 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "started", 2024-02-05T04:53:54.806 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:53:54.806 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:53:54.806 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:54.806 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 391 2024-02-05T04:53:54.806 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.806 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:54.806 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.807 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:54.807 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.172014+0000", 2024-02-05T04:53:54.807 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.808 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.808 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.809 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:54.809 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.172014+0000", 2024-02-05T04:53:54.809 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.809 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.809 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.810 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:54.810 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.172015+0000", 2024-02-05T04:53:54.810 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.5199999999999996e-07 2024-02-05T04:53:54.810 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.810 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.810 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:54.811 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.172018+0000", 2024-02-05T04:53:54.811 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.5319999999999998e-06 2024-02-05T04:53:54.811 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.811 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.811 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:54.811 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.172028+0000", 2024-02-05T04:53:54.811 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.5990000000000008e-06 2024-02-05T04:53:54.812 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.812 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.812 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:54.813 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.172034+0000", 2024-02-05T04:53:54.813 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.742e-06 2024-02-05T04:53:54.813 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.813 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.813 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:54.813 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.172290+0000", 2024-02-05T04:53:54.813 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00025550300000000002 2024-02-05T04:53:54.813 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.814 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.814 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for blocked object", 2024-02-05T04:53:54.814 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.172342+0000", 2024-02-05T04:53:54.814 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.1657999999999999e-05 2024-02-05T04:53:54.814 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.814 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.814 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:54.814 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.182668+0000", 2024-02-05T04:53:54.814 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.010326223000000001 2024-02-05T04:53:54.815 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.815 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.815 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:53:54.815 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.182773+0000", 2024-02-05T04:53:54.815 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00010530399999999999 2024-02-05T04:53:54.815 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.815 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.815 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:54.815 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.182871+0000", 2024-02-05T04:53:54.815 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.8098999999999997e-05 2024-02-05T04:53:54.815 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.815 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:54.815 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.815 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.816 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.816 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.1s2 43/31 ECSubWrite(tid=140, reqid=osd.13.0:140, at_version=43'29, trim_to=0'0, roll_forward_to=42'27))", 2024-02-05T04:53:54.816 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:35.182198+0000", 2024-02-05T04:53:54.816 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 19.564651202, 2024-02-05T04:53:54.816 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0053024429999999996, 2024-02-05T04:53:54.816 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:54.816 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:54.816 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:54.816 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.816 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:54.816 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.182198+0000", 2024-02-05T04:53:54.816 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.816 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.817 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.817 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:54.817 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.182198+0000", 2024-02-05T04:53:54.817 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.817 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.817 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.817 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:54.817 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.182199+0000", 2024-02-05T04:53:54.817 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.2e-07 2024-02-05T04:53:54.817 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.817 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.817 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:54.817 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.182212+0000", 2024-02-05T04:53:54.818 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.311e-05 2024-02-05T04:53:54.818 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.818 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.818 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:54.818 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.182219+0000", 2024-02-05T04:53:54.818 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.4880000000000001e-06 2024-02-05T04:53:54.818 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.818 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.818 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:54.818 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.182232+0000", 2024-02-05T04:53:54.818 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.2330000000000001e-05 2024-02-05T04:53:54.818 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.818 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.818 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:54.818 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.182291+0000", 2024-02-05T04:53:54.819 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.9080999999999999e-05 2024-02-05T04:53:54.819 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.819 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.819 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:54.819 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.182300+0000", 2024-02-05T04:53:54.819 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.9080000000000001e-06 2024-02-05T04:53:54.819 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.819 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.819 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:54.819 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.187398+0000", 2024-02-05T04:53:54.819 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.005097964 2024-02-05T04:53:54.819 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.819 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.819 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:54.819 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.187501+0000", 2024-02-05T04:53:54.819 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00010304199999999999 2024-02-05T04:53:54.820 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.820 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:54.820 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.820 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.820 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.820 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.4s5 43/31 ECSubWrite(tid=46, reqid=osd.1.0:46, at_version=43'29, trim_to=0'0, roll_forward_to=42'27))", 2024-02-05T04:53:54.820 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:35.182637+0000", 2024-02-05T04:53:54.820 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 19.564212759, 2024-02-05T04:53:54.820 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0075236640000000002, 2024-02-05T04:53:54.820 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:54.820 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:54.820 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:54.820 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.820 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:54.820 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.182637+0000", 2024-02-05T04:53:54.821 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.821 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.821 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.821 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:54.821 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.182637+0000", 2024-02-05T04:53:54.821 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.821 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.821 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.821 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:54.821 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.182637+0000", 2024-02-05T04:53:54.821 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.3000000000000002e-07 2024-02-05T04:53:54.821 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.821 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.822 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:54.822 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.182698+0000", 2024-02-05T04:53:54.822 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.1062000000000006e-05 2024-02-05T04:53:54.822 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.822 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.822 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:54.822 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.182708+0000", 2024-02-05T04:53:54.822 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0186999999999999e-05 2024-02-05T04:53:54.822 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.822 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.822 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:54.822 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.182715+0000", 2024-02-05T04:53:54.822 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.0040000000000002e-06 2024-02-05T04:53:54.822 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.823 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.823 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:54.823 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.182796+0000", 2024-02-05T04:53:54.823 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.0364000000000005e-05 2024-02-05T04:53:54.823 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.823 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.823 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:54.823 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.182805+0000", 2024-02-05T04:53:54.823 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.5990000000000008e-06 2024-02-05T04:53:54.823 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.823 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.823 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:54.823 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.190124+0000", 2024-02-05T04:53:54.824 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0073191009999999997 2024-02-05T04:53:54.824 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.824 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.824 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:54.824 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.190160+0000", 2024-02-05T04:53:54.824 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.6016999999999998e-05 2024-02-05T04:53:54.824 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.824 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:54.824 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.824 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.824 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.825 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.1s2 43/31 ECSubWrite(tid=142, reqid=osd.13.0:142, at_version=43'31, trim_to=0'0, roll_forward_to=42'27))", 2024-02-05T04:53:54.825 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:35.183951+0000", 2024-02-05T04:53:54.825 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 19.562898409999999, 2024-02-05T04:53:54.825 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0062194069999999997, 2024-02-05T04:53:54.825 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:54.825 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:54.825 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:54.825 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.826 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:54.826 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.183951+0000", 2024-02-05T04:53:54.826 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.826 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.826 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.826 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:54.826 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.183951+0000", 2024-02-05T04:53:54.826 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.826 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.826 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.826 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:54.826 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.183952+0000", 2024-02-05T04:53:54.826 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.06e-07 2024-02-05T04:53:54.826 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.826 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.827 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:54.827 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.183966+0000", 2024-02-05T04:53:54.827 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.4335000000000001e-05 2024-02-05T04:53:54.827 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.827 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.827 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:54.827 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.183975+0000", 2024-02-05T04:53:54.827 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.2629999999999992e-06 2024-02-05T04:53:54.827 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.827 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.827 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:54.827 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.183986+0000", 2024-02-05T04:53:54.827 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1204e-05 2024-02-05T04:53:54.827 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.828 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.828 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:54.828 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.184032+0000", 2024-02-05T04:53:54.828 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.5643999999999999e-05 2024-02-05T04:53:54.828 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.828 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.828 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:54.828 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.184038+0000", 2024-02-05T04:53:54.828 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.2249999999999997e-06 2024-02-05T04:53:54.828 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.828 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.828 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:54.828 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.190128+0000", 2024-02-05T04:53:54.828 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0060899819999999999 2024-02-05T04:53:54.829 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.829 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.829 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:54.829 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.190171+0000", 2024-02-05T04:53:54.829 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.2247999999999999e-05 2024-02-05T04:53:54.829 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.829 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:54.829 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.829 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.829 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.829 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:419 3.3s0 3:ce48de55:::smithi19639362-16:head [create,setxattr Rj\\mZOfQXo^b_ZWWfqN]m[iPt^aRVgsYNetVd`VO (19508) in=40b,setxattr [adnUNmYmWdN__]tR]msPnVSRiY^psh`dY\\Z (630) in=36b,setxattr cPsWXtVeXTPagdn_WVZbVcV\\UjSZUqsVlPZbY` (630) in=38b,setxattr pqbrReOSsgZlmftc]`Z`]USaXjsacf`fao_VP`eU (1190) in=40b,setxattr aYfgr^q[\\aPg_aZsakfUfsdhYbra^QXjlkrpdY\\\\ (138) in=40b,setxattr e\\XXWiVknihNQQ[OsThQUbRmtprrfYPeRX[[mRcQ (1719) in=40b,setxattr TNQYR^jgR`pPaYitmrcliV[dttobncTntcfTlQkt (8420) in=40b,setxattr W_aq[RVsaUpQtX_RpV^Vpqrg`QgQViPeVaRqZSmf (525) in=40b,setxattr RWdkeaaUP\\PZteTNscS^UVRtjo`oqqdNqe]cdZUa (245) in=40b,setxattr R^`al]TqifatYOiOU]p`ZegfXmNsXVaZcdd]b]YX (1435) in=40b,setxattr p]OS_Znl[dNa^qTn`WW[j\\dtgbNac_bijYmg (630) in=36b,setxattr n_jr[tV[VhYPqU]UenciobhtYfY^a^c\\hd^YrcQ[ (13472) in=40b,setxattr gReQSUg`cdl_ilU_lo[ggg\\fZsVQY[PgY]jPsZ_U (14664) in=40b,setxattr cNWaQrsdt^dic_qN_^mTXmc`_X]YOTXNk[WldRe` (523) in=40b,setxattr tZ]grhlaq]kUO[k^d^dr``XNio^^Oaebpj\\[Ytdq (33) in=40b,setxattr jqS_j^cefaXab`pQoPXfRkSmdako`fZ_heb^_fma (1191) in=40b,setxattr dgXPi (630) in=5b,setxattr Or_kkpbnNkTrQ\\apUaUsZR\\XW^nabhaiQ\\]gOjR\\ (525) in=40b,setxattr Rf`aR[fdtjQ\\NaqZke]l_rUssrVpOPXYWpXkl_kd (2876) in=40b,setxattr _sUSYWYUmOrlUVhZajZeYbbnsdra[]`Q`^Ogea[m (631) in=40b] snapc 9=[9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-02-05T04:53:54.829 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:38.943383+0000", 2024-02-05T04:53:54.829 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 15.803466903, 2024-02-05T04:53:54.829 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0065115019999999997, 2024-02-05T04:53:54.830 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:54.830 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:54.830 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:53:54.830 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:53:54.830 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:54.830 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 419 2024-02-05T04:53:54.830 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.830 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:54.830 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.830 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:54.830 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:38.943383+0000", 2024-02-05T04:53:54.830 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.831 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.831 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.831 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:54.831 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:38.943383+0000", 2024-02-05T04:53:54.831 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.831 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.831 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.831 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:54.831 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:38.943384+0000", 2024-02-05T04:53:54.831 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.621e-06 2024-02-05T04:53:54.831 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.831 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.831 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:54.831 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:38.943495+0000", 2024-02-05T04:53:54.832 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000110534 2024-02-05T04:53:54.832 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.832 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.832 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:54.832 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:38.943513+0000", 2024-02-05T04:53:54.832 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.7810000000000001e-05 2024-02-05T04:53:54.832 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.832 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.832 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:54.832 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:38.943524+0000", 2024-02-05T04:53:54.832 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1219e-05 2024-02-05T04:53:54.832 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.832 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.832 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:54.833 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:38.943595+0000", 2024-02-05T04:53:54.833 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.1056000000000005e-05 2024-02-05T04:53:54.833 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.833 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.833 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:53:54.833 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:38.943731+0000", 2024-02-05T04:53:54.833 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00013657599999999999 2024-02-05T04:53:54.833 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.833 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.833 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:54.833 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:38.944552+0000", 2024-02-05T04:53:54.833 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000820896 2024-02-05T04:53:54.833 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.833 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.834 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:54.834 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:38.948061+0000", 2024-02-05T04:53:54.834 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.003508429 2024-02-05T04:53:54.834 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.834 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.834 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:53:54.834 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:38.949791+0000", 2024-02-05T04:53:54.834 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0017302800000000001 2024-02-05T04:53:54.834 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.834 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.834 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:54.834 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:38.949894+0000", 2024-02-05T04:53:54.834 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000103081 2024-02-05T04:53:54.834 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.835 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:54.835 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.835 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.835 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.835 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.7s4 57/31 ECSubWrite(tid=238, reqid=osd.13.0:238, at_version=57'27, trim_to=0'0, roll_forward_to=52'25))", 2024-02-05T04:53:54.835 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:48.947200+0000", 2024-02-05T04:53:54.835 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 5.7996493290000002, 2024-02-05T04:53:54.836 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0053445769999999997, 2024-02-05T04:53:54.836 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:54.836 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:54.836 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:54.836 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.836 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:54.836 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:48.947200+0000", 2024-02-05T04:53:54.836 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.836 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.836 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.836 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:54.836 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:48.947200+0000", 2024-02-05T04:53:54.836 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.836 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.837 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.837 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:54.837 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:48.947201+0000", 2024-02-05T04:53:54.837 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.89e-07 2024-02-05T04:53:54.837 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.837 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.837 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:54.837 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:48.947223+0000", 2024-02-05T04:53:54.837 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.2255e-05 2024-02-05T04:53:54.837 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.837 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.837 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:54.837 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:48.947234+0000", 2024-02-05T04:53:54.838 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0913e-05 2024-02-05T04:53:54.838 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.838 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.838 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:54.838 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:48.947243+0000", 2024-02-05T04:53:54.838 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.7600000000000008e-06 2024-02-05T04:53:54.838 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.838 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.838 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:54.838 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:48.947395+0000", 2024-02-05T04:53:54.838 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000152509 2024-02-05T04:53:54.838 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.838 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.838 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:54.838 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:48.947406+0000", 2024-02-05T04:53:54.839 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0631999999999999e-05 2024-02-05T04:53:54.839 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.839 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.839 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:54.839 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:48.952512+0000", 2024-02-05T04:53:54.839 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0051067400000000002 2024-02-05T04:53:54.839 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.839 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.839 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:54.839 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:48.952545+0000", 2024-02-05T04:53:54.839 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.2379000000000003e-05 2024-02-05T04:53:54.839 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.839 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:54.839 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.839 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.840 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.840 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.7s4 57/31 ECSubWrite(tid=239, reqid=osd.13.0:239, at_version=57'29, trim_to=0'0, roll_forward_to=52'25))", 2024-02-05T04:53:54.840 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:48.947256+0000", 2024-02-05T04:53:54.840 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 5.7995934760000001, 2024-02-05T04:53:54.840 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0054473569999999999, 2024-02-05T04:53:54.840 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:54.840 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:54.840 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:54.840 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.840 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:54.840 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:48.947256+0000", 2024-02-05T04:53:54.840 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.840 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.840 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.841 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:54.841 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:48.947256+0000", 2024-02-05T04:53:54.841 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.841 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.841 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.841 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:54.841 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:48.947256+0000", 2024-02-05T04:53:54.841 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.2000000000000001e-07 2024-02-05T04:53:54.841 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.841 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.841 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:54.841 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:48.947276+0000", 2024-02-05T04:53:54.841 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.0111000000000001e-05 2024-02-05T04:53:54.841 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.841 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.842 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:54.842 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:48.947285+0000", 2024-02-05T04:53:54.842 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.3999999999999992e-06 2024-02-05T04:53:54.842 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.842 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.842 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:54.842 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:48.947291+0000", 2024-02-05T04:53:54.842 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.4150000000000004e-06 2024-02-05T04:53:54.842 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.842 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.842 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:54.842 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:48.947749+0000", 2024-02-05T04:53:54.842 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000458092 2024-02-05T04:53:54.842 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.842 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.843 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:54.843 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:48.947759+0000", 2024-02-05T04:53:54.843 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.1840000000000005e-06 2024-02-05T04:53:54.843 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.843 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.843 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:54.843 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:48.952680+0000", 2024-02-05T04:53:54.843 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0049218040000000001 2024-02-05T04:53:54.843 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.843 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.843 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:54.843 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:48.952703+0000", 2024-02-05T04:53:54.843 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.3031e-05 2024-02-05T04:53:54.843 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.844 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:54.844 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.844 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.844 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.844 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.4s5 58/57 ECSubWrite(tid=108, reqid=osd.1.0:108, at_version=58'61, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:53:54.844 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:49.955832+0000", 2024-02-05T04:53:54.844 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 4.7910177840000001, 2024-02-05T04:53:54.844 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0057648270000000001, 2024-02-05T04:53:54.844 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:54.844 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:54.844 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:54.844 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.844 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:54.844 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.955832+0000", 2024-02-05T04:53:54.844 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.845 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.845 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.845 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:54.845 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.955832+0000", 2024-02-05T04:53:54.845 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:54.845 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.845 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.845 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:54.845 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.955833+0000", 2024-02-05T04:53:54.845 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0359999999999999e-06 2024-02-05T04:53:54.845 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.845 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.845 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:54.845 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.955863+0000", 2024-02-05T04:53:54.846 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.0729999999999999e-05 2024-02-05T04:53:54.846 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.846 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.846 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:54.846 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.955931+0000", 2024-02-05T04:53:54.846 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.7614999999999996e-05 2024-02-05T04:53:54.846 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.846 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.846 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:54.846 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.955942+0000", 2024-02-05T04:53:54.846 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1035e-05 2024-02-05T04:53:54.846 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.846 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.846 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:54.847 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956045+0000", 2024-02-05T04:53:54.847 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000102808 2024-02-05T04:53:54.847 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.847 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.847 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:54.847 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956056+0000", 2024-02-05T04:53:54.847 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1479e-05 2024-02-05T04:53:54.847 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.847 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.847 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:54.847 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.961545+0000", 2024-02-05T04:53:54.847 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.005488995 2024-02-05T04:53:54.847 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:54.847 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:54.847 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:54.848 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.961597+0000", 2024-02-05T04:53:54.848 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.1128999999999998e-05 2024-02-05T04:53:54.848 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.848 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:54.848 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.848 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:54.848 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:54.848 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-02-05T04:53:54.850 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-02-05T04:53:54.850 INFO:tasks.ceph.osd.14.smithi150.stderr:2024-02-05T04:53:54.848+0000 7f5f81083640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 34020) UID: 0 2024-02-05T04:53:54.951 INFO:tasks.ceph.osd.13.smithi091.stderr:2024-02-05T04:53:54.950+0000 7f526fd4d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 34281) UID: 0 2024-02-05T04:53:54.977 INFO:tasks.rados.rados.0.smithi196.stdout:232: finishing write tid 3 to smithi19639362-4 2024-02-05T04:53:54.978 INFO:tasks.rados.rados.0.smithi196.stdout:232: finishing write tid 4 to smithi19639362-4 2024-02-05T04:53:54.978 INFO:tasks.rados.rados.0.smithi196.stdout:232: finishing write tid 5 to smithi19639362-4 2024-02-05T04:53:54.978 INFO:tasks.rados.rados.0.smithi196.stdout:232: oid 4 updating version 0 to 23 2024-02-05T04:53:54.978 INFO:tasks.rados.rados.0.smithi196.stdout:232: oid 4 version 23 is already newer than 22 2024-02-05T04:53:54.978 INFO:tasks.rados.rados.0.smithi196.stdout:232: oid 4 version 23 is already newer than 20 2024-02-05T04:53:54.978 INFO:tasks.rados.rados.0.smithi196.stdout:232: oid 4 version 23 is already newer than 21 2024-02-05T04:53:54.978 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 4 v 23 (ObjNum 113 snap 18 seq_num 113) dirty exists 2024-02-05T04:53:54.978 INFO:tasks.rados.rados.0.smithi196.stdout:232: left oid 4 (ObjNum 113 snap 18 seq_num 113) 2024-02-05T04:53:54.978 INFO:tasks.rados.rados.0.smithi196.stdout:233: expect (ObjNum 96 snap 11 seq_num 96) 2024-02-05T04:53:55.020 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-02-05T04:53:55.020 INFO:teuthology.orchestra.run.smithi062.stdout: "ops": [], 2024-02-05T04:53:55.020 INFO:teuthology.orchestra.run.smithi062.stdout: "num_ops": 0 2024-02-05T04:53:55.020 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-02-05T04:53:55.029 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-02-05T04:53:55.073 INFO:tasks.ceph.osd.12.smithi062.stderr:2024-02-05T04:53:55.072+0000 7f5e5847e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 34695) UID: 0 2024-02-05T04:53:55.086 INFO:tasks.rados.rados.0.smithi196.stdout:234: finishing rollback tid 0 to smithi19639362-30 2024-02-05T04:53:55.086 INFO:tasks.rados.rados.0.smithi196.stdout:234: finishing rollback tid 1 to smithi19639362-30 2024-02-05T04:53:55.086 INFO:tasks.rados.rados.0.smithi196.stdout:234: finishing rollback tid 2 to smithi19639362-30 2024-02-05T04:53:55.086 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 30 v 27 (ObjNum 60 snap 2 seq_num 60) dirty exists 2024-02-05T04:53:55.086 INFO:tasks.rados.rados.0.smithi196.stdout:230: finishing write tid 1 to smithi19639362-20 2024-02-05T04:53:55.086 INFO:tasks.rados.rados.0.smithi196.stdout:230: finishing write tid 2 to smithi19639362-20 2024-02-05T04:53:55.086 INFO:tasks.rados.rados.0.smithi196.stdout:230: finishing write tid 3 to smithi19639362-20 2024-02-05T04:53:55.086 INFO:tasks.rados.rados.0.smithi196.stdout:230: finishing write tid 4 to smithi19639362-20 2024-02-05T04:53:55.086 INFO:tasks.rados.rados.0.smithi196.stdout:235: finishing write tid 1 to smithi19639362-21 2024-02-05T04:53:55.086 INFO:tasks.rados.rados.0.smithi196.stdout:232: done (6 left) 2024-02-05T04:53:55.086 INFO:tasks.rados.rados.0.smithi196.stdout:233: done (5 left) 2024-02-05T04:53:55.087 INFO:tasks.rados.rados.0.smithi196.stdout:234: done (4 left) 2024-02-05T04:53:55.087 INFO:tasks.rados.rados.0.smithi196.stdout:236: done (3 left) 2024-02-05T04:53:55.087 INFO:tasks.rados.rados.0.smithi196.stdout:237: read oid 34 snap 16 2024-02-05T04:53:55.087 INFO:tasks.rados.rados.0.smithi196.stdout:237: expect (ObjNum -1103623168 snap 22054 seq_num 3191333056) 2024-02-05T04:53:55.087 INFO:tasks.rados.rados.0.smithi196.stdout:238: read oid 47 snap -1 2024-02-05T04:53:55.087 INFO:tasks.rados.rados.0.smithi196.stdout:238: expect (ObjNum 46 snap 0 seq_num 46) 2024-02-05T04:53:55.087 INFO:tasks.rados.rados.0.smithi196.stdout:239: snap_create 2024-02-05T04:53:55.087 INFO:tasks.rados.rados.0.smithi196.stdout:235: finishing write tid 2 to smithi19639362-21 2024-02-05T04:53:55.087 INFO:tasks.rados.rados.0.smithi196.stdout:235: finishing write tid 3 to smithi19639362-21 2024-02-05T04:53:55.087 INFO:tasks.rados.rados.0.smithi196.stdout:235: finishing write tid 4 to smithi19639362-21 2024-02-05T04:53:55.087 INFO:tasks.rados.rados.0.smithi196.stdout:231: finishing copy_from racing read to smithi19639362-24 2024-02-05T04:53:55.087 INFO:tasks.rados.rados.0.smithi196.stdout:230: finishing write tid 5 to smithi19639362-20 2024-02-05T04:53:55.087 INFO:tasks.rados.rados.0.smithi196.stdout:230: oid 20 updating version 0 to 8 2024-02-05T04:53:55.087 INFO:tasks.rados.rados.0.smithi196.stdout:230: oid 20 version 8 is already newer than 7 2024-02-05T04:53:55.087 INFO:tasks.rados.rados.0.smithi196.stdout:230: oid 20 updating version 8 to 9 2024-02-05T04:53:55.088 INFO:tasks.rados.rados.0.smithi196.stdout:230: oid 20 version 9 is already newer than 6 2024-02-05T04:53:55.088 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 20 v 9 (ObjNum 112 snap 18 seq_num 112) dirty exists 2024-02-05T04:53:55.088 INFO:tasks.rados.rados.0.smithi196.stdout:230: left oid 20 (ObjNum 112 snap 18 seq_num 112) 2024-02-05T04:53:55.088 INFO:tasks.rados.rados.0.smithi196.stdout:235: finishing write tid 5 to smithi19639362-21 2024-02-05T04:53:55.088 INFO:tasks.rados.rados.0.smithi196.stdout:235: oid 21 updating version 0 to 53 2024-02-05T04:53:55.088 INFO:tasks.rados.rados.0.smithi196.stdout:235: oid 21 updating version 53 to 54 2024-02-05T04:53:55.088 INFO:tasks.rados.rados.0.smithi196.stdout:235: oid 21 updating version 54 to 55 2024-02-05T04:53:55.088 INFO:tasks.rados.rados.0.smithi196.stdout:235: oid 21 updating version 55 to 56 2024-02-05T04:53:55.088 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 21 v 56 (ObjNum 114 snap 18 seq_num 114) dirty exists 2024-02-05T04:53:55.088 INFO:tasks.rados.rados.0.smithi196.stdout:235: left oid 21 (ObjNum 114 snap 18 seq_num 114) 2024-02-05T04:53:55.088 INFO:tasks.rados.rados.0.smithi196.stdout:231: finishing copy_from to smithi19639362-24 2024-02-05T04:53:55.088 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 24 v 20 (ObjNum 60 snap 2 seq_num 60) dirty exists 2024-02-05T04:53:55.088 INFO:tasks.rados.rados.0.smithi196.stdout:237: expect (ObjNum -1103623168 snap 22054 seq_num 3191333056) 2024-02-05T04:53:55.090 INFO:tasks.rados.rados.0.smithi196.stdout:238: expect (ObjNum 46 snap 0 seq_num 46) 2024-02-05T04:53:55.137 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-02-05T04:53:55.137 INFO:teuthology.orchestra.run.smithi062.stdout: "ops": [], 2024-02-05T04:53:55.137 INFO:teuthology.orchestra.run.smithi062.stdout: "complaint_time": 30, 2024-02-05T04:53:55.137 INFO:teuthology.orchestra.run.smithi062.stdout: "num_blocked_ops": 0 2024-02-05T04:53:55.137 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-02-05T04:53:55.145 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-02-05T04:53:55.153 INFO:tasks.ceph.osd.11.smithi196.stderr:2024-02-05T04:53:55.152+0000 7f85a8d86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 34609) UID: 0 2024-02-05T04:53:55.254 INFO:tasks.ceph.osd.15.smithi196.stderr:2024-02-05T04:53:55.253+0000 7f1facc45640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 34611) UID: 0 2024-02-05T04:53:55.294 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-02-05T04:53:55.294 INFO:teuthology.orchestra.run.smithi062.stdout: "size": 20, 2024-02-05T04:53:55.294 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 600, 2024-02-05T04:53:55.294 INFO:teuthology.orchestra.run.smithi062.stdout: "ops": [ 2024-02-05T04:53:55.295 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.295 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4209.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-02-05T04:53:55.295 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:08.919510+0000", 2024-02-05T04:53:55.295 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 46.372765117, 2024-02-05T04:53:55.295 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.70688627900000001, 2024-02-05T04:53:55.295 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:55.295 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:55.295 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:53:55.295 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4209", 2024-02-05T04:53:55.295 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3672061555", 2024-02-05T04:53:55.295 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 1 2024-02-05T04:53:55.295 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.295 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:55.295 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.295 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:55.296 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:08.919510+0000", 2024-02-05T04:53:55.296 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:55.296 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.296 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.296 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:55.296 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:08.919510+0000", 2024-02-05T04:53:55.296 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:55.296 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.296 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.296 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:55.296 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:08.919512+0000", 2024-02-05T04:53:55.296 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1620000000000001e-06 2024-02-05T04:53:55.296 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.296 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.296 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:55.297 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:08.919525+0000", 2024-02-05T04:53:55.297 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.3369e-05 2024-02-05T04:53:55.297 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.297 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.297 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:55.297 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:08.919536+0000", 2024-02-05T04:53:55.297 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0621e-05 2024-02-05T04:53:55.297 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.297 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.297 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:55.297 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:08.919550+0000", 2024-02-05T04:53:55.297 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.403e-05 2024-02-05T04:53:55.297 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.297 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.297 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:55.299 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:08.919625+0000", 2024-02-05T04:53:55.299 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.5328999999999997e-05 2024-02-05T04:53:55.299 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.299 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.300 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for peered", 2024-02-05T04:53:55.300 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:08.919634+0000", 2024-02-05T04:53:55.300 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.4879999999999996e-06 2024-02-05T04:53:55.300 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.300 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.300 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:55.300 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.624946+0000", 2024-02-05T04:53:55.300 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.70531134600000001 2024-02-05T04:53:55.300 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.300 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.300 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:53:55.300 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.625129+0000", 2024-02-05T04:53:55.300 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00018288199999999999 2024-02-05T04:53:55.300 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.300 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.301 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for subops from 2", 2024-02-05T04:53:55.301 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.625339+0000", 2024-02-05T04:53:55.301 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00021016900000000001 2024-02-05T04:53:55.301 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.301 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.301 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "op_commit", 2024-02-05T04:53:55.301 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.625989+0000", 2024-02-05T04:53:55.301 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00065006700000000005 2024-02-05T04:53:55.301 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.301 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.301 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_commit_rec", 2024-02-05T04:53:55.301 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.626301+0000", 2024-02-05T04:53:55.301 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00031220800000000002 2024-02-05T04:53:55.301 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.301 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.301 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:53:55.302 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.626352+0000", 2024-02-05T04:53:55.302 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.0973000000000001e-05 2024-02-05T04:53:55.302 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.302 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.302 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:55.302 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.626397+0000", 2024-02-05T04:53:55.302 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.4635000000000002e-05 2024-02-05T04:53:55.302 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.302 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:55.302 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.302 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.302 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.302 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4209.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-02-05T04:53:55.302 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:09.696463+0000", 2024-02-05T04:53:55.302 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 45.595812885999997, 2024-02-05T04:53:55.303 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0095939449999999996, 2024-02-05T04:53:55.303 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:55.303 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:55.303 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:53:55.303 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4209", 2024-02-05T04:53:55.303 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3672061555", 2024-02-05T04:53:55.303 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 85 2024-02-05T04:53:55.303 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.303 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:55.303 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.303 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:55.303 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.696463+0000", 2024-02-05T04:53:55.303 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:55.303 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.303 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.303 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:55.304 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.696463+0000", 2024-02-05T04:53:55.304 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:55.304 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.304 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.304 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:55.304 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.696463+0000", 2024-02-05T04:53:55.304 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.9400000000000001e-07 2024-02-05T04:53:55.304 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.304 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.304 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:55.304 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.696465+0000", 2024-02-05T04:53:55.304 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.1019999999999999e-06 2024-02-05T04:53:55.304 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.304 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.304 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:55.305 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.696470+0000", 2024-02-05T04:53:55.305 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.6099999999999999e-06 2024-02-05T04:53:55.305 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.305 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.305 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:55.305 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.696473+0000", 2024-02-05T04:53:55.305 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.8949999999999998e-06 2024-02-05T04:53:55.305 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.305 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.305 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:55.305 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.697422+0000", 2024-02-05T04:53:55.305 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00094939000000000004 2024-02-05T04:53:55.305 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.305 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.305 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:53:55.305 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.697489+0000", 2024-02-05T04:53:55.306 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.7293999999999996e-05 2024-02-05T04:53:55.306 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.306 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.306 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for subops from 2", 2024-02-05T04:53:55.306 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.697604+0000", 2024-02-05T04:53:55.306 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000114288 2024-02-05T04:53:55.306 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.306 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.306 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "op_commit", 2024-02-05T04:53:55.306 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.702424+0000", 2024-02-05T04:53:55.306 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0048202649999999998 2024-02-05T04:53:55.306 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.306 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.306 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_commit_rec", 2024-02-05T04:53:55.306 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.706007+0000", 2024-02-05T04:53:55.306 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.003583059 2024-02-05T04:53:55.307 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.307 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.307 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:53:55.307 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.706037+0000", 2024-02-05T04:53:55.307 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.9944999999999999e-05 2024-02-05T04:53:55.307 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.307 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.307 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:55.307 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.706057+0000", 2024-02-05T04:53:55.307 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.9803e-05 2024-02-05T04:53:55.307 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.307 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:55.307 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.307 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.307 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.307 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4209.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-02-05T04:53:55.308 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:09.696477+0000", 2024-02-05T04:53:55.308 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 45.595798532000003, 2024-02-05T04:53:55.308 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0096605600000000003, 2024-02-05T04:53:55.308 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:55.308 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:55.308 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:53:55.308 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4209", 2024-02-05T04:53:55.308 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3672061555", 2024-02-05T04:53:55.308 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 86 2024-02-05T04:53:55.308 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.308 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:55.308 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.308 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:55.308 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.696477+0000", 2024-02-05T04:53:55.308 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:55.309 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.309 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.309 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:55.309 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.696477+0000", 2024-02-05T04:53:55.309 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:55.309 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.309 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.309 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:55.309 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.696477+0000", 2024-02-05T04:53:55.309 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.3200000000000001e-07 2024-02-05T04:53:55.309 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.309 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.309 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:55.309 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.696480+0000", 2024-02-05T04:53:55.309 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.3779999999999999e-06 2024-02-05T04:53:55.310 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.310 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.310 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:55.310 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.696483+0000", 2024-02-05T04:53:55.310 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.23e-06 2024-02-05T04:53:55.310 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.310 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.310 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:55.310 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.696485+0000", 2024-02-05T04:53:55.310 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.4200000000000001e-06 2024-02-05T04:53:55.310 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.310 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.310 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:55.310 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.697977+0000", 2024-02-05T04:53:55.310 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0014921299999999999 2024-02-05T04:53:55.311 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.311 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.311 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:53:55.311 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.698018+0000", 2024-02-05T04:53:55.311 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.0676999999999997e-05 2024-02-05T04:53:55.311 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.311 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.311 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for subops from 2", 2024-02-05T04:53:55.311 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.698082+0000", 2024-02-05T04:53:55.311 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.3466000000000005e-05 2024-02-05T04:53:55.311 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.311 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.311 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "op_commit", 2024-02-05T04:53:55.311 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.702654+0000", 2024-02-05T04:53:55.311 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0045719710000000002 2024-02-05T04:53:55.312 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.312 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.312 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_commit_rec", 2024-02-05T04:53:55.312 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.706093+0000", 2024-02-05T04:53:55.312 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0034398749999999998 2024-02-05T04:53:55.312 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.312 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.312 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:53:55.312 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.706120+0000", 2024-02-05T04:53:55.312 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.6801999999999999e-05 2024-02-05T04:53:55.312 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.312 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.312 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:55.312 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.706138+0000", 2024-02-05T04:53:55.312 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.7379000000000001e-05 2024-02-05T04:53:55.313 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.313 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:55.313 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.313 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.313 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.313 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4209.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-02-05T04:53:55.313 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:09.696526+0000", 2024-02-05T04:53:55.313 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 45.595749822999998, 2024-02-05T04:53:55.313 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0096876010000000005, 2024-02-05T04:53:55.313 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:55.313 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:55.313 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:53:55.313 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4209", 2024-02-05T04:53:55.313 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3672061555", 2024-02-05T04:53:55.313 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 87 2024-02-05T04:53:55.314 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.314 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:55.314 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.314 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:55.314 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.696526+0000", 2024-02-05T04:53:55.314 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:55.314 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.314 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.314 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:55.314 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.696526+0000", 2024-02-05T04:53:55.314 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:55.314 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.314 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.314 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:55.314 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.696526+0000", 2024-02-05T04:53:55.314 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.2899999999999999e-07 2024-02-05T04:53:55.315 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.315 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.315 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:55.315 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.696602+0000", 2024-02-05T04:53:55.315 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.5636999999999994e-05 2024-02-05T04:53:55.315 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.315 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.315 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:55.315 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.696607+0000", 2024-02-05T04:53:55.315 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.463e-06 2024-02-05T04:53:55.315 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.315 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.315 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:55.315 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.696611+0000", 2024-02-05T04:53:55.315 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.4199999999999999e-06 2024-02-05T04:53:55.315 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.316 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.316 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:55.316 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.698261+0000", 2024-02-05T04:53:55.316 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0016498039999999999 2024-02-05T04:53:55.316 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.316 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.316 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:53:55.316 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.698300+0000", 2024-02-05T04:53:55.316 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.9468000000000002e-05 2024-02-05T04:53:55.316 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.316 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.316 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for subops from 2", 2024-02-05T04:53:55.316 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.698363+0000", 2024-02-05T04:53:55.316 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.3360000000000003e-05 2024-02-05T04:53:55.317 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.317 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.317 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "op_commit", 2024-02-05T04:53:55.317 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.702658+0000", 2024-02-05T04:53:55.317 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0042946929999999996 2024-02-05T04:53:55.317 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.317 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.317 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_commit_rec", 2024-02-05T04:53:55.317 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.706170+0000", 2024-02-05T04:53:55.317 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0035117070000000002 2024-02-05T04:53:55.317 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.317 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.317 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:53:55.317 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.706196+0000", 2024-02-05T04:53:55.317 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.6129e-05 2024-02-05T04:53:55.317 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.318 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.318 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:55.318 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.706213+0000", 2024-02-05T04:53:55.318 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.7490999999999999e-05 2024-02-05T04:53:55.318 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.318 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:55.318 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.318 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.318 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.318 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4209.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-02-05T04:53:55.318 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:09.696618+0000", 2024-02-05T04:53:55.318 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 45.595657527, 2024-02-05T04:53:55.318 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0096608100000000006, 2024-02-05T04:53:55.318 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:55.318 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:55.318 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:53:55.319 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4209", 2024-02-05T04:53:55.319 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3672061555", 2024-02-05T04:53:55.319 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 88 2024-02-05T04:53:55.319 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.319 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:55.319 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.319 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:55.319 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.696618+0000", 2024-02-05T04:53:55.319 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:55.319 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.319 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.319 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:55.319 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.696618+0000", 2024-02-05T04:53:55.319 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:55.319 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.320 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.320 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:55.320 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.696618+0000", 2024-02-05T04:53:55.320 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.7099999999999998e-07 2024-02-05T04:53:55.320 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.320 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.320 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:55.320 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.696621+0000", 2024-02-05T04:53:55.320 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.401e-06 2024-02-05T04:53:55.320 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.320 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.320 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:55.320 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.696624+0000", 2024-02-05T04:53:55.320 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.4680000000000001e-06 2024-02-05T04:53:55.320 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.321 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.321 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:55.321 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.696627+0000", 2024-02-05T04:53:55.321 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.525e-06 2024-02-05T04:53:55.321 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.321 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.321 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:55.321 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.698695+0000", 2024-02-05T04:53:55.321 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0020686290000000002 2024-02-05T04:53:55.321 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.321 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.321 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:53:55.321 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.698736+0000", 2024-02-05T04:53:55.321 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.0555e-05 2024-02-05T04:53:55.321 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.322 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.322 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for subops from 2", 2024-02-05T04:53:55.322 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.698800+0000", 2024-02-05T04:53:55.322 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.4437000000000007e-05 2024-02-05T04:53:55.322 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.322 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.322 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "op_commit", 2024-02-05T04:53:55.322 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.702661+0000", 2024-02-05T04:53:55.322 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0038606640000000002 2024-02-05T04:53:55.322 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.322 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.322 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_commit_rec", 2024-02-05T04:53:55.322 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.706238+0000", 2024-02-05T04:53:55.322 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0035770250000000002 2024-02-05T04:53:55.322 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.323 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.323 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:53:55.323 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.706263+0000", 2024-02-05T04:53:55.323 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.4797999999999999e-05 2024-02-05T04:53:55.323 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.323 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.323 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:55.323 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.706279+0000", 2024-02-05T04:53:55.323 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.6036999999999999e-05 2024-02-05T04:53:55.323 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.323 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:55.323 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.323 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.323 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.323 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4209.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-02-05T04:53:55.324 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:09.850779+0000", 2024-02-05T04:53:55.324 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 45.441496610999998, 2024-02-05T04:53:55.324 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0095974279999999999, 2024-02-05T04:53:55.324 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:55.324 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:55.324 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:53:55.324 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4209", 2024-02-05T04:53:55.324 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3672061555", 2024-02-05T04:53:55.324 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 110 2024-02-05T04:53:55.324 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.324 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:55.324 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.324 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:55.324 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.850779+0000", 2024-02-05T04:53:55.324 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:55.325 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.325 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.325 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:55.325 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.850779+0000", 2024-02-05T04:53:55.325 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:55.325 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.325 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.325 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:55.325 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.850780+0000", 2024-02-05T04:53:55.325 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.5199999999999997e-07 2024-02-05T04:53:55.325 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.325 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.325 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:55.325 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.850784+0000", 2024-02-05T04:53:55.325 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.0099999999999997e-06 2024-02-05T04:53:55.326 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.326 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.326 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:55.326 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.850791+0000", 2024-02-05T04:53:55.326 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.2259999999999999e-06 2024-02-05T04:53:55.326 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.326 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.326 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:55.326 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.850796+0000", 2024-02-05T04:53:55.326 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.2089999999999996e-06 2024-02-05T04:53:55.326 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.326 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.326 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:55.326 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.853636+0000", 2024-02-05T04:53:55.326 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0028403199999999999 2024-02-05T04:53:55.326 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.327 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.327 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:53:55.327 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.853688+0000", 2024-02-05T04:53:55.327 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.1391000000000003e-05 2024-02-05T04:53:55.327 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.327 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.327 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for subops from 2", 2024-02-05T04:53:55.327 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.853766+0000", 2024-02-05T04:53:55.327 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.7875999999999998e-05 2024-02-05T04:53:55.327 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.327 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.327 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "op_commit", 2024-02-05T04:53:55.327 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.855430+0000", 2024-02-05T04:53:55.328 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.001664603 2024-02-05T04:53:55.328 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.328 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.328 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_commit_rec", 2024-02-05T04:53:55.328 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.860285+0000", 2024-02-05T04:53:55.328 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0048545230000000003 2024-02-05T04:53:55.328 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.328 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.328 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:53:55.328 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.860340+0000", 2024-02-05T04:53:55.328 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.5531000000000001e-05 2024-02-05T04:53:55.328 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.328 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.328 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:55.328 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.860376+0000", 2024-02-05T04:53:55.329 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.6186999999999998e-05 2024-02-05T04:53:55.329 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.329 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:55.329 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.329 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.329 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.329 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4209.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-02-05T04:53:55.329 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:09.850805+0000", 2024-02-05T04:53:55.329 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 45.441470801000001, 2024-02-05T04:53:55.329 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0096654600000000007, 2024-02-05T04:53:55.329 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:55.329 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:55.329 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:53:55.329 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4209", 2024-02-05T04:53:55.329 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3672061555", 2024-02-05T04:53:55.329 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 111 2024-02-05T04:53:55.330 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.330 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:55.330 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.330 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:55.330 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.850805+0000", 2024-02-05T04:53:55.330 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:55.330 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.330 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.330 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:55.330 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.850805+0000", 2024-02-05T04:53:55.330 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:55.330 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.330 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.330 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:55.331 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.850805+0000", 2024-02-05T04:53:55.331 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.3099999999999998e-07 2024-02-05T04:53:55.331 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.331 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.331 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:55.331 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.850809+0000", 2024-02-05T04:53:55.331 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.6370000000000001e-06 2024-02-05T04:53:55.331 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.331 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.331 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:55.331 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.850817+0000", 2024-02-05T04:53:55.331 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.3909999999999998e-06 2024-02-05T04:53:55.331 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.331 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.331 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:55.332 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.850831+0000", 2024-02-05T04:53:55.332 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.3942e-05 2024-02-05T04:53:55.332 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.332 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.332 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:55.332 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.854167+0000", 2024-02-05T04:53:55.332 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0033358530000000002 2024-02-05T04:53:55.332 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.332 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.332 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:53:55.332 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.854212+0000", 2024-02-05T04:53:55.332 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.5393999999999999e-05 2024-02-05T04:53:55.332 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.332 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.332 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for subops from 2", 2024-02-05T04:53:55.333 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.854290+0000", 2024-02-05T04:53:55.333 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.7189999999999998e-05 2024-02-05T04:53:55.333 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.333 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.333 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "op_commit", 2024-02-05T04:53:55.333 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.857996+0000", 2024-02-05T04:53:55.333 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0037059889999999998 2024-02-05T04:53:55.333 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.333 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.333 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_commit_rec", 2024-02-05T04:53:55.334 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.860419+0000", 2024-02-05T04:53:55.334 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0024236309999999999 2024-02-05T04:53:55.334 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.334 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.334 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:53:55.334 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.860451+0000", 2024-02-05T04:53:55.334 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.1433000000000002e-05 2024-02-05T04:53:55.334 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.334 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.334 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:55.334 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.860470+0000", 2024-02-05T04:53:55.335 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.9568999999999998e-05 2024-02-05T04:53:55.335 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.335 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:55.335 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.335 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.335 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.335 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4209.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-02-05T04:53:55.335 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:09.850841+0000", 2024-02-05T04:53:55.335 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 45.441434684000001, 2024-02-05T04:53:55.335 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0096977799999999996, 2024-02-05T04:53:55.335 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:55.335 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:55.335 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:53:55.335 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4209", 2024-02-05T04:53:55.335 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3672061555", 2024-02-05T04:53:55.336 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 112 2024-02-05T04:53:55.336 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.336 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:55.336 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.336 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:55.336 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.850841+0000", 2024-02-05T04:53:55.336 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:55.336 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.336 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.336 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:55.336 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.850841+0000", 2024-02-05T04:53:55.336 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:55.336 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.336 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.336 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:55.337 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.850841+0000", 2024-02-05T04:53:55.337 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.3000000000000001e-07 2024-02-05T04:53:55.337 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.337 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.337 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:55.337 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.850961+0000", 2024-02-05T04:53:55.337 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00011995 2024-02-05T04:53:55.337 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.337 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.337 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:55.337 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.850981+0000", 2024-02-05T04:53:55.337 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.9323000000000001e-05 2024-02-05T04:53:55.337 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.337 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.338 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:55.338 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.850987+0000", 2024-02-05T04:53:55.338 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.5729999999999996e-06 2024-02-05T04:53:55.338 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.338 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.338 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:55.338 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.854544+0000", 2024-02-05T04:53:55.338 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0035569809999999999 2024-02-05T04:53:55.338 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.338 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.338 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:53:55.338 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.854626+0000", 2024-02-05T04:53:55.338 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.1725999999999996e-05 2024-02-05T04:53:55.338 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.338 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.339 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for subops from 2", 2024-02-05T04:53:55.339 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.854754+0000", 2024-02-05T04:53:55.339 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00012792899999999999 2024-02-05T04:53:55.339 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.339 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.339 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "op_commit", 2024-02-05T04:53:55.339 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.857998+0000", 2024-02-05T04:53:55.339 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0032445070000000002 2024-02-05T04:53:55.339 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.339 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.339 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_commit_rec", 2024-02-05T04:53:55.339 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.860493+0000", 2024-02-05T04:53:55.339 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0024945700000000002 2024-02-05T04:53:55.339 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.339 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.339 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:53:55.340 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.860520+0000", 2024-02-05T04:53:55.340 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.7141999999999999e-05 2024-02-05T04:53:55.340 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.340 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.340 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:55.340 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.860539+0000", 2024-02-05T04:53:55.340 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.8649000000000001e-05 2024-02-05T04:53:55.340 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.340 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:55.340 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.340 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.340 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.340 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4209.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-02-05T04:53:55.340 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:09.850999+0000", 2024-02-05T04:53:55.340 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 45.441276436999999, 2024-02-05T04:53:55.341 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0095989739999999997, 2024-02-05T04:53:55.341 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:55.341 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:55.341 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:53:55.341 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4209", 2024-02-05T04:53:55.341 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3672061555", 2024-02-05T04:53:55.341 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 113 2024-02-05T04:53:55.341 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.341 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:55.341 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.341 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:55.341 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.850999+0000", 2024-02-05T04:53:55.341 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:55.341 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.341 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.342 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:55.342 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.850999+0000", 2024-02-05T04:53:55.342 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:55.342 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.342 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.342 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:55.342 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851000+0000", 2024-02-05T04:53:55.342 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.0900000000000002e-07 2024-02-05T04:53:55.342 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.342 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.342 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:55.342 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851004+0000", 2024-02-05T04:53:55.342 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.4669999999999998e-06 2024-02-05T04:53:55.342 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.342 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.342 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:55.342 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851014+0000", 2024-02-05T04:53:55.343 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.7540000000000001e-06 2024-02-05T04:53:55.343 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.343 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.343 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:55.343 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851019+0000", 2024-02-05T04:53:55.343 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.2610000000000001e-06 2024-02-05T04:53:55.343 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.343 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.343 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:55.343 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.855104+0000", 2024-02-05T04:53:55.343 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0040849609999999998 2024-02-05T04:53:55.343 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.343 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.343 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:53:55.343 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.855150+0000", 2024-02-05T04:53:55.344 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.6143999999999997e-05 2024-02-05T04:53:55.344 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.344 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.344 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for subops from 2", 2024-02-05T04:53:55.344 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.855224+0000", 2024-02-05T04:53:55.344 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.4024999999999997e-05 2024-02-05T04:53:55.344 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.344 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.344 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "op_commit", 2024-02-05T04:53:55.344 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.858002+0000", 2024-02-05T04:53:55.344 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.002777336 2024-02-05T04:53:55.344 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.344 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.344 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_commit_rec", 2024-02-05T04:53:55.344 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.860559+0000", 2024-02-05T04:53:55.345 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0025570910000000001 2024-02-05T04:53:55.345 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.345 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.345 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:53:55.345 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.860583+0000", 2024-02-05T04:53:55.345 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.4562999999999998e-05 2024-02-05T04:53:55.345 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.345 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.345 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:55.345 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.860598+0000", 2024-02-05T04:53:55.345 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.4863e-05 2024-02-05T04:53:55.345 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.345 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:55.345 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.345 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.346 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.346 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4209.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-02-05T04:53:55.346 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:09.851028+0000", 2024-02-05T04:53:55.346 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 45.441247253999997, 2024-02-05T04:53:55.346 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0097085690000000002, 2024-02-05T04:53:55.346 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:55.346 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:55.346 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:53:55.346 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4209", 2024-02-05T04:53:55.346 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3672061555", 2024-02-05T04:53:55.346 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 114 2024-02-05T04:53:55.346 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.346 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:55.346 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.346 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:55.347 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851028+0000", 2024-02-05T04:53:55.347 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:55.347 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.347 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.347 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:55.347 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851028+0000", 2024-02-05T04:53:55.347 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:55.347 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.347 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.347 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:55.347 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851029+0000", 2024-02-05T04:53:55.347 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.9599999999999999e-07 2024-02-05T04:53:55.347 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.347 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.347 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:55.348 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851033+0000", 2024-02-05T04:53:55.348 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.0169999999999998e-06 2024-02-05T04:53:55.348 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.348 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.348 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:55.348 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851044+0000", 2024-02-05T04:53:55.348 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1388e-05 2024-02-05T04:53:55.348 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.348 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.348 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:55.348 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851050+0000", 2024-02-05T04:53:55.348 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.7250000000000002e-06 2024-02-05T04:53:55.348 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.348 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.348 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:55.348 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.855468+0000", 2024-02-05T04:53:55.349 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0044182170000000003 2024-02-05T04:53:55.349 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.349 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.349 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:53:55.349 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.855549+0000", 2024-02-05T04:53:55.349 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.1248999999999994e-05 2024-02-05T04:53:55.349 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.349 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.349 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for subops from 2", 2024-02-05T04:53:55.349 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.855675+0000", 2024-02-05T04:53:55.349 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000126029 2024-02-05T04:53:55.349 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.349 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.349 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "op_commit", 2024-02-05T04:53:55.349 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.858005+0000", 2024-02-05T04:53:55.350 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0023293329999999998 2024-02-05T04:53:55.350 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.350 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.350 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_commit_rec", 2024-02-05T04:53:55.350 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.860645+0000", 2024-02-05T04:53:55.350 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0026406960000000001 2024-02-05T04:53:55.350 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.350 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.350 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:53:55.350 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.860701+0000", 2024-02-05T04:53:55.350 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.5492999999999999e-05 2024-02-05T04:53:55.350 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.350 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.350 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:55.350 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.860737+0000", 2024-02-05T04:53:55.350 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.5926000000000001e-05 2024-02-05T04:53:55.351 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.351 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:55.351 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.351 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.351 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.351 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4209.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-02-05T04:53:55.351 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:09.851060+0000", 2024-02-05T04:53:55.351 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 45.441215454000002, 2024-02-05T04:53:55.351 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.011939682, 2024-02-05T04:53:55.351 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:55.351 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:55.351 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:53:55.351 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4209", 2024-02-05T04:53:55.351 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3672061555", 2024-02-05T04:53:55.351 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 115 2024-02-05T04:53:55.352 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.352 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:55.352 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.352 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:55.352 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851060+0000", 2024-02-05T04:53:55.352 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:55.352 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.352 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.352 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:55.352 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851060+0000", 2024-02-05T04:53:55.353 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:55.353 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.353 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.353 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:55.353 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851061+0000", 2024-02-05T04:53:55.353 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.5900000000000002e-07 2024-02-05T04:53:55.353 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.353 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.353 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:55.353 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851142+0000", 2024-02-05T04:53:55.353 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.1631000000000005e-05 2024-02-05T04:53:55.353 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.353 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.353 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:55.353 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851150+0000", 2024-02-05T04:53:55.353 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.2679999999999994e-06 2024-02-05T04:53:55.354 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.354 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.354 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:55.354 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851156+0000", 2024-02-05T04:53:55.354 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.7200000000000003e-06 2024-02-05T04:53:55.354 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.354 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.354 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:55.354 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.856546+0000", 2024-02-05T04:53:55.354 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0053899839999999996 2024-02-05T04:53:55.354 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.354 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.354 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:53:55.354 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.856623+0000", 2024-02-05T04:53:55.354 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.6921000000000004e-05 2024-02-05T04:53:55.355 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.355 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.355 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for subops from 2", 2024-02-05T04:53:55.355 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.856748+0000", 2024-02-05T04:53:55.355 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00012522999999999999 2024-02-05T04:53:55.355 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.355 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.355 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "op_commit", 2024-02-05T04:53:55.355 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.858007+0000", 2024-02-05T04:53:55.355 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0012587779999999999 2024-02-05T04:53:55.355 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.355 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.355 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_commit_rec", 2024-02-05T04:53:55.355 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.862907+0000", 2024-02-05T04:53:55.355 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0048998729999999999 2024-02-05T04:53:55.355 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.356 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.356 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:53:55.356 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.862964+0000", 2024-02-05T04:53:55.356 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.7391e-05 2024-02-05T04:53:55.356 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.356 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.356 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:55.356 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.863000+0000", 2024-02-05T04:53:55.356 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.5426999999999998e-05 2024-02-05T04:53:55.356 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.356 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:55.356 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.356 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.356 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.356 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4209.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-02-05T04:53:55.357 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:09.851167+0000", 2024-02-05T04:53:55.357 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 45.441108610000001, 2024-02-05T04:53:55.357 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.011918911000000001, 2024-02-05T04:53:55.357 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:55.357 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:55.357 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:53:55.357 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4209", 2024-02-05T04:53:55.357 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3672061555", 2024-02-05T04:53:55.357 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 116 2024-02-05T04:53:55.357 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.357 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:55.357 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.357 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:55.357 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851167+0000", 2024-02-05T04:53:55.357 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:55.357 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.358 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.358 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:55.358 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851167+0000", 2024-02-05T04:53:55.358 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:55.358 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.358 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.358 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:55.358 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851167+0000", 2024-02-05T04:53:55.358 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.9999999999999998e-07 2024-02-05T04:53:55.358 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.358 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.358 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:55.358 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851172+0000", 2024-02-05T04:53:55.358 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.0400000000000003e-06 2024-02-05T04:53:55.358 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.359 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.359 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:55.359 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851179+0000", 2024-02-05T04:53:55.359 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.063e-06 2024-02-05T04:53:55.359 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.359 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.359 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:55.359 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851184+0000", 2024-02-05T04:53:55.359 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.1220000000000003e-06 2024-02-05T04:53:55.359 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.359 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.359 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:55.359 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.857205+0000", 2024-02-05T04:53:55.359 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0060208630000000004 2024-02-05T04:53:55.359 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.360 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.360 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:53:55.360 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.857278+0000", 2024-02-05T04:53:55.360 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.3089000000000004e-05 2024-02-05T04:53:55.360 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.360 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.360 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for subops from 2", 2024-02-05T04:53:55.360 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.857412+0000", 2024-02-05T04:53:55.360 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00013471800000000001 2024-02-05T04:53:55.360 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.360 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.360 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "op_commit", 2024-02-05T04:53:55.360 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.858009+0000", 2024-02-05T04:53:55.360 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00059697400000000001 2024-02-05T04:53:55.360 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.361 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.361 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_commit_rec", 2024-02-05T04:53:55.361 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.863035+0000", 2024-02-05T04:53:55.361 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0050251489999999996 2024-02-05T04:53:55.361 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.361 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.361 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:53:55.361 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.863066+0000", 2024-02-05T04:53:55.361 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.1680999999999997e-05 2024-02-05T04:53:55.361 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.361 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.361 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:55.361 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.863086+0000", 2024-02-05T04:53:55.361 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.9712e-05 2024-02-05T04:53:55.361 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.361 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:55.362 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.362 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.362 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.362 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4209.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-02-05T04:53:55.362 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:09.851192+0000", 2024-02-05T04:53:55.362 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 45.441083608, 2024-02-05T04:53:55.362 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.014624346999999999, 2024-02-05T04:53:55.362 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:55.362 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:55.362 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:53:55.362 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4209", 2024-02-05T04:53:55.362 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3672061555", 2024-02-05T04:53:55.362 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 117 2024-02-05T04:53:55.362 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.362 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:55.363 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.363 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:55.363 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851192+0000", 2024-02-05T04:53:55.363 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:55.363 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.363 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.363 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:55.363 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851192+0000", 2024-02-05T04:53:55.363 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:55.363 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.363 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.363 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:55.363 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851192+0000", 2024-02-05T04:53:55.363 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.5900000000000003e-07 2024-02-05T04:53:55.363 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.364 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.364 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:55.364 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851200+0000", 2024-02-05T04:53:55.364 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.2420000000000003e-06 2024-02-05T04:53:55.364 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.364 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.364 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:55.364 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851206+0000", 2024-02-05T04:53:55.364 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.2419999999999997e-06 2024-02-05T04:53:55.364 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.364 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.364 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:55.364 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851211+0000", 2024-02-05T04:53:55.364 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.7339999999999999e-06 2024-02-05T04:53:55.365 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.365 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.365 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:55.365 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.857705+0000", 2024-02-05T04:53:55.365 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0064942890000000003 2024-02-05T04:53:55.365 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.365 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.365 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:53:55.365 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.857756+0000", 2024-02-05T04:53:55.365 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.1360999999999998e-05 2024-02-05T04:53:55.365 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.365 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.365 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for subops from 2", 2024-02-05T04:53:55.365 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.857841+0000", 2024-02-05T04:53:55.365 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.4610000000000005e-05 2024-02-05T04:53:55.366 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.366 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.366 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "op_commit", 2024-02-05T04:53:55.366 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.858257+0000", 2024-02-05T04:53:55.366 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00041574599999999999 2024-02-05T04:53:55.366 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.366 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.366 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_commit_rec", 2024-02-05T04:53:55.366 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.865722+0000", 2024-02-05T04:53:55.366 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.007465902 2024-02-05T04:53:55.366 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.366 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.366 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:53:55.366 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.865782+0000", 2024-02-05T04:53:55.366 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.9879e-05 2024-02-05T04:53:55.366 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.367 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.367 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:55.367 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.865816+0000", 2024-02-05T04:53:55.367 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.3983000000000003e-05 2024-02-05T04:53:55.367 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.367 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:55.367 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.367 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.367 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.367 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4209.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-02-05T04:53:55.367 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:09.993364+0000", 2024-02-05T04:53:55.367 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 45.298911506000003, 2024-02-05T04:53:55.367 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0097019250000000001, 2024-02-05T04:53:55.368 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:55.368 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:55.368 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:53:55.368 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4209", 2024-02-05T04:53:55.368 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3672061555", 2024-02-05T04:53:55.368 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 138 2024-02-05T04:53:55.368 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.368 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:55.368 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.368 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:55.368 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.993364+0000", 2024-02-05T04:53:55.368 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:55.368 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.368 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.368 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:55.368 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.993364+0000", 2024-02-05T04:53:55.369 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:55.369 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.369 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.369 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:55.369 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.993364+0000", 2024-02-05T04:53:55.369 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.8700000000000002e-07 2024-02-05T04:53:55.369 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.369 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.369 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:55.369 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.993368+0000", 2024-02-05T04:53:55.369 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.6720000000000002e-06 2024-02-05T04:53:55.369 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.369 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.369 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:55.369 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.993372+0000", 2024-02-05T04:53:55.370 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.625e-06 2024-02-05T04:53:55.370 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.370 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.370 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:55.370 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.993379+0000", 2024-02-05T04:53:55.370 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.1210000000000001e-06 2024-02-05T04:53:55.370 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.370 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.370 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:55.370 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.000600+0000", 2024-02-05T04:53:55.370 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0072211829999999999 2024-02-05T04:53:55.370 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.370 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.370 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:53:55.370 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.000672+0000", 2024-02-05T04:53:55.370 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.1967999999999999e-05 2024-02-05T04:53:55.371 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.371 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.371 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for subops from 2", 2024-02-05T04:53:55.371 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.000765+0000", 2024-02-05T04:53:55.371 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.3139000000000004e-05 2024-02-05T04:53:55.371 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.371 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.371 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "op_commit", 2024-02-05T04:53:55.371 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.002487+0000", 2024-02-05T04:53:55.371 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.001721783 2024-02-05T04:53:55.371 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.371 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.371 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_commit_rec", 2024-02-05T04:53:55.371 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.003017+0000", 2024-02-05T04:53:55.371 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00053045200000000005 2024-02-05T04:53:55.371 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.372 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.372 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:53:55.372 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.003047+0000", 2024-02-05T04:53:55.372 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.9229000000000001e-05 2024-02-05T04:53:55.372 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.372 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.372 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:55.372 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.003066+0000", 2024-02-05T04:53:55.372 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.9466e-05 2024-02-05T04:53:55.372 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.372 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:55.372 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.372 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.372 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.372 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4209.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-02-05T04:53:55.373 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:09.993384+0000", 2024-02-05T04:53:55.373 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 45.298891744999999, 2024-02-05T04:53:55.373 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.013170685999999999, 2024-02-05T04:53:55.373 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:55.373 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:55.373 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:53:55.373 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4209", 2024-02-05T04:53:55.373 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3672061555", 2024-02-05T04:53:55.373 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 139 2024-02-05T04:53:55.373 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.373 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:55.373 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.373 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:55.373 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.993384+0000", 2024-02-05T04:53:55.373 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:55.374 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.374 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.374 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:55.374 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.993384+0000", 2024-02-05T04:53:55.374 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:55.374 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.374 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.374 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:55.374 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.993384+0000", 2024-02-05T04:53:55.374 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.1400000000000001e-07 2024-02-05T04:53:55.374 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.374 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.374 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:55.374 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.993386+0000", 2024-02-05T04:53:55.374 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.9829999999999999e-06 2024-02-05T04:53:55.374 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.375 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.375 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:55.375 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.993390+0000", 2024-02-05T04:53:55.375 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.4209999999999997e-06 2024-02-05T04:53:55.375 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.375 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.375 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:55.375 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.993393+0000", 2024-02-05T04:53:55.375 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.508e-06 2024-02-05T04:53:55.375 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.375 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.375 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:55.375 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.001088+0000", 2024-02-05T04:53:55.375 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0076952230000000002 2024-02-05T04:53:55.376 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.376 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.376 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:53:55.376 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.001134+0000", 2024-02-05T04:53:55.376 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.5467999999999999e-05 2024-02-05T04:53:55.376 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.376 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.376 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for subops from 2", 2024-02-05T04:53:55.376 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.001203+0000", 2024-02-05T04:53:55.376 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.9509000000000001e-05 2024-02-05T04:53:55.376 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.376 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.376 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "op_commit", 2024-02-05T04:53:55.376 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.002497+0000", 2024-02-05T04:53:55.376 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0012934000000000001 2024-02-05T04:53:55.377 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.377 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.377 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_commit_rec", 2024-02-05T04:53:55.377 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.006502+0000", 2024-02-05T04:53:55.377 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0040053980000000003 2024-02-05T04:53:55.377 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.377 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.377 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:53:55.377 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.006534+0000", 2024-02-05T04:53:55.377 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.2351999999999998e-05 2024-02-05T04:53:55.377 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.377 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.377 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:55.377 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.006555+0000", 2024-02-05T04:53:55.377 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.0210000000000001e-05 2024-02-05T04:53:55.378 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.378 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:55.378 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.378 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.378 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.378 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4209.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-02-05T04:53:55.378 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:09.993397+0000", 2024-02-05T04:53:55.378 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 45.298878498000001, 2024-02-05T04:53:55.378 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.013240236000000001, 2024-02-05T04:53:55.378 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:55.378 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:55.378 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:53:55.378 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4209", 2024-02-05T04:53:55.378 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3672061555", 2024-02-05T04:53:55.378 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 140 2024-02-05T04:53:55.378 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.379 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:55.379 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.379 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:55.379 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.993397+0000", 2024-02-05T04:53:55.379 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:55.379 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.379 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.379 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:55.379 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.993397+0000", 2024-02-05T04:53:55.379 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:55.379 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.379 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.379 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:55.379 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.993397+0000", 2024-02-05T04:53:55.379 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.2700000000000001e-07 2024-02-05T04:53:55.379 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.380 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.380 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:55.380 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.993449+0000", 2024-02-05T04:53:55.380 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.1613000000000002e-05 2024-02-05T04:53:55.380 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.380 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.380 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:55.380 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.993453+0000", 2024-02-05T04:53:55.380 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.245e-06 2024-02-05T04:53:55.380 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.380 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.380 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:55.380 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.993456+0000", 2024-02-05T04:53:55.380 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.942e-06 2024-02-05T04:53:55.380 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.381 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.381 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:55.381 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.001438+0000", 2024-02-05T04:53:55.381 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0079815589999999992 2024-02-05T04:53:55.381 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.381 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.381 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:53:55.381 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.001514+0000", 2024-02-05T04:53:55.381 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.6038999999999995e-05 2024-02-05T04:53:55.381 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.381 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.381 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for subops from 2", 2024-02-05T04:53:55.381 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.001642+0000", 2024-02-05T04:53:55.381 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000128713 2024-02-05T04:53:55.381 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.382 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.382 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "op_commit", 2024-02-05T04:53:55.382 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.002506+0000", 2024-02-05T04:53:55.382 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00086372900000000004 2024-02-05T04:53:55.382 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.382 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.382 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_commit_rec", 2024-02-05T04:53:55.382 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.006589+0000", 2024-02-05T04:53:55.382 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0040831690000000002 2024-02-05T04:53:55.382 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.382 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.382 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:53:55.382 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.006620+0000", 2024-02-05T04:53:55.382 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.0817e-05 2024-02-05T04:53:55.382 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.382 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.383 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:55.383 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.006637+0000", 2024-02-05T04:53:55.383 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.7183000000000001e-05 2024-02-05T04:53:55.383 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.383 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:55.383 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.383 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.383 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.383 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4209.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-02-05T04:53:55.383 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:09.993462+0000", 2024-02-05T04:53:55.383 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 45.298813709000001, 2024-02-05T04:53:55.383 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.013244182, 2024-02-05T04:53:55.383 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:55.383 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:55.383 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:53:55.384 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4209", 2024-02-05T04:53:55.384 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3672061555", 2024-02-05T04:53:55.384 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 141 2024-02-05T04:53:55.384 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.384 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:55.384 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.384 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:55.384 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.993462+0000", 2024-02-05T04:53:55.384 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:55.384 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.384 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.384 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:55.384 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.993462+0000", 2024-02-05T04:53:55.384 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:55.385 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.385 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.385 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:55.385 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.993462+0000", 2024-02-05T04:53:55.385 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.6100000000000002e-07 2024-02-05T04:53:55.385 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.385 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.385 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:55.385 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.993464+0000", 2024-02-05T04:53:55.385 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.0480000000000001e-06 2024-02-05T04:53:55.385 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.385 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.385 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:55.385 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.993467+0000", 2024-02-05T04:53:55.385 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.1480000000000002e-06 2024-02-05T04:53:55.386 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.386 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.386 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:55.386 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.993470+0000", 2024-02-05T04:53:55.386 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.5320000000000001e-06 2024-02-05T04:53:55.386 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.386 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.386 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:55.386 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.002184+0000", 2024-02-05T04:53:55.386 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0087141649999999994 2024-02-05T04:53:55.386 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.386 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.386 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:53:55.386 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.002227+0000", 2024-02-05T04:53:55.386 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.3272000000000002e-05 2024-02-05T04:53:55.386 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.387 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.387 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for subops from 2", 2024-02-05T04:53:55.387 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.002296+0000", 2024-02-05T04:53:55.387 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.8268000000000004e-05 2024-02-05T04:53:55.387 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.387 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.387 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "op_commit", 2024-02-05T04:53:55.387 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.002752+0000", 2024-02-05T04:53:55.387 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00045654199999999999 2024-02-05T04:53:55.387 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.387 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.387 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_commit_rec", 2024-02-05T04:53:55.387 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.006663+0000", 2024-02-05T04:53:55.387 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.003911126 2024-02-05T04:53:55.387 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.388 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.388 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:53:55.388 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.006689+0000", 2024-02-05T04:53:55.388 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.6061999999999999e-05 2024-02-05T04:53:55.388 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.388 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.388 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:55.388 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.006706+0000", 2024-02-05T04:53:55.388 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.6758000000000001e-05 2024-02-05T04:53:55.388 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.388 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:55.388 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.388 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.388 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.388 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.es2 38/31 ECSubWrite(tid=25, reqid=osd.2.0:25, at_version=38'20, trim_to=0'0, roll_forward_to=37'16))", 2024-02-05T04:53:55.389 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:30.134965+0000", 2024-02-05T04:53:55.389 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 25.157310236000001, 2024-02-05T04:53:55.389 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.010779733, 2024-02-05T04:53:55.389 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:55.389 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:55.389 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:55.389 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.389 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:55.389 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.134965+0000", 2024-02-05T04:53:55.389 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:55.389 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.389 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.389 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:55.389 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.134965+0000", 2024-02-05T04:53:55.390 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:55.390 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.390 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.390 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:55.390 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.134966+0000", 2024-02-05T04:53:55.390 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.4399999999999999e-07 2024-02-05T04:53:55.390 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.390 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.390 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:55.390 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.134990+0000", 2024-02-05T04:53:55.390 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.3895000000000001e-05 2024-02-05T04:53:55.391 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.391 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.391 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:55.391 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.135002+0000", 2024-02-05T04:53:55.391 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1868000000000001e-05 2024-02-05T04:53:55.391 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.391 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.391 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:55.391 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.135011+0000", 2024-02-05T04:53:55.391 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.8920000000000006e-06 2024-02-05T04:53:55.391 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.391 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.391 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:55.391 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.135354+0000", 2024-02-05T04:53:55.391 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00034297400000000001 2024-02-05T04:53:55.392 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.392 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.392 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:55.392 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.135364+0000", 2024-02-05T04:53:55.392 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0205e-05 2024-02-05T04:53:55.392 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.392 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.392 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:55.392 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.145707+0000", 2024-02-05T04:53:55.392 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.010343250999999999 2024-02-05T04:53:55.392 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.392 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.392 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:55.392 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.145745+0000", 2024-02-05T04:53:55.393 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.7904000000000002e-05 2024-02-05T04:53:55.393 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.393 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:55.393 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.393 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.393 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.393 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.2s5 38/31 ECSubWrite(tid=28, reqid=osd.12.0:28, at_version=38'22, trim_to=0'0, roll_forward_to=37'18))", 2024-02-05T04:53:55.393 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:30.144335+0000", 2024-02-05T04:53:55.393 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 25.147941041999999, 2024-02-05T04:53:55.393 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0094635199999999996, 2024-02-05T04:53:55.393 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:55.393 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:55.393 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:55.394 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.394 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:55.394 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.144335+0000", 2024-02-05T04:53:55.394 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:55.394 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.394 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.394 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:55.394 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.144335+0000", 2024-02-05T04:53:55.394 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:55.394 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.394 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.394 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:55.394 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.144335+0000", 2024-02-05T04:53:55.394 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.5600000000000004e-07 2024-02-05T04:53:55.395 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.395 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.395 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:55.395 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.144373+0000", 2024-02-05T04:53:55.395 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.7731000000000002e-05 2024-02-05T04:53:55.395 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.395 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.395 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:55.395 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.144387+0000", 2024-02-05T04:53:55.395 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.3788000000000001e-05 2024-02-05T04:53:55.395 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.395 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.395 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:55.395 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.144394+0000", 2024-02-05T04:53:55.396 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.2450000000000001e-06 2024-02-05T04:53:55.396 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.396 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.396 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:55.396 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.146480+0000", 2024-02-05T04:53:55.396 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0020859120000000001 2024-02-05T04:53:55.396 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.396 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.396 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:55.396 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.146504+0000", 2024-02-05T04:53:55.396 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.4091999999999999e-05 2024-02-05T04:53:55.396 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.396 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.396 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:55.396 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.153756+0000", 2024-02-05T04:53:55.397 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0072520470000000002 2024-02-05T04:53:55.397 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.397 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.397 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:55.397 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.153798+0000", 2024-02-05T04:53:55.397 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.1848999999999999e-05 2024-02-05T04:53:55.397 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.397 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:55.397 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.397 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.397 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.397 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.4s3 43/31 ECSubWrite(tid=46, reqid=osd.1.0:46, at_version=43'29, trim_to=0'0, roll_forward_to=42'27))", 2024-02-05T04:53:55.397 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:35.182639+0000", 2024-02-05T04:53:55.397 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 20.109636335000001, 2024-02-05T04:53:55.397 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.010458696999999999, 2024-02-05T04:53:55.397 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:55.398 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:55.398 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:55.398 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.398 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:55.398 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.182639+0000", 2024-02-05T04:53:55.398 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:55.398 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.398 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.398 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:55.398 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.182639+0000", 2024-02-05T04:53:55.398 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:55.398 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.398 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.398 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:55.398 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.182639+0000", 2024-02-05T04:53:55.399 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.0599999999999999e-07 2024-02-05T04:53:55.399 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.399 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.399 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:55.399 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.182658+0000", 2024-02-05T04:53:55.399 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.8646000000000001e-05 2024-02-05T04:53:55.399 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.399 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.399 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:55.399 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.182670+0000", 2024-02-05T04:53:55.399 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1629e-05 2024-02-05T04:53:55.399 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.399 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.399 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:55.399 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.182678+0000", 2024-02-05T04:53:55.399 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.7740000000000004e-06 2024-02-05T04:53:55.400 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.400 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.400 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:55.400 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.182738+0000", 2024-02-05T04:53:55.400 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.0123e-05 2024-02-05T04:53:55.400 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.400 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.400 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:55.400 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.182748+0000", 2024-02-05T04:53:55.400 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0078000000000001e-05 2024-02-05T04:53:55.400 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.401 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.401 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:55.401 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.193064+0000", 2024-02-05T04:53:55.401 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.010316561 2024-02-05T04:53:55.401 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.401 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.401 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:55.401 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.193098+0000", 2024-02-05T04:53:55.401 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.3680000000000003e-05 2024-02-05T04:53:55.401 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.401 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:55.401 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.401 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.401 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:55.401 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-02-05T04:53:55.403 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-02-05T04:53:55.404 INFO:tasks.ceph.osd.6.smithi150.stderr:2024-02-05T04:53:55.402+0000 7f6bea9f0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34015) UID: 0 2024-02-05T04:53:55.504 INFO:tasks.ceph.osd.13.smithi091.stderr:2024-02-05T04:53:55.504+0000 7f526fd4d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 34281) UID: 0 2024-02-05T04:53:55.511 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-02-05T04:53:55.511 INFO:teuthology.orchestra.run.smithi062.stdout: "ops": [], 2024-02-05T04:53:55.511 INFO:teuthology.orchestra.run.smithi062.stdout: "num_ops": 0 2024-02-05T04:53:55.511 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-02-05T04:53:55.520 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-02-05T04:53:55.605 INFO:tasks.ceph.osd.10.smithi150.stderr:2024-02-05T04:53:55.604+0000 7f3075a4d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 34018) UID: 0 2024-02-05T04:53:55.627 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-02-05T04:53:55.627 INFO:teuthology.orchestra.run.smithi062.stdout: "ops": [], 2024-02-05T04:53:55.627 INFO:teuthology.orchestra.run.smithi062.stdout: "complaint_time": 30, 2024-02-05T04:53:55.627 INFO:teuthology.orchestra.run.smithi062.stdout: "num_blocked_ops": 0 2024-02-05T04:53:55.627 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-02-05T04:53:55.636 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-02-05T04:53:55.708 INFO:tasks.ceph.osd.13.smithi091.stderr:2024-02-05T04:53:55.707+0000 7f526fd4d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 34281) UID: 0 2024-02-05T04:53:55.744 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-02-05T04:53:55.745 INFO:teuthology.orchestra.run.smithi062.stdout: "size": 20, 2024-02-05T04:53:55.745 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 600, 2024-02-05T04:53:55.745 INFO:teuthology.orchestra.run.smithi062.stdout: "ops": [ 2024-02-05T04:53:55.745 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.745 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:250 3.2s0 3:47e81a58:::smithi19639362-13:head [append 0~0] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:53:55.745 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:29.198960+0000", 2024-02-05T04:53:55.745 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 26.543940573, 2024-02-05T04:53:55.745 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0047770479999999999, 2024-02-05T04:53:55.745 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:55.745 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:55.745 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:53:55.746 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:53:55.746 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:55.746 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 250 2024-02-05T04:53:55.746 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.746 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:55.746 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.746 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:55.746 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.198960+0000", 2024-02-05T04:53:55.746 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:55.746 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.746 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.746 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:55.746 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.198960+0000", 2024-02-05T04:53:55.746 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:55.746 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.746 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.746 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:55.747 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.198962+0000", 2024-02-05T04:53:55.747 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.694e-06 2024-02-05T04:53:55.747 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.747 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.747 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:55.747 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.198969+0000", 2024-02-05T04:53:55.747 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.1749999999999999e-06 2024-02-05T04:53:55.747 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.747 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.747 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:55.747 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.198986+0000", 2024-02-05T04:53:55.747 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.6254999999999999e-05 2024-02-05T04:53:55.747 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.748 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.748 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:55.748 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.198996+0000", 2024-02-05T04:53:55.748 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.006e-05 2024-02-05T04:53:55.748 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.748 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.748 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:55.748 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.199067+0000", 2024-02-05T04:53:55.748 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.1493000000000001e-05 2024-02-05T04:53:55.748 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.748 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.748 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:53:55.749 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.199175+0000", 2024-02-05T04:53:55.749 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00010778299999999999 2024-02-05T04:53:55.749 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.749 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.749 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:55.749 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.200020+0000", 2024-02-05T04:53:55.749 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00084552300000000004 2024-02-05T04:53:55.749 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.749 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.749 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:55.750 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.203411+0000", 2024-02-05T04:53:55.750 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0033904059999999999 2024-02-05T04:53:55.750 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.750 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.750 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:53:55.750 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.203677+0000", 2024-02-05T04:53:55.750 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00026615900000000002 2024-02-05T04:53:55.750 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.750 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.750 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:55.750 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.203738+0000", 2024-02-05T04:53:55.750 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.05e-05 2024-02-05T04:53:55.750 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.751 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:55.751 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.751 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.751 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.751 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:251 3.2s0 3:47e81a58:::smithi19639362-13:head [rollback 3] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:53:55.751 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:29.199025+0000", 2024-02-05T04:53:55.751 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 26.543876431000001, 2024-02-05T04:53:55.751 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.010965871, 2024-02-05T04:53:55.751 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:55.751 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:55.751 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:53:55.751 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:53:55.751 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:55.751 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 251 2024-02-05T04:53:55.751 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.751 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:55.752 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.752 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:55.752 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.199025+0000", 2024-02-05T04:53:55.752 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:55.752 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.752 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.752 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:55.752 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.199025+0000", 2024-02-05T04:53:55.752 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:55.752 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.752 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.752 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:55.752 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.199025+0000", 2024-02-05T04:53:55.752 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.6700000000000003e-07 2024-02-05T04:53:55.753 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.753 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.753 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:55.753 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.199030+0000", 2024-02-05T04:53:55.753 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.9620000000000001e-06 2024-02-05T04:53:55.753 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.753 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.753 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:55.753 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.199039+0000", 2024-02-05T04:53:55.753 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.5660000000000007e-06 2024-02-05T04:53:55.753 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.753 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.753 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:55.753 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.199045+0000", 2024-02-05T04:53:55.753 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.9229999999999999e-06 2024-02-05T04:53:55.754 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.754 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.754 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:55.754 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.200375+0000", 2024-02-05T04:53:55.754 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0013305610000000001 2024-02-05T04:53:55.754 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.754 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.754 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:53:55.754 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.200435+0000", 2024-02-05T04:53:55.754 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.9790999999999997e-05 2024-02-05T04:53:55.754 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.754 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.754 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:55.754 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.200748+0000", 2024-02-05T04:53:55.754 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00031337299999999998 2024-02-05T04:53:55.755 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.755 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.755 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:55.755 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.203751+0000", 2024-02-05T04:53:55.755 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.003003019 2024-02-05T04:53:55.755 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.755 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.755 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:53:55.755 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.209915+0000", 2024-02-05T04:53:55.755 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0061635099999999996 2024-02-05T04:53:55.755 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.755 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.755 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:55.755 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.209990+0000", 2024-02-05T04:53:55.755 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.5599000000000006e-05 2024-02-05T04:53:55.755 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.756 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:55.756 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.756 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.756 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.756 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:252 3.2s0 3:47e81a58:::smithi19639362-13:head [append 0~0] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:53:55.756 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:29.199055+0000", 2024-02-05T04:53:55.756 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 26.54384627, 2024-02-05T04:53:55.756 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.011150785, 2024-02-05T04:53:55.756 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:55.756 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:55.756 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:53:55.756 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:53:55.756 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:55.756 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 252 2024-02-05T04:53:55.756 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.757 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:55.757 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.757 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:55.757 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.199055+0000", 2024-02-05T04:53:55.757 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:55.757 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.757 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.757 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:55.757 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.199055+0000", 2024-02-05T04:53:55.757 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:55.757 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.757 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.757 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:55.757 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.199055+0000", 2024-02-05T04:53:55.757 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.3500000000000002e-07 2024-02-05T04:53:55.758 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.758 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.758 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:55.758 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.199058+0000", 2024-02-05T04:53:55.758 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.779e-06 2024-02-05T04:53:55.758 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.758 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.758 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:55.758 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.199065+0000", 2024-02-05T04:53:55.758 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.5139999999999998e-06 2024-02-05T04:53:55.758 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.758 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.758 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:55.758 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.199069+0000", 2024-02-05T04:53:55.758 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.5820000000000003e-06 2024-02-05T04:53:55.759 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.759 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.759 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:55.759 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.201028+0000", 2024-02-05T04:53:55.759 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.001959199 2024-02-05T04:53:55.759 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.759 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.759 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:53:55.759 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.201092+0000", 2024-02-05T04:53:55.759 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.3375999999999998e-05 2024-02-05T04:53:55.760 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.760 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.760 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:55.760 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.201409+0000", 2024-02-05T04:53:55.760 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00031745599999999998 2024-02-05T04:53:55.760 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.760 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.760 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:55.760 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.203757+0000", 2024-02-05T04:53:55.760 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0023475219999999999 2024-02-05T04:53:55.760 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.760 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.760 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:53:55.760 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.210134+0000", 2024-02-05T04:53:55.761 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0063771130000000002 2024-02-05T04:53:55.761 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.761 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.761 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:55.761 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.210206+0000", 2024-02-05T04:53:55.761 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.1809000000000003e-05 2024-02-05T04:53:55.761 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.761 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:55.761 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.761 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.761 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.761 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:253 3.2s0 3:41121c54:::smithi19639362-23:head [append 0~688128] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:53:55.761 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:29.205018+0000", 2024-02-05T04:53:55.761 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 26.537883386000001, 2024-02-05T04:53:55.761 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0073893079999999998, 2024-02-05T04:53:55.762 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:55.762 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:55.762 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:53:55.762 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:53:55.762 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:55.762 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 253 2024-02-05T04:53:55.762 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.762 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:55.762 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.762 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:55.762 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.205018+0000", 2024-02-05T04:53:55.762 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:55.762 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.762 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.762 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:55.763 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.205018+0000", 2024-02-05T04:53:55.763 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:55.763 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.763 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.763 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:55.763 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.205019+0000", 2024-02-05T04:53:55.763 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.282e-06 2024-02-05T04:53:55.763 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.763 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.763 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:55.763 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.205752+0000", 2024-02-05T04:53:55.763 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00073318000000000005 2024-02-05T04:53:55.763 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.763 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.763 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:55.763 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.205764+0000", 2024-02-05T04:53:55.764 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1936000000000001e-05 2024-02-05T04:53:55.764 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.764 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.764 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:55.764 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.205769+0000", 2024-02-05T04:53:55.764 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.2979999999999999e-06 2024-02-05T04:53:55.764 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.764 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.764 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:55.764 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.205811+0000", 2024-02-05T04:53:55.764 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.1891000000000003e-05 2024-02-05T04:53:55.764 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.764 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.764 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:53:55.764 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.205885+0000", 2024-02-05T04:53:55.765 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.3384000000000001e-05 2024-02-05T04:53:55.765 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.765 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.765 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:55.765 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.206604+0000", 2024-02-05T04:53:55.765 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000719343 2024-02-05T04:53:55.765 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.765 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.765 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:55.765 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.210004+0000", 2024-02-05T04:53:55.765 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0034002799999999999 2024-02-05T04:53:55.765 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.765 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.765 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:53:55.765 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.212332+0000", 2024-02-05T04:53:55.766 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0023273209999999998 2024-02-05T04:53:55.766 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.766 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.766 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:55.766 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.212407+0000", 2024-02-05T04:53:55.766 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.5393000000000001e-05 2024-02-05T04:53:55.766 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.766 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:55.766 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.766 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.766 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.766 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:254 3.2s0 3:41121c54:::smithi19639362-23:head [append 0~376832] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:53:55.766 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:29.207235+0000", 2024-02-05T04:53:55.766 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 26.535666345999999, 2024-02-05T04:53:55.766 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0064199799999999996, 2024-02-05T04:53:55.766 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:55.767 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:55.767 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:53:55.767 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:53:55.767 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:55.767 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 254 2024-02-05T04:53:55.767 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.767 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:55.767 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.767 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:55.767 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.207235+0000", 2024-02-05T04:53:55.767 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:55.767 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.767 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.767 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:55.767 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.207235+0000", 2024-02-05T04:53:55.768 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:55.768 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.768 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.768 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:55.768 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.207236+0000", 2024-02-05T04:53:55.768 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.37e-06 2024-02-05T04:53:55.768 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.768 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.768 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:55.768 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.207708+0000", 2024-02-05T04:53:55.768 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00047221200000000001 2024-02-05T04:53:55.768 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.768 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.768 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:55.768 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.207721+0000", 2024-02-05T04:53:55.769 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.2446000000000001e-05 2024-02-05T04:53:55.769 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.769 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.769 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:55.769 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.207730+0000", 2024-02-05T04:53:55.769 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.4870000000000008e-06 2024-02-05T04:53:55.769 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.769 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.769 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:55.769 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.207796+0000", 2024-02-05T04:53:55.769 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.5852999999999994e-05 2024-02-05T04:53:55.770 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.770 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.770 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:53:55.770 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.207899+0000", 2024-02-05T04:53:55.770 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00010255900000000001 2024-02-05T04:53:55.770 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.770 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.770 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:55.770 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.208541+0000", 2024-02-05T04:53:55.770 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00064210899999999997 2024-02-05T04:53:55.770 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.770 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.770 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:55.770 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.212149+0000", 2024-02-05T04:53:55.770 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0036078170000000001 2024-02-05T04:53:55.771 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.771 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.771 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:53:55.771 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.213599+0000", 2024-02-05T04:53:55.771 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.001450935 2024-02-05T04:53:55.771 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.771 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.771 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:55.771 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.213655+0000", 2024-02-05T04:53:55.771 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.5192000000000003e-05 2024-02-05T04:53:55.771 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.771 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:55.771 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.771 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.771 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.772 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:255 3.2s0 3:41121c54:::smithi19639362-23:head [setxattr _header (56) in=7b] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:53:55.772 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:29.207757+0000", 2024-02-05T04:53:55.772 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 26.535144311, 2024-02-05T04:53:55.772 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0060483919999999997, 2024-02-05T04:53:55.772 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:55.772 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:55.772 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:53:55.772 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:53:55.772 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:55.772 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 255 2024-02-05T04:53:55.772 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.772 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:55.772 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.772 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:55.773 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.207757+0000", 2024-02-05T04:53:55.773 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:55.773 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.773 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.773 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:55.773 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.207757+0000", 2024-02-05T04:53:55.773 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:55.773 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.773 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.773 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:55.773 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.207758+0000", 2024-02-05T04:53:55.773 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.2200000000000003e-07 2024-02-05T04:53:55.773 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.773 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.773 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:55.774 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.207761+0000", 2024-02-05T04:53:55.774 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.422e-06 2024-02-05T04:53:55.774 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.774 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.774 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:55.774 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.207771+0000", 2024-02-05T04:53:55.774 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.6600000000000007e-06 2024-02-05T04:53:55.774 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.774 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.774 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:55.774 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.207777+0000", 2024-02-05T04:53:55.774 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.2870000000000001e-06 2024-02-05T04:53:55.774 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.774 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.774 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:55.774 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.209029+0000", 2024-02-05T04:53:55.775 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.001251699 2024-02-05T04:53:55.775 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.775 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.775 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:53:55.775 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.209123+0000", 2024-02-05T04:53:55.775 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.4444000000000006e-05 2024-02-05T04:53:55.775 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.775 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.775 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:55.775 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.209555+0000", 2024-02-05T04:53:55.775 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00043236300000000001 2024-02-05T04:53:55.775 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.775 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.775 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:55.775 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.212475+0000", 2024-02-05T04:53:55.775 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0029194709999999999 2024-02-05T04:53:55.776 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.776 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.776 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:53:55.776 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.213756+0000", 2024-02-05T04:53:55.776 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.001281194 2024-02-05T04:53:55.776 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.776 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.776 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:55.776 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.213805+0000", 2024-02-05T04:53:55.776 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.9030000000000003e-05 2024-02-05T04:53:55.776 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.776 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:55.776 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.776 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.776 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.777 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:256 3.2s0 3:41121c54:::smithi19639362-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:53:55.777 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:29.207798+0000", 2024-02-05T04:53:55.777 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 26.535102958, 2024-02-05T04:53:55.777 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0068739830000000002, 2024-02-05T04:53:55.777 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:55.777 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "started", 2024-02-05T04:53:55.777 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:53:55.777 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:53:55.777 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:55.777 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 256 2024-02-05T04:53:55.777 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.777 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:55.777 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.777 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:55.777 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.207798+0000", 2024-02-05T04:53:55.777 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:55.778 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.778 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.778 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:55.778 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.207798+0000", 2024-02-05T04:53:55.778 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:55.778 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.778 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.778 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:55.778 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.207799+0000", 2024-02-05T04:53:55.778 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.5899999999999996e-07 2024-02-05T04:53:55.778 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.778 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.778 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:55.778 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.207808+0000", 2024-02-05T04:53:55.778 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.7720000000000006e-06 2024-02-05T04:53:55.778 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.779 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.779 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:55.779 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.207818+0000", 2024-02-05T04:53:55.779 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.1050000000000001e-06 2024-02-05T04:53:55.779 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.779 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.779 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:55.779 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.207823+0000", 2024-02-05T04:53:55.779 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.8409999999999996e-06 2024-02-05T04:53:55.779 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.779 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.779 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:55.779 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.210496+0000", 2024-02-05T04:53:55.779 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0026729879999999998 2024-02-05T04:53:55.779 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.780 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.780 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for rw locks", 2024-02-05T04:53:55.780 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.210593+0000", 2024-02-05T04:53:55.780 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.6873999999999994e-05 2024-02-05T04:53:55.780 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.780 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.780 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:55.780 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.213830+0000", 2024-02-05T04:53:55.780 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0032368560000000002 2024-02-05T04:53:55.780 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.780 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.780 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:53:55.780 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.213901+0000", 2024-02-05T04:53:55.780 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.0813000000000001e-05 2024-02-05T04:53:55.781 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.781 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.781 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:55.781 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.214672+0000", 2024-02-05T04:53:55.781 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00077117500000000003 2024-02-05T04:53:55.781 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.781 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:55.781 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.781 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.781 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.781 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:257 3.2s0 3:4f006f6c:::smithi19639362-3:head [stat,delete] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:53:55.781 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:29.207834+0000", 2024-02-05T04:53:55.781 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 26.535066918999998, 2024-02-05T04:53:55.781 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0085459170000000001, 2024-02-05T04:53:55.781 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:55.781 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:55.782 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:53:55.782 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:53:55.782 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:55.782 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 257 2024-02-05T04:53:55.782 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.782 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:55.782 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.782 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:55.782 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.207834+0000", 2024-02-05T04:53:55.782 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:55.782 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.782 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.782 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:55.782 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.207834+0000", 2024-02-05T04:53:55.782 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:55.782 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.783 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.783 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:55.783 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.207835+0000", 2024-02-05T04:53:55.783 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.13e-07 2024-02-05T04:53:55.783 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.783 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.783 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:55.783 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.207838+0000", 2024-02-05T04:53:55.783 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.96e-06 2024-02-05T04:53:55.783 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.783 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.783 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:55.783 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.207845+0000", 2024-02-05T04:53:55.783 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.6699999999999994e-06 2024-02-05T04:53:55.783 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.783 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.784 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:55.784 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.207851+0000", 2024-02-05T04:53:55.784 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.896e-06 2024-02-05T04:53:55.784 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.784 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.784 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:55.784 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.210632+0000", 2024-02-05T04:53:55.784 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00278087 2024-02-05T04:53:55.784 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.784 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.784 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:53:55.784 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.210723+0000", 2024-02-05T04:53:55.784 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.1265000000000001e-05 2024-02-05T04:53:55.784 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.785 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.785 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:55.785 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.211433+0000", 2024-02-05T04:53:55.785 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00070996899999999996 2024-02-05T04:53:55.785 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.785 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.785 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:55.785 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.214530+0000", 2024-02-05T04:53:55.785 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.003096634 2024-02-05T04:53:55.785 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.785 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.785 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:53:55.785 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.216308+0000", 2024-02-05T04:53:55.785 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00177809 2024-02-05T04:53:55.785 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.786 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.786 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:55.786 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.216380+0000", 2024-02-05T04:53:55.786 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.1950000000000001e-05 2024-02-05T04:53:55.786 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.786 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:55.786 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.786 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.786 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.786 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.6s1 38/31 ECSubWrite(tid=90, reqid=osd.13.0:90, at_version=38'11, trim_to=0'0, roll_forward_to=37'8))", 2024-02-05T04:53:55.786 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:30.141755+0000", 2024-02-05T04:53:55.786 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 25.601145575, 2024-02-05T04:53:55.786 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0044243779999999996, 2024-02-05T04:53:55.786 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:55.786 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:55.787 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:55.787 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.787 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:55.787 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.141755+0000", 2024-02-05T04:53:55.787 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:55.787 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.787 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.787 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:55.787 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.141755+0000", 2024-02-05T04:53:55.787 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:55.787 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.787 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.787 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:55.787 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.141756+0000", 2024-02-05T04:53:55.788 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.2099999999999996e-07 2024-02-05T04:53:55.788 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.788 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.788 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:55.788 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.141777+0000", 2024-02-05T04:53:55.788 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.1135999999999999e-05 2024-02-05T04:53:55.788 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.789 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.789 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:55.789 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.141789+0000", 2024-02-05T04:53:55.789 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1562999999999999e-05 2024-02-05T04:53:55.789 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.789 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.789 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:55.789 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.141797+0000", 2024-02-05T04:53:55.789 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.5920000000000001e-06 2024-02-05T04:53:55.789 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.789 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.789 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:55.789 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.142042+0000", 2024-02-05T04:53:55.789 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00024489899999999997 2024-02-05T04:53:55.790 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.790 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.790 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:55.790 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.142054+0000", 2024-02-05T04:53:55.790 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1482e-05 2024-02-05T04:53:55.790 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.790 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.790 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:55.790 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.146133+0000", 2024-02-05T04:53:55.790 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.004079348 2024-02-05T04:53:55.790 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.790 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.790 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:55.790 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.146180+0000", 2024-02-05T04:53:55.791 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.6637000000000001e-05 2024-02-05T04:53:55.791 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.791 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:55.791 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.791 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.791 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.791 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.7s2 38/31 ECSubWrite(tid=92, reqid=osd.13.0:92, at_version=38'8, trim_to=0'0, roll_forward_to=37'6))", 2024-02-05T04:53:55.791 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:30.149790+0000", 2024-02-05T04:53:55.791 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 25.593110970000001, 2024-02-05T04:53:55.791 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0040142550000000004, 2024-02-05T04:53:55.791 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:55.791 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:55.791 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:55.791 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.791 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:55.791 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.149790+0000", 2024-02-05T04:53:55.792 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:55.792 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.792 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.792 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:55.792 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.149790+0000", 2024-02-05T04:53:55.792 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:55.792 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.792 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.792 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:55.792 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.149791+0000", 2024-02-05T04:53:55.792 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.7000000000000003e-07 2024-02-05T04:53:55.792 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.792 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.792 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:55.792 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.149813+0000", 2024-02-05T04:53:55.793 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.1571000000000001e-05 2024-02-05T04:53:55.793 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.793 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.793 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:55.793 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.149824+0000", 2024-02-05T04:53:55.793 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1051e-05 2024-02-05T04:53:55.793 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.793 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.793 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:55.793 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.149831+0000", 2024-02-05T04:53:55.793 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.3749999999999997e-06 2024-02-05T04:53:55.793 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.793 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.793 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:55.793 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.149897+0000", 2024-02-05T04:53:55.793 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.5648999999999994e-05 2024-02-05T04:53:55.794 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.794 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.794 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:55.794 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.149906+0000", 2024-02-05T04:53:55.794 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.5980000000000003e-06 2024-02-05T04:53:55.794 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.794 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.794 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:55.794 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.153758+0000", 2024-02-05T04:53:55.794 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0038518269999999999 2024-02-05T04:53:55.794 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.794 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.794 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:55.794 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.153804+0000", 2024-02-05T04:53:55.794 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.6313999999999997e-05 2024-02-05T04:53:55.795 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.795 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:55.795 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.795 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.795 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.795 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:279 3.2s0 3:41121c54:::smithi19639362-23:head [read 0~1441792,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-02-05T04:53:55.795 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:30.195246+0000", 2024-02-05T04:53:55.795 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 25.547655076000002, 2024-02-05T04:53:55.795 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.004417514, 2024-02-05T04:53:55.795 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:55.795 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "started", 2024-02-05T04:53:55.795 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:53:55.795 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:53:55.795 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:55.795 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 279 2024-02-05T04:53:55.795 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.796 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:55.796 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.796 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:55.796 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.195246+0000", 2024-02-05T04:53:55.796 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:55.796 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.796 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.796 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:55.796 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.195246+0000", 2024-02-05T04:53:55.796 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:55.796 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.796 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.796 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:55.796 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.195247+0000", 2024-02-05T04:53:55.796 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.449e-06 2024-02-05T04:53:55.797 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.797 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.797 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:55.797 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.195253+0000", 2024-02-05T04:53:55.797 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.7819999999999999e-06 2024-02-05T04:53:55.797 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.797 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.797 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:55.797 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.195268+0000", 2024-02-05T04:53:55.797 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.4514e-05 2024-02-05T04:53:55.797 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.797 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.797 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:55.797 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.195276+0000", 2024-02-05T04:53:55.797 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.2390000000000002e-06 2024-02-05T04:53:55.797 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.798 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.798 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:55.798 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.195352+0000", 2024-02-05T04:53:55.798 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.5665000000000001e-05 2024-02-05T04:53:55.798 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.798 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.798 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:53:55.798 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.195419+0000", 2024-02-05T04:53:55.798 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.7486999999999999e-05 2024-02-05T04:53:55.798 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.798 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.798 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:55.798 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.199663+0000", 2024-02-05T04:53:55.798 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.004244378 2024-02-05T04:53:55.799 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.799 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:55.799 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.799 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.799 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.799 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:280 3.2s0 3:41121c54:::smithi19639362-23:head [read 0~1441792,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-02-05T04:53:55.799 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:30.195290+0000", 2024-02-05T04:53:55.799 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 25.547610936000002, 2024-02-05T04:53:55.799 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0053070849999999996, 2024-02-05T04:53:55.799 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:55.799 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "started", 2024-02-05T04:53:55.799 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:53:55.799 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:53:55.799 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:55.799 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 280 2024-02-05T04:53:55.799 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.800 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:55.800 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.800 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:55.800 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.195290+0000", 2024-02-05T04:53:55.800 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:55.800 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.800 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.800 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:55.800 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.195290+0000", 2024-02-05T04:53:55.800 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:55.800 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.800 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.800 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:55.800 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.195290+0000", 2024-02-05T04:53:55.800 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.4799999999999999e-07 2024-02-05T04:53:55.801 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.801 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.801 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:55.801 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.195294+0000", 2024-02-05T04:53:55.801 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.2160000000000002e-06 2024-02-05T04:53:55.801 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.801 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.801 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:55.801 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.195299+0000", 2024-02-05T04:53:55.801 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.3449999999999996e-06 2024-02-05T04:53:55.801 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.801 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.801 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:55.801 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.195302+0000", 2024-02-05T04:53:55.801 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.4249999999999998e-06 2024-02-05T04:53:55.801 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.802 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.802 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:55.802 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.195655+0000", 2024-02-05T04:53:55.802 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00035282899999999998 2024-02-05T04:53:55.802 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.802 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.802 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:53:55.802 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.195704+0000", 2024-02-05T04:53:55.802 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.8402999999999997e-05 2024-02-05T04:53:55.802 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.802 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.802 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:55.802 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.200597+0000", 2024-02-05T04:53:55.802 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0048935189999999998 2024-02-05T04:53:55.802 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.803 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:55.803 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.803 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.803 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.803 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:281 3.2s0 3:41121c54:::smithi19639362-23:head [read 0~1441792,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-02-05T04:53:55.803 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:30.195307+0000", 2024-02-05T04:53:55.803 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 25.547593753000001, 2024-02-05T04:53:55.803 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0063372669999999997, 2024-02-05T04:53:55.803 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:55.803 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "started", 2024-02-05T04:53:55.803 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:53:55.803 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:53:55.803 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:55.803 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 281 2024-02-05T04:53:55.803 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.803 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:55.804 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.804 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:55.804 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.195307+0000", 2024-02-05T04:53:55.804 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:55.804 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.804 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.804 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:55.804 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.195307+0000", 2024-02-05T04:53:55.804 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:55.804 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.804 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.804 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:55.804 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.195308+0000", 2024-02-05T04:53:55.804 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.41e-07 2024-02-05T04:53:55.804 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.805 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.805 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:55.805 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.195309+0000", 2024-02-05T04:53:55.805 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.5659999999999999e-06 2024-02-05T04:53:55.805 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.805 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.805 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:55.805 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.195312+0000", 2024-02-05T04:53:55.805 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.242e-06 2024-02-05T04:53:55.805 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.805 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.805 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:55.805 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.195316+0000", 2024-02-05T04:53:55.805 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.2559999999999998e-06 2024-02-05T04:53:55.805 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.805 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.806 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:55.806 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.195897+0000", 2024-02-05T04:53:55.806 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00058133200000000005 2024-02-05T04:53:55.806 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.806 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.806 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:53:55.806 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.195943+0000", 2024-02-05T04:53:55.806 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.6452999999999997e-05 2024-02-05T04:53:55.806 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.806 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.806 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:55.806 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.201645+0000", 2024-02-05T04:53:55.806 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.005701177 2024-02-05T04:53:55.806 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.806 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:55.806 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.807 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.807 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.807 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:397 3.2s0 3:4d05e2f1:::smithi19639362-34:head [delete] snapc 7=[7,6,4] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e43)", 2024-02-05T04:53:55.807 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:35.161905+0000", 2024-02-05T04:53:55.807 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 20.580996017, 2024-02-05T04:53:55.807 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0050118419999999999, 2024-02-05T04:53:55.807 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:55.807 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "started", 2024-02-05T04:53:55.807 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:53:55.807 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:53:55.807 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:55.807 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 397 2024-02-05T04:53:55.807 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.807 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:55.807 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.807 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:55.808 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.161905+0000", 2024-02-05T04:53:55.808 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:55.808 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.808 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.808 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:55.808 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.161905+0000", 2024-02-05T04:53:55.808 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:55.808 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.808 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.808 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:55.808 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.161907+0000", 2024-02-05T04:53:55.808 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.5689999999999999e-06 2024-02-05T04:53:55.808 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.808 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.808 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:55.809 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.161913+0000", 2024-02-05T04:53:55.809 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.3879999999999997e-06 2024-02-05T04:53:55.809 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.809 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.809 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:55.809 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.161928+0000", 2024-02-05T04:53:55.809 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.5159e-05 2024-02-05T04:53:55.809 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.809 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.809 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:55.809 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.161937+0000", 2024-02-05T04:53:55.809 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.1500000000000005e-06 2024-02-05T04:53:55.809 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.809 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.809 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:55.809 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.161997+0000", 2024-02-05T04:53:55.810 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.0149000000000003e-05 2024-02-05T04:53:55.810 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.810 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.810 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "op must wait for map", 2024-02-05T04:53:55.810 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.162002+0000", 2024-02-05T04:53:55.810 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.8969999999999999e-06 2024-02-05T04:53:55.810 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.810 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.810 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:55.810 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.163449+0000", 2024-02-05T04:53:55.810 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.001446744 2024-02-05T04:53:55.810 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.810 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.810 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:53:55.810 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.163519+0000", 2024-02-05T04:53:55.811 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.025e-05 2024-02-05T04:53:55.811 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.811 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.811 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:55.811 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.166917+0000", 2024-02-05T04:53:55.811 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.003397536 2024-02-05T04:53:55.811 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.811 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:55.811 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.811 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.811 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.811 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.6s1 43/31 ECSubWrite(tid=141, reqid=osd.13.0:138, at_version=0'0, trim_to=0'0, roll_forward_to=43'24))", 2024-02-05T04:53:55.811 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:35.182323+0000", 2024-02-05T04:53:55.811 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 20.560577964, 2024-02-05T04:53:55.811 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0048598620000000004, 2024-02-05T04:53:55.812 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:55.812 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:55.812 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:55.812 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.812 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:55.812 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.182323+0000", 2024-02-05T04:53:55.812 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:55.812 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.812 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.812 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:55.812 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.182323+0000", 2024-02-05T04:53:55.812 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:55.812 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.812 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.812 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:55.813 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.182324+0000", 2024-02-05T04:53:55.813 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.9800000000000003e-07 2024-02-05T04:53:55.813 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.813 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.813 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:55.813 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.182332+0000", 2024-02-05T04:53:55.813 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.6789999999999994e-06 2024-02-05T04:53:55.813 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.813 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.813 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:55.813 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.182347+0000", 2024-02-05T04:53:55.813 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.4460999999999999e-05 2024-02-05T04:53:55.813 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.813 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.813 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:55.813 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.182361+0000", 2024-02-05T04:53:55.814 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.3774000000000001e-05 2024-02-05T04:53:55.814 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.814 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.814 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:55.814 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.182435+0000", 2024-02-05T04:53:55.814 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.4276999999999994e-05 2024-02-05T04:53:55.814 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.814 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.814 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:55.814 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.182445+0000", 2024-02-05T04:53:55.814 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0282000000000001e-05 2024-02-05T04:53:55.814 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.814 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.814 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:55.814 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.187146+0000", 2024-02-05T04:53:55.815 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0047009460000000001 2024-02-05T04:53:55.815 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.815 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.815 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:55.815 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.187183+0000", 2024-02-05T04:53:55.815 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.6844999999999997e-05 2024-02-05T04:53:55.815 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.815 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:55.815 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.815 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.815 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.815 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.bs6 52/31 ECSubWrite(tid=202, reqid=osd.13.0:200, at_version=0'0, trim_to=0'0, roll_forward_to=52'50))", 2024-02-05T04:53:55.815 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:43.959719+0000", 2024-02-05T04:53:55.815 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 11.783182249999999, 2024-02-05T04:53:55.815 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0051711140000000001, 2024-02-05T04:53:55.815 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:55.816 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:55.816 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:55.816 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.816 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:55.816 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.959719+0000", 2024-02-05T04:53:55.816 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:55.816 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.816 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.816 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:55.816 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.959719+0000", 2024-02-05T04:53:55.816 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:55.816 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.816 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.816 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:55.816 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.959719+0000", 2024-02-05T04:53:55.817 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.6199999999999997e-07 2024-02-05T04:53:55.817 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.817 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.817 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:55.817 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.959728+0000", 2024-02-05T04:53:55.817 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.5779999999999999e-06 2024-02-05T04:53:55.817 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.817 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.817 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:55.817 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.959739+0000", 2024-02-05T04:53:55.817 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0915e-05 2024-02-05T04:53:55.817 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.817 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.817 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:55.817 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.959746+0000", 2024-02-05T04:53:55.818 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.525e-06 2024-02-05T04:53:55.818 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.818 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.818 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:55.818 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.959803+0000", 2024-02-05T04:53:55.818 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.6344000000000001e-05 2024-02-05T04:53:55.818 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.818 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.818 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:55.818 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.959812+0000", 2024-02-05T04:53:55.818 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.9469999999999992e-06 2024-02-05T04:53:55.818 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.818 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.818 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:55.818 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.964851+0000", 2024-02-05T04:53:55.819 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0050395809999999996 2024-02-05T04:53:55.819 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.819 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.819 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:55.819 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.964890+0000", 2024-02-05T04:53:55.819 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.8562000000000001e-05 2024-02-05T04:53:55.819 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.819 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:55.819 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.819 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.819 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.819 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.7s2 52/31 ECSubWrite(tid=204, reqid=osd.13.0:201, at_version=0'0, trim_to=0'0, roll_forward_to=52'21))", 2024-02-05T04:53:55.819 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:43.962223+0000", 2024-02-05T04:53:55.819 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 11.780678066, 2024-02-05T04:53:55.819 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.005675561, 2024-02-05T04:53:55.819 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:55.820 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:55.820 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:55.820 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.820 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:55.820 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.962223+0000", 2024-02-05T04:53:55.820 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:55.820 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.820 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.820 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:55.820 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.962223+0000", 2024-02-05T04:53:55.820 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:55.820 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.820 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.820 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:55.820 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.962224+0000", 2024-02-05T04:53:55.821 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.7199999999999998e-07 2024-02-05T04:53:55.821 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.821 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.821 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:55.821 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.962234+0000", 2024-02-05T04:53:55.821 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0088999999999999e-05 2024-02-05T04:53:55.821 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.821 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.821 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:55.821 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.962247+0000", 2024-02-05T04:53:55.821 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.3441000000000001e-05 2024-02-05T04:53:55.821 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.821 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.821 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:55.821 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.962256+0000", 2024-02-05T04:53:55.822 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.4470000000000006e-06 2024-02-05T04:53:55.822 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.822 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.822 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:55.822 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.962314+0000", 2024-02-05T04:53:55.822 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.8068e-05 2024-02-05T04:53:55.822 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.822 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.822 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:55.822 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.962325+0000", 2024-02-05T04:53:55.822 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0905e-05 2024-02-05T04:53:55.822 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.822 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.822 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:55.822 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.967862+0000", 2024-02-05T04:53:55.823 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0055378119999999996 2024-02-05T04:53:55.823 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.823 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.823 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:55.823 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.967899+0000", 2024-02-05T04:53:55.823 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.6127000000000002e-05 2024-02-05T04:53:55.823 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.823 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:55.823 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.823 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.823 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.823 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.7s2 52/31 ECSubWrite(tid=209, reqid=client.4634.0:497, at_version=52'25, trim_to=0'0, roll_forward_to=52'22))", 2024-02-05T04:53:55.823 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:43.976418+0000", 2024-02-05T04:53:55.823 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 11.766483158, 2024-02-05T04:53:55.823 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0047079380000000001, 2024-02-05T04:53:55.824 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:55.824 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:55.824 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:55.824 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.824 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:55.824 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.976418+0000", 2024-02-05T04:53:55.824 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:55.824 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.824 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.824 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:55.824 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.976418+0000", 2024-02-05T04:53:55.824 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:55.824 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.824 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.824 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:55.825 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.976419+0000", 2024-02-05T04:53:55.825 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.4099999999999998e-07 2024-02-05T04:53:55.825 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.825 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.825 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:55.825 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.976433+0000", 2024-02-05T04:53:55.825 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.448e-05 2024-02-05T04:53:55.825 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.825 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.825 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:55.825 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.976449+0000", 2024-02-05T04:53:55.825 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.5798e-05 2024-02-05T04:53:55.825 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.825 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.825 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:55.826 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.976460+0000", 2024-02-05T04:53:55.826 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0747000000000001e-05 2024-02-05T04:53:55.826 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.826 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.826 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:55.826 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.976517+0000", 2024-02-05T04:53:55.826 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.7176000000000003e-05 2024-02-05T04:53:55.826 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.826 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.826 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:55.826 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.976527+0000", 2024-02-05T04:53:55.826 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0502999999999999e-05 2024-02-05T04:53:55.826 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.826 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.826 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:55.826 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.981093+0000", 2024-02-05T04:53:55.827 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0045661089999999996 2024-02-05T04:53:55.827 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.827 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.827 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:55.827 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.981126+0000", 2024-02-05T04:53:55.827 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.2484000000000003e-05 2024-02-05T04:53:55.827 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.827 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:55.827 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.827 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.827 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.827 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(osd.7.23:2 3.2s0 3:41121c54:::smithi19639362-23:head [assert-version v29,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-02-05T04:53:55.827 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:49.168864+0000", 2024-02-05T04:53:55.827 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 6.5740368130000002, 2024-02-05T04:53:55.827 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.79151548400000005, 2024-02-05T04:53:55.828 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:55.828 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "started", 2024-02-05T04:53:55.828 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:55.828 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.828 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:55.828 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.168864+0000", 2024-02-05T04:53:55.828 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:55.828 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.828 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.828 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:55.828 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.168864+0000", 2024-02-05T04:53:55.828 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:55.828 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.828 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.829 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:55.829 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.168866+0000", 2024-02-05T04:53:55.829 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.6899999999999999e-06 2024-02-05T04:53:55.829 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.829 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.829 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:55.829 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.168872+0000", 2024-02-05T04:53:55.829 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.2720000000000003e-06 2024-02-05T04:53:55.829 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.829 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.829 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:55.829 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.168883+0000", 2024-02-05T04:53:55.829 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1294000000000001e-05 2024-02-05T04:53:55.829 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.829 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.830 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:55.830 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.168894+0000", 2024-02-05T04:53:55.830 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.047e-05 2024-02-05T04:53:55.830 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.830 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.830 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:55.830 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.168957+0000", 2024-02-05T04:53:55.830 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.3275000000000007e-05 2024-02-05T04:53:55.830 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.830 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.830 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for peered", 2024-02-05T04:53:55.830 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.168961+0000", 2024-02-05T04:53:55.830 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.5499999999999999e-06 2024-02-05T04:53:55.830 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.830 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.830 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:55.831 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.955268+0000", 2024-02-05T04:53:55.831 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.78630735900000004 2024-02-05T04:53:55.831 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.831 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.831 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:53:55.831 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.955409+0000", 2024-02-05T04:53:55.831 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00014131199999999999 2024-02-05T04:53:55.831 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.831 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.831 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:55.831 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.960380+0000", 2024-02-05T04:53:55.831 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0049702619999999996 2024-02-05T04:53:55.831 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.831 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:55.831 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.832 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.832 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.832 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.0s4 63/62 ECSubWrite(tid=52, reqid=client.4634.0:576, at_version=63'6, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:53:55.832 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:55.000036+0000", 2024-02-05T04:53:55.832 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 0.74286486100000004, 2024-02-05T04:53:55.832 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0038751250000000001, 2024-02-05T04:53:55.832 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:53:55.832 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:53:55.832 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:53:55.832 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.832 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:53:55.832 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:55.000036+0000", 2024-02-05T04:53:55.832 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:55.832 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.832 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.833 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:53:55.833 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:55.000036+0000", 2024-02-05T04:53:55.833 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:53:55.833 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.833 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.833 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:53:55.833 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:55.000037+0000", 2024-02-05T04:53:55.833 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.4000000000000001e-07 2024-02-05T04:53:55.833 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.833 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.833 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:53:55.833 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:55.000434+0000", 2024-02-05T04:53:55.833 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00039749700000000002 2024-02-05T04:53:55.833 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.833 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.834 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:53:55.834 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:55.000450+0000", 2024-02-05T04:53:55.834 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.5603999999999998e-05 2024-02-05T04:53:55.834 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.834 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.834 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:53:55.834 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:55.000459+0000", 2024-02-05T04:53:55.834 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.2420000000000006e-06 2024-02-05T04:53:55.834 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.834 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.834 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:53:55.834 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:55.000531+0000", 2024-02-05T04:53:55.834 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.1729000000000004e-05 2024-02-05T04:53:55.834 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.834 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.834 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:53:55.835 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:55.000542+0000", 2024-02-05T04:53:55.835 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0719e-05 2024-02-05T04:53:55.835 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.835 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.835 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:53:55.835 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:55.003877+0000", 2024-02-05T04:53:55.835 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0033353279999999998 2024-02-05T04:53:55.835 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:53:55.835 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:53:55.835 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:53:55.835 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:55.003911+0000", 2024-02-05T04:53:55.835 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.4266e-05 2024-02-05T04:53:55.835 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.835 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:55.835 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.836 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:53:55.836 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:53:55.836 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-02-05T04:53:55.837 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-02-05T04:53:55.838 INFO:tasks.ceph.osd.8.smithi062.stderr:2024-02-05T04:53:55.836+0000 7f9385bf2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 34693) UID: 0 2024-02-05T04:53:55.938 INFO:tasks.ceph.osd.3.smithi196.stderr:2024-02-05T04:53:55.937+0000 7fc8817c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34605) UID: 0 2024-02-05T04:53:55.954 INFO:teuthology.orchestra.run.smithi091.stdout:{ 2024-02-05T04:53:55.954 INFO:teuthology.orchestra.run.smithi091.stdout: "ops": [], 2024-02-05T04:53:55.954 INFO:teuthology.orchestra.run.smithi091.stdout: "num_ops": 0 2024-02-05T04:53:55.954 INFO:teuthology.orchestra.run.smithi091.stdout:} 2024-02-05T04:53:55.963 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-02-05T04:53:55.988 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-02-05T04:53:55.989 INFO:tasks.rados.rados.0.smithi196.stdout:230: done (5 left) 2024-02-05T04:53:55.989 INFO:tasks.rados.rados.0.smithi196.stdout:231: done (4 left) 2024-02-05T04:53:55.990 INFO:tasks.rados.rados.0.smithi196.stdout:235: done (3 left) 2024-02-05T04:53:55.990 INFO:tasks.rados.rados.0.smithi196.stdout:237: done (2 left) 2024-02-05T04:53:55.990 INFO:tasks.rados.rados.0.smithi196.stdout:238: done (1 left) 2024-02-05T04:53:55.990 INFO:tasks.rados.rados.0.smithi196.stdout:239: done (0 left) 2024-02-05T04:53:55.990 INFO:tasks.rados.rados.0.smithi196.stdout:240: snap_remove snap 18 2024-02-05T04:53:56.038 INFO:tasks.ceph.osd.14.smithi150.stderr:2024-02-05T04:53:56.037+0000 7f5f81083640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 34020) UID: 0 2024-02-05T04:53:56.073 INFO:teuthology.orchestra.run.smithi091.stdout:{ 2024-02-05T04:53:56.074 INFO:teuthology.orchestra.run.smithi091.stdout: "ops": [], 2024-02-05T04:53:56.074 INFO:teuthology.orchestra.run.smithi091.stdout: "complaint_time": 30, 2024-02-05T04:53:56.074 INFO:teuthology.orchestra.run.smithi091.stdout: "num_blocked_ops": 0 2024-02-05T04:53:56.074 INFO:teuthology.orchestra.run.smithi091.stdout:} 2024-02-05T04:53:56.082 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-02-05T04:53:56.139 INFO:tasks.ceph.osd.6.smithi150.stderr:2024-02-05T04:53:56.138+0000 7f6bea9f0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34015) UID: 0 2024-02-05T04:53:56.189 INFO:teuthology.orchestra.run.smithi091.stdout:{ 2024-02-05T04:53:56.189 INFO:teuthology.orchestra.run.smithi091.stdout: "size": 20, 2024-02-05T04:53:56.189 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 600, 2024-02-05T04:53:56.189 INFO:teuthology.orchestra.run.smithi091.stdout: "ops": [ 2024-02-05T04:53:56.189 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.189 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:239 3.4s0 3:2c242b83:::smithi19639362-9:head [append 0~655360] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:53:56.189 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:29.155211+0000", 2024-02-05T04:53:56.189 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 27.032036499, 2024-02-05T04:53:56.190 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0067722930000000004, 2024-02-05T04:53:56.190 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:56.190 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:56.190 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:53:56.190 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:53:56.190 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:56.190 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 239 2024-02-05T04:53:56.190 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.190 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:56.190 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.190 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:56.190 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.155211+0000", 2024-02-05T04:53:56.190 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:56.190 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.191 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.191 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:56.191 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.155211+0000", 2024-02-05T04:53:56.191 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:56.191 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.191 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.191 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:56.191 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.155212+0000", 2024-02-05T04:53:56.191 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.302e-06 2024-02-05T04:53:56.191 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.191 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.191 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:56.191 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.156080+0000", 2024-02-05T04:53:56.191 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00086819000000000002 2024-02-05T04:53:56.191 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.192 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.192 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:56.192 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.156094+0000", 2024-02-05T04:53:56.192 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.3637e-05 2024-02-05T04:53:56.192 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.192 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.192 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:56.192 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.156101+0000", 2024-02-05T04:53:56.192 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.5619999999999998e-06 2024-02-05T04:53:56.192 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.192 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.192 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:56.192 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.156158+0000", 2024-02-05T04:53:56.192 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.6595999999999998e-05 2024-02-05T04:53:56.193 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.193 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.193 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:53:56.193 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.156235+0000", 2024-02-05T04:53:56.193 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.7181999999999994e-05 2024-02-05T04:53:56.193 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.193 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.193 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:56.193 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.157325+0000", 2024-02-05T04:53:56.193 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.001089558 2024-02-05T04:53:56.193 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.193 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.193 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:56.193 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.160456+0000", 2024-02-05T04:53:56.193 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.003131694 2024-02-05T04:53:56.194 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.194 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.194 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "commit_sent", 2024-02-05T04:53:56.194 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.161914+0000", 2024-02-05T04:53:56.194 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.001457814 2024-02-05T04:53:56.194 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.194 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.194 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:56.194 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.161983+0000", 2024-02-05T04:53:56.194 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.8757999999999995e-05 2024-02-05T04:53:56.194 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:56.194 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:56.194 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:56.195 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.195 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.195 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:284 3.4s0 3:264f0b4b:::smithi19639362-35:head [setxattr _header (56) in=7b] snapc 3=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-02-05T04:53:56.195 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:30.205620+0000", 2024-02-05T04:53:56.195 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 25.981627551999999, 2024-02-05T04:53:56.195 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0067013009999999998, 2024-02-05T04:53:56.195 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:56.195 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:56.195 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:53:56.195 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:53:56.195 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:56.195 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 284 2024-02-05T04:53:56.195 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.195 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:56.195 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.195 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:56.196 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.205620+0000", 2024-02-05T04:53:56.196 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:56.196 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.196 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.196 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:56.196 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.205620+0000", 2024-02-05T04:53:56.196 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:56.196 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.196 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.196 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:56.196 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.205621+0000", 2024-02-05T04:53:56.196 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.8800000000000001e-07 2024-02-05T04:53:56.196 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.196 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.197 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:56.197 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.205626+0000", 2024-02-05T04:53:56.197 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.028e-06 2024-02-05T04:53:56.197 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.197 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.197 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:56.197 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.205634+0000", 2024-02-05T04:53:56.197 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.8820000000000001e-06 2024-02-05T04:53:56.197 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.197 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.197 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:56.197 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.205639+0000", 2024-02-05T04:53:56.197 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.6670000000000001e-06 2024-02-05T04:53:56.197 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.197 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.197 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:56.198 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.207319+0000", 2024-02-05T04:53:56.198 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.001680029 2024-02-05T04:53:56.198 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.198 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.198 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:53:56.198 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.207416+0000", 2024-02-05T04:53:56.198 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.6576000000000005e-05 2024-02-05T04:53:56.198 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.198 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.198 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:56.198 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.207826+0000", 2024-02-05T04:53:56.198 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00041016499999999999 2024-02-05T04:53:56.198 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.198 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.198 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:56.199 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.208446+0000", 2024-02-05T04:53:56.199 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00061980300000000004 2024-02-05T04:53:56.199 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.199 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.199 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "commit_sent", 2024-02-05T04:53:56.199 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.212268+0000", 2024-02-05T04:53:56.199 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0038218890000000002 2024-02-05T04:53:56.199 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.199 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.199 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:56.199 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.212321+0000", 2024-02-05T04:53:56.199 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.3374000000000001e-05 2024-02-05T04:53:56.199 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:56.199 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:56.199 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:56.199 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.200 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.200 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:285 3.4s0 3:264f0b4b:::smithi19639362-35:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-02-05T04:53:56.200 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:30.205648+0000", 2024-02-05T04:53:56.200 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 25.981599335999999, 2024-02-05T04:53:56.200 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0099691669999999993, 2024-02-05T04:53:56.200 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:56.200 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "started", 2024-02-05T04:53:56.200 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:53:56.200 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:53:56.200 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:56.200 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 285 2024-02-05T04:53:56.200 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.200 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:56.200 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.200 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:56.201 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.205648+0000", 2024-02-05T04:53:56.201 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:56.201 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.201 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.201 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:56.201 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.205648+0000", 2024-02-05T04:53:56.201 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:56.201 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.201 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.201 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:56.201 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.205648+0000", 2024-02-05T04:53:56.201 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.7100000000000002e-07 2024-02-05T04:53:56.201 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.201 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.201 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:56.202 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.205651+0000", 2024-02-05T04:53:56.202 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.277e-06 2024-02-05T04:53:56.202 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.202 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.202 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:56.202 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.205658+0000", 2024-02-05T04:53:56.202 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.1709999999999996e-06 2024-02-05T04:53:56.202 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.202 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.202 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:56.202 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.205663+0000", 2024-02-05T04:53:56.202 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.4009999999999996e-06 2024-02-05T04:53:56.202 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.202 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.202 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:56.203 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.208044+0000", 2024-02-05T04:53:56.203 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.002380957 2024-02-05T04:53:56.203 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.203 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.203 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "waiting for rw locks", 2024-02-05T04:53:56.203 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.208122+0000", 2024-02-05T04:53:56.203 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.7688999999999994e-05 2024-02-05T04:53:56.203 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.203 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.203 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:56.203 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.212367+0000", 2024-02-05T04:53:56.203 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0042449569999999997 2024-02-05T04:53:56.203 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.203 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.203 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:53:56.204 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.212502+0000", 2024-02-05T04:53:56.204 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00013503199999999999 2024-02-05T04:53:56.204 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.204 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.204 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:56.204 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.215617+0000", 2024-02-05T04:53:56.204 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0031151120000000002 2024-02-05T04:53:56.204 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:56.204 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:56.204 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:56.204 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.204 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.204 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:394 3.4s0 3:2c242b83:::smithi19639362-9:head [setxattr _header (55) in=7b] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-02-05T04:53:56.204 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:34.187824+0000", 2024-02-05T04:53:56.205 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 21.999423013000001, 2024-02-05T04:53:56.205 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0063472709999999998, 2024-02-05T04:53:56.205 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:56.205 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:56.205 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:53:56.205 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:53:56.205 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:56.205 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 394 2024-02-05T04:53:56.205 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.205 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:56.205 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.205 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:56.205 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.187824+0000", 2024-02-05T04:53:56.205 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:56.205 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.206 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.206 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:56.206 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.187824+0000", 2024-02-05T04:53:56.206 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:56.206 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.206 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.206 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:56.206 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.187826+0000", 2024-02-05T04:53:56.206 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.342e-06 2024-02-05T04:53:56.206 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.206 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.206 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:56.206 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.187832+0000", 2024-02-05T04:53:56.206 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.2040000000000002e-06 2024-02-05T04:53:56.207 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.207 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.207 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:56.207 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.187841+0000", 2024-02-05T04:53:56.207 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.0529999999999996e-06 2024-02-05T04:53:56.207 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.207 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.207 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:56.207 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.187847+0000", 2024-02-05T04:53:56.207 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.3570000000000003e-06 2024-02-05T04:53:56.207 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.207 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.207 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:56.207 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.189963+0000", 2024-02-05T04:53:56.207 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0021155969999999999 2024-02-05T04:53:56.207 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.208 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.208 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:53:56.208 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.190114+0000", 2024-02-05T04:53:56.208 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00015165600000000001 2024-02-05T04:53:56.208 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.208 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.208 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:56.208 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.190493+0000", 2024-02-05T04:53:56.208 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00037816 2024-02-05T04:53:56.208 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.208 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.208 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:56.208 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.192865+0000", 2024-02-05T04:53:56.208 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0023725840000000001 2024-02-05T04:53:56.208 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.209 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.209 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "commit_sent", 2024-02-05T04:53:56.209 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.194107+0000", 2024-02-05T04:53:56.209 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.001242139 2024-02-05T04:53:56.209 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.209 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.209 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:56.209 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.194172+0000", 2024-02-05T04:53:56.209 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.4178999999999997e-05 2024-02-05T04:53:56.209 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:56.209 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:56.209 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:56.209 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.209 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.209 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:395 3.4s0 3:2c242b83:::smithi19639362-9:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-02-05T04:53:56.210 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:34.187868+0000", 2024-02-05T04:53:56.210 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 21.999379676, 2024-02-05T04:53:56.210 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0080281970000000008, 2024-02-05T04:53:56.210 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:56.210 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "started", 2024-02-05T04:53:56.210 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:53:56.210 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:53:56.210 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:56.210 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 395 2024-02-05T04:53:56.210 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.210 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:56.210 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.210 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:56.210 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.187868+0000", 2024-02-05T04:53:56.211 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:56.211 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.211 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.211 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:56.211 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.187868+0000", 2024-02-05T04:53:56.211 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:56.211 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.211 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.211 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:56.211 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.187868+0000", 2024-02-05T04:53:56.211 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.6499999999999999e-07 2024-02-05T04:53:56.211 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.211 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.211 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:56.212 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.187872+0000", 2024-02-05T04:53:56.212 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.9659999999999998e-06 2024-02-05T04:53:56.212 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.212 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.212 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:56.212 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.187881+0000", 2024-02-05T04:53:56.212 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.8519999999999993e-06 2024-02-05T04:53:56.212 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.212 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.212 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:56.212 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.187887+0000", 2024-02-05T04:53:56.212 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.3579999999999999e-06 2024-02-05T04:53:56.212 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.212 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.212 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:56.213 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.191261+0000", 2024-02-05T04:53:56.213 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.003373095 2024-02-05T04:53:56.213 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.213 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.213 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "waiting for rw locks", 2024-02-05T04:53:56.213 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.191336+0000", 2024-02-05T04:53:56.213 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.5731999999999999e-05 2024-02-05T04:53:56.213 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.213 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.213 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:56.213 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.194503+0000", 2024-02-05T04:53:56.213 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0031670069999999999 2024-02-05T04:53:56.213 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.214 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.214 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:53:56.214 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.194590+0000", 2024-02-05T04:53:56.214 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.6650000000000006e-05 2024-02-05T04:53:56.214 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.214 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.214 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:56.214 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.195896+0000", 2024-02-05T04:53:56.214 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.001305872 2024-02-05T04:53:56.214 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:56.214 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:56.214 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:56.214 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.214 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.214 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.6s4 43/31 ECSubWrite(tid=138, reqid=osd.13.0:138, at_version=43'24, trim_to=0'0, roll_forward_to=40'22))", 2024-02-05T04:53:56.215 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:35.172360+0000", 2024-02-05T04:53:56.215 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 21.014887471000002, 2024-02-05T04:53:56.215 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00794804, 2024-02-05T04:53:56.215 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:56.215 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:56.215 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:56.215 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.215 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:56.215 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.172360+0000", 2024-02-05T04:53:56.215 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:56.215 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.215 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.215 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:56.215 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.172360+0000", 2024-02-05T04:53:56.215 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:56.216 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.216 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.216 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:56.216 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.172360+0000", 2024-02-05T04:53:56.216 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.9400000000000005e-07 2024-02-05T04:53:56.216 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.216 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.216 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:56.216 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.172384+0000", 2024-02-05T04:53:56.216 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.3186000000000001e-05 2024-02-05T04:53:56.216 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.216 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.216 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:56.217 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.172425+0000", 2024-02-05T04:53:56.217 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.1069000000000003e-05 2024-02-05T04:53:56.217 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.217 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.217 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:56.217 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.172435+0000", 2024-02-05T04:53:56.217 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.9150000000000008e-06 2024-02-05T04:53:56.217 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.217 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.217 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:56.217 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.172511+0000", 2024-02-05T04:53:56.217 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.6118000000000005e-05 2024-02-05T04:53:56.217 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.217 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.218 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:56.218 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.172521+0000", 2024-02-05T04:53:56.218 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.0429e-05 2024-02-05T04:53:56.218 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.218 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.218 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:56.218 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.180273+0000", 2024-02-05T04:53:56.218 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0077523109999999996 2024-02-05T04:53:56.218 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.218 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.218 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:56.218 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.180308+0000", 2024-02-05T04:53:56.218 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.4418000000000001e-05 2024-02-05T04:53:56.218 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:56.218 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:56.219 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:56.219 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.219 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.219 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.8s1 43/31 ECSubWrite(tid=62, reqid=osd.4.0:62, at_version=43'19, trim_to=0'0, roll_forward_to=42'17))", 2024-02-05T04:53:56.219 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:35.172380+0000", 2024-02-05T04:53:56.219 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 21.014867690999999, 2024-02-05T04:53:56.219 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0081089249999999995, 2024-02-05T04:53:56.219 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:56.219 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:56.219 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:56.219 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.219 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:56.219 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.172380+0000", 2024-02-05T04:53:56.220 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:56.220 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.220 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.220 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:56.220 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.172380+0000", 2024-02-05T04:53:56.220 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:56.220 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.220 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.220 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:56.220 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.172380+0000", 2024-02-05T04:53:56.220 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.48e-07 2024-02-05T04:53:56.220 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.220 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.220 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:56.220 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.172399+0000", 2024-02-05T04:53:56.221 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.9377000000000001e-05 2024-02-05T04:53:56.221 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.221 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.221 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:56.221 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.172407+0000", 2024-02-05T04:53:56.221 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.7729999999999999e-06 2024-02-05T04:53:56.221 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.221 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.221 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:56.221 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.172415+0000", 2024-02-05T04:53:56.221 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.8010000000000004e-06 2024-02-05T04:53:56.221 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.221 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.221 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:56.222 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.172494+0000", 2024-02-05T04:53:56.222 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.9208999999999993e-05 2024-02-05T04:53:56.222 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.222 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.222 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:56.222 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.172503+0000", 2024-02-05T04:53:56.222 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.1439999999999992e-06 2024-02-05T04:53:56.222 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.222 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.222 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:56.222 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.180452+0000", 2024-02-05T04:53:56.222 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0079488509999999998 2024-02-05T04:53:56.222 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.223 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.223 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:56.223 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.180488+0000", 2024-02-05T04:53:56.223 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.6622000000000003e-05 2024-02-05T04:53:56.223 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:56.223 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:56.223 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:56.223 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.223 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.223 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.3s2 43/31 ECSubWrite(tid=63, reqid=osd.4.0:63, at_version=43'27, trim_to=0'0, roll_forward_to=40'25))", 2024-02-05T04:53:56.223 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:35.173633+0000", 2024-02-05T04:53:56.223 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 21.013614304000001, 2024-02-05T04:53:56.223 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0074948189999999998, 2024-02-05T04:53:56.223 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:56.223 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:56.224 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:56.224 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.224 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:56.224 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.173633+0000", 2024-02-05T04:53:56.224 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:56.224 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.224 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.224 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:56.224 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.173633+0000", 2024-02-05T04:53:56.224 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:56.224 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.224 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.224 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:56.224 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.173634+0000", 2024-02-05T04:53:56.225 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.2099999999999996e-07 2024-02-05T04:53:56.225 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.225 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.225 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:56.225 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.173648+0000", 2024-02-05T04:53:56.225 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.3957999999999999e-05 2024-02-05T04:53:56.225 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.225 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.225 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:56.225 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.173657+0000", 2024-02-05T04:53:56.225 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.9289999999999993e-06 2024-02-05T04:53:56.225 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.225 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.226 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:56.226 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.173663+0000", 2024-02-05T04:53:56.226 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.2689999999999999e-06 2024-02-05T04:53:56.226 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.226 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.226 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:56.226 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.173709+0000", 2024-02-05T04:53:56.227 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.6238e-05 2024-02-05T04:53:56.227 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.227 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.227 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:56.227 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.173714+0000", 2024-02-05T04:53:56.227 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.4199999999999998e-06 2024-02-05T04:53:56.227 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.227 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.227 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:56.227 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.181098+0000", 2024-02-05T04:53:56.227 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0073840570000000003 2024-02-05T04:53:56.227 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.227 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.227 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:56.227 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.181128+0000", 2024-02-05T04:53:56.227 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.9326999999999999e-05 2024-02-05T04:53:56.228 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:56.228 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:56.228 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:56.228 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.228 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.228 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:500 3.4s0 3:2c15cb2e:::smithi19639362-6:head [append 0~524288] snapc d=[d,b,9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-02-05T04:53:56.228 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:44.963298+0000", 2024-02-05T04:53:56.228 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 11.223949054, 2024-02-05T04:53:56.228 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.010488153, 2024-02-05T04:53:56.228 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:56.228 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:56.228 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:53:56.228 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:53:56.228 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:56.228 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 500 2024-02-05T04:53:56.229 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.229 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:56.229 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.229 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:56.229 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.963298+0000", 2024-02-05T04:53:56.229 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:56.229 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.229 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.229 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:56.229 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.963298+0000", 2024-02-05T04:53:56.229 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:56.229 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.229 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.229 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:56.229 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.963301+0000", 2024-02-05T04:53:56.230 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.6769999999999999e-06 2024-02-05T04:53:56.230 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.230 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.230 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:56.230 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.965096+0000", 2024-02-05T04:53:56.230 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.001794618 2024-02-05T04:53:56.230 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.230 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.230 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:56.230 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.965120+0000", 2024-02-05T04:53:56.230 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.4161000000000001e-05 2024-02-05T04:53:56.230 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.230 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.230 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:56.230 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.965132+0000", 2024-02-05T04:53:56.231 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.2658e-05 2024-02-05T04:53:56.231 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.231 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.231 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:56.231 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.965206+0000", 2024-02-05T04:53:56.231 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.3632000000000002e-05 2024-02-05T04:53:56.231 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.231 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.231 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "op must wait for map", 2024-02-05T04:53:56.231 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.965213+0000", 2024-02-05T04:53:56.231 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.8399999999999997e-06 2024-02-05T04:53:56.231 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.231 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.231 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:56.231 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.967118+0000", 2024-02-05T04:53:56.232 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.001905688 2024-02-05T04:53:56.232 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.232 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.232 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:53:56.232 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.967190+0000", 2024-02-05T04:53:56.232 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.161e-05 2024-02-05T04:53:56.232 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.232 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.232 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:56.232 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.968120+0000", 2024-02-05T04:53:56.232 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00092962600000000002 2024-02-05T04:53:56.232 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.232 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.232 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:56.233 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.973623+0000", 2024-02-05T04:53:56.233 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0055030770000000003 2024-02-05T04:53:56.233 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.233 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.233 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "commit_sent", 2024-02-05T04:53:56.233 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.973717+0000", 2024-02-05T04:53:56.233 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.3845999999999996e-05 2024-02-05T04:53:56.233 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.233 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.233 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:56.233 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.973786+0000", 2024-02-05T04:53:56.233 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.9720000000000003e-05 2024-02-05T04:53:56.233 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:56.233 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:56.233 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:56.234 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.234 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.234 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:501 3.4s0 3:2c15cb2e:::smithi19639362-6:head [append 0~753664] snapc d=[d,b,9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-02-05T04:53:56.234 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:44.967519+0000", 2024-02-05T04:53:56.234 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 11.21972843, 2024-02-05T04:53:56.234 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.010834552000000001, 2024-02-05T04:53:56.234 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:56.234 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:56.234 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:53:56.234 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:53:56.234 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:56.234 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 501 2024-02-05T04:53:56.234 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.235 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:56.235 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.235 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:56.235 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.967519+0000", 2024-02-05T04:53:56.235 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:56.235 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.235 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.235 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:56.235 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.967519+0000", 2024-02-05T04:53:56.235 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:56.235 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.235 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.235 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:56.235 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.967520+0000", 2024-02-05T04:53:56.235 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.142e-06 2024-02-05T04:53:56.236 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.236 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.236 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:56.236 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.969864+0000", 2024-02-05T04:53:56.236 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0023435349999999999 2024-02-05T04:53:56.236 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.236 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.236 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:56.236 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.969881+0000", 2024-02-05T04:53:56.236 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.7592999999999999e-05 2024-02-05T04:53:56.236 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.236 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.236 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:56.236 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.969890+0000", 2024-02-05T04:53:56.237 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.2609999999999999e-06 2024-02-05T04:53:56.237 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.237 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.237 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:56.237 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.969975+0000", 2024-02-05T04:53:56.237 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.4382999999999995e-05 2024-02-05T04:53:56.237 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.237 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.237 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:53:56.237 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.970100+0000", 2024-02-05T04:53:56.237 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00012555199999999999 2024-02-05T04:53:56.237 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.237 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.237 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:56.237 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.971116+0000", 2024-02-05T04:53:56.237 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.001015665 2024-02-05T04:53:56.238 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.238 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.238 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:56.238 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.977653+0000", 2024-02-05T04:53:56.238 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0065369429999999999 2024-02-05T04:53:56.238 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.238 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.238 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "commit_sent", 2024-02-05T04:53:56.238 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.978281+0000", 2024-02-05T04:53:56.238 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00062775600000000004 2024-02-05T04:53:56.238 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.238 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.238 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:56.238 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.978353+0000", 2024-02-05T04:53:56.238 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.2721999999999999e-05 2024-02-05T04:53:56.239 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:56.239 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:56.239 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:56.239 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.239 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.239 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:502 3.4s0 3:2c15cb2e:::smithi19639362-6:head [append 0~409600] snapc d=[d,b,9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-02-05T04:53:56.239 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:44.970001+0000", 2024-02-05T04:53:56.239 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 11.217245811, 2024-02-05T04:53:56.239 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0088644669999999991, 2024-02-05T04:53:56.239 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:56.239 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:56.239 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:53:56.239 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:53:56.239 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:56.239 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 502 2024-02-05T04:53:56.239 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.240 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:56.240 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.240 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:56.241 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.970001+0000", 2024-02-05T04:53:56.241 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:56.241 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.241 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.241 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:56.241 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.970001+0000", 2024-02-05T04:53:56.241 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:56.241 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.241 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.241 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:56.241 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.970003+0000", 2024-02-05T04:53:56.241 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.7120000000000001e-06 2024-02-05T04:53:56.241 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.241 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.242 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:56.242 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.971437+0000", 2024-02-05T04:53:56.242 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.001433911 2024-02-05T04:53:56.242 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.242 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.243 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:56.243 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.971460+0000", 2024-02-05T04:53:56.243 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.3394e-05 2024-02-05T04:53:56.243 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.244 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.244 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:56.244 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.971482+0000", 2024-02-05T04:53:56.244 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.1137000000000001e-05 2024-02-05T04:53:56.244 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.244 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.244 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:56.244 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.972029+0000", 2024-02-05T04:53:56.244 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00054736499999999996 2024-02-05T04:53:56.244 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.244 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.244 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:53:56.244 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.972141+0000", 2024-02-05T04:53:56.244 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.000111742 2024-02-05T04:53:56.244 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.245 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.245 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:56.245 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.972917+0000", 2024-02-05T04:53:56.245 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00077584400000000003 2024-02-05T04:53:56.245 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.245 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.245 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:56.245 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.978062+0000", 2024-02-05T04:53:56.245 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0051453000000000002 2024-02-05T04:53:56.245 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.245 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.245 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "commit_sent", 2024-02-05T04:53:56.245 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.978801+0000", 2024-02-05T04:53:56.246 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00073907499999999995 2024-02-05T04:53:56.246 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.246 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.246 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:56.246 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.978866+0000", 2024-02-05T04:53:56.246 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.4987000000000006e-05 2024-02-05T04:53:56.246 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:56.246 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:56.246 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:56.246 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.246 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.246 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:503 3.4s0 3:2c15cb2e:::smithi19639362-6:head [setxattr _header (56) in=7b] snapc d=[d,b,9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-02-05T04:53:56.246 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:44.971507+0000", 2024-02-05T04:53:56.246 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 11.215740307000001, 2024-02-05T04:53:56.246 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0075038589999999999, 2024-02-05T04:53:56.247 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:56.247 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:56.247 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:53:56.247 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:53:56.247 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:56.247 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 503 2024-02-05T04:53:56.247 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.247 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:56.247 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.247 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:56.247 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.971507+0000", 2024-02-05T04:53:56.247 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:56.247 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.247 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.248 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:56.248 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.971507+0000", 2024-02-05T04:53:56.248 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:56.248 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.248 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.249 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:56.249 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.971508+0000", 2024-02-05T04:53:56.249 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.229e-06 2024-02-05T04:53:56.249 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.249 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.249 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:56.249 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.971513+0000", 2024-02-05T04:53:56.249 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.6430000000000003e-06 2024-02-05T04:53:56.249 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.249 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.249 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:56.249 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.971523+0000", 2024-02-05T04:53:56.250 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.7589999999999992e-06 2024-02-05T04:53:56.250 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.250 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.250 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:56.250 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.971529+0000", 2024-02-05T04:53:56.250 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.7109999999999998e-06 2024-02-05T04:53:56.250 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.250 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.250 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:56.250 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.973834+0000", 2024-02-05T04:53:56.250 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0023050010000000001 2024-02-05T04:53:56.250 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.250 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.250 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:53:56.250 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.973944+0000", 2024-02-05T04:53:56.250 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.000110202 2024-02-05T04:53:56.251 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.251 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.251 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:56.251 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.974385+0000", 2024-02-05T04:53:56.251 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.000440884 2024-02-05T04:53:56.251 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.251 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.251 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:56.251 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.978066+0000", 2024-02-05T04:53:56.251 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0036805060000000001 2024-02-05T04:53:56.251 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.251 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.251 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "commit_sent", 2024-02-05T04:53:56.251 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.978951+0000", 2024-02-05T04:53:56.251 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.000884756 2024-02-05T04:53:56.251 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.251 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.252 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:56.252 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.979011+0000", 2024-02-05T04:53:56.252 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.0167999999999997e-05 2024-02-05T04:53:56.252 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:56.252 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:56.252 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:56.252 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.252 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.252 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:504 3.4s0 3:2c15cb2e:::smithi19639362-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-02-05T04:53:56.252 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:44.971539+0000", 2024-02-05T04:53:56.252 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 11.215708421, 2024-02-05T04:53:56.252 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0087845130000000007, 2024-02-05T04:53:56.252 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:56.252 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "started", 2024-02-05T04:53:56.253 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:53:56.253 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:53:56.253 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:56.253 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 504 2024-02-05T04:53:56.253 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.253 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:56.253 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.253 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:56.253 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.971539+0000", 2024-02-05T04:53:56.253 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:56.253 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.253 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.253 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:56.253 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.971539+0000", 2024-02-05T04:53:56.253 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:56.253 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.254 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.254 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:56.254 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.971539+0000", 2024-02-05T04:53:56.254 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.0500000000000003e-07 2024-02-05T04:53:56.254 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.254 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.254 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:56.254 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.971542+0000", 2024-02-05T04:53:56.254 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.6970000000000002e-06 2024-02-05T04:53:56.254 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.254 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.254 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:56.254 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.971549+0000", 2024-02-05T04:53:56.254 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.9890000000000003e-06 2024-02-05T04:53:56.255 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.255 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.255 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:56.255 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.971555+0000", 2024-02-05T04:53:56.255 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.6690000000000003e-06 2024-02-05T04:53:56.255 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.255 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.255 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:56.255 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.974686+0000", 2024-02-05T04:53:56.255 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0031309939999999998 2024-02-05T04:53:56.255 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.255 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.255 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "waiting for rw locks", 2024-02-05T04:53:56.255 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.974749+0000", 2024-02-05T04:53:56.256 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.3167e-05 2024-02-05T04:53:56.256 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.256 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.256 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:56.256 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.979371+0000", 2024-02-05T04:53:56.256 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0046216349999999998 2024-02-05T04:53:56.256 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.256 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.256 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:53:56.256 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.979458+0000", 2024-02-05T04:53:56.256 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.7661e-05 2024-02-05T04:53:56.256 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.257 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.257 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:56.257 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.980323+0000", 2024-02-05T04:53:56.257 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00086509599999999999 2024-02-05T04:53:56.257 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:56.257 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:56.257 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:56.257 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.257 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.257 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.7s3 57/31 ECSubWrite(tid=238, reqid=osd.13.0:238, at_version=57'27, trim_to=0'0, roll_forward_to=52'25))", 2024-02-05T04:53:56.257 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:48.946942+0000", 2024-02-05T04:53:56.257 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 7.2403053899999996, 2024-02-05T04:53:56.257 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0069219049999999999, 2024-02-05T04:53:56.257 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:56.257 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:56.257 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:56.257 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.258 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:56.258 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.946942+0000", 2024-02-05T04:53:56.258 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:56.258 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.258 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.258 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:56.258 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.946942+0000", 2024-02-05T04:53:56.258 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:56.258 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.258 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.258 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:56.258 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.946943+0000", 2024-02-05T04:53:56.258 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.4499999999999996e-07 2024-02-05T04:53:56.258 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.258 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.258 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:56.259 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.946964+0000", 2024-02-05T04:53:56.259 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.1387999999999999e-05 2024-02-05T04:53:56.259 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.259 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.259 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:56.259 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.946978+0000", 2024-02-05T04:53:56.259 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.4251e-05 2024-02-05T04:53:56.259 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.259 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.259 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:56.259 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.946992+0000", 2024-02-05T04:53:56.259 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.4008e-05 2024-02-05T04:53:56.259 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.259 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.260 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:56.260 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.947849+0000", 2024-02-05T04:53:56.260 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.000857171 2024-02-05T04:53:56.260 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.260 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.260 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:56.260 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.947862+0000", 2024-02-05T04:53:56.260 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.24e-05 2024-02-05T04:53:56.260 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.260 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.260 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:56.260 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.953833+0000", 2024-02-05T04:53:56.260 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0059707290000000001 2024-02-05T04:53:56.260 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.260 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.261 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:56.261 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.953864+0000", 2024-02-05T04:53:56.261 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.1213000000000001e-05 2024-02-05T04:53:56.261 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:56.261 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:56.261 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:56.261 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.261 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.261 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.7s3 57/31 ECSubWrite(tid=239, reqid=osd.13.0:239, at_version=57'29, trim_to=0'0, roll_forward_to=52'25))", 2024-02-05T04:53:56.261 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:48.947283+0000", 2024-02-05T04:53:56.261 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 7.2399644050000003, 2024-02-05T04:53:56.261 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0066039970000000003, 2024-02-05T04:53:56.261 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:56.261 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:56.262 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:56.262 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.262 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:56.262 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.947283+0000", 2024-02-05T04:53:56.262 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:56.262 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.262 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.262 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:56.262 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.947283+0000", 2024-02-05T04:53:56.262 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:56.262 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.262 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.262 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:56.262 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.947283+0000", 2024-02-05T04:53:56.262 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.2399999999999999e-07 2024-02-05T04:53:56.263 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.263 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.263 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:56.263 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.947304+0000", 2024-02-05T04:53:56.263 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.1069000000000001e-05 2024-02-05T04:53:56.263 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.263 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.263 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:56.263 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.947313+0000", 2024-02-05T04:53:56.263 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.5480000000000002e-06 2024-02-05T04:53:56.263 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.263 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.263 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:56.263 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.947320+0000", 2024-02-05T04:53:56.263 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.7789999999999996e-06 2024-02-05T04:53:56.264 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.264 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.264 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:56.264 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.948226+0000", 2024-02-05T04:53:56.264 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00090585000000000004 2024-02-05T04:53:56.264 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.264 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.264 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:56.264 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.948234+0000", 2024-02-05T04:53:56.264 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.123e-06 2024-02-05T04:53:56.264 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.264 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.264 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:56.264 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.953868+0000", 2024-02-05T04:53:56.264 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0056336370000000004 2024-02-05T04:53:56.265 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.265 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.265 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:56.265 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.953887+0000", 2024-02-05T04:53:56.265 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.8766999999999998e-05 2024-02-05T04:53:56.265 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:56.265 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:56.265 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:56.265 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.265 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.265 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.fs6 58/31 ECSubWrite(tid=104, reqid=client.4634.0:518, at_version=58'51, trim_to=0'0, roll_forward_to=57'49))", 2024-02-05T04:53:56.265 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:49.973813+0000", 2024-02-05T04:53:56.265 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 6.2134342440000001, 2024-02-05T04:53:56.265 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0070122819999999999, 2024-02-05T04:53:56.265 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:56.266 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:56.266 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:56.266 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.266 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:56.266 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:49.973813+0000", 2024-02-05T04:53:56.266 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:56.266 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.266 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.266 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:56.266 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:49.973813+0000", 2024-02-05T04:53:56.266 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:56.266 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.266 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.266 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:56.267 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:49.973814+0000", 2024-02-05T04:53:56.267 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.1399999999999997e-07 2024-02-05T04:53:56.267 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.267 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.267 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:56.267 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:49.975839+0000", 2024-02-05T04:53:56.267 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.002025411 2024-02-05T04:53:56.267 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.267 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.267 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:56.267 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:49.975850+0000", 2024-02-05T04:53:56.267 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.1532000000000001e-05 2024-02-05T04:53:56.267 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.267 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.267 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:56.267 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:49.975857+0000", 2024-02-05T04:53:56.268 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.2380000000000002e-06 2024-02-05T04:53:56.268 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.268 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.268 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:56.268 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:49.975913+0000", 2024-02-05T04:53:56.268 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.6647000000000002e-05 2024-02-05T04:53:56.268 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.268 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.268 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:56.268 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:49.975924+0000", 2024-02-05T04:53:56.268 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.041e-05 2024-02-05T04:53:56.268 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.268 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.268 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:56.269 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:49.980799+0000", 2024-02-05T04:53:56.269 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.004875646 2024-02-05T04:53:56.269 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.269 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.269 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:56.269 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:49.980825+0000", 2024-02-05T04:53:56.269 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.5884000000000002e-05 2024-02-05T04:53:56.269 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:56.269 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:56.269 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:56.269 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.269 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.269 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.bs1 63/57 ECSubWrite(tid=263, reqid=osd.13.0:263, at_version=63'58, trim_to=0'0, roll_forward_to=61'56))", 2024-02-05T04:53:56.269 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:54.983538+0000", 2024-02-05T04:53:56.269 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 1.203709194, 2024-02-05T04:53:56.269 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0074848459999999999, 2024-02-05T04:53:56.270 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:56.270 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:56.270 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:56.270 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.270 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:56.270 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:54.983538+0000", 2024-02-05T04:53:56.270 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:56.270 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.270 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.270 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:56.270 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:54.983538+0000", 2024-02-05T04:53:56.270 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:56.270 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.270 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.270 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:56.270 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:54.983538+0000", 2024-02-05T04:53:56.271 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.4799999999999999e-07 2024-02-05T04:53:56.271 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.271 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.271 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:56.271 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:54.983656+0000", 2024-02-05T04:53:56.271 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.000117396 2024-02-05T04:53:56.271 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.271 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.271 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:56.271 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:54.983683+0000", 2024-02-05T04:53:56.271 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.7619000000000001e-05 2024-02-05T04:53:56.271 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.271 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.271 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:56.271 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:54.983694+0000", 2024-02-05T04:53:56.272 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.0641e-05 2024-02-05T04:53:56.272 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.272 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.272 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:56.272 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:54.984576+0000", 2024-02-05T04:53:56.272 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00088206899999999997 2024-02-05T04:53:56.272 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.272 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.272 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:56.272 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:54.984588+0000", 2024-02-05T04:53:56.272 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.2187999999999999e-05 2024-02-05T04:53:56.272 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.272 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.272 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:56.273 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:54.990980+0000", 2024-02-05T04:53:56.273 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0063921680000000002 2024-02-05T04:53:56.273 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.273 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.273 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:56.273 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:54.991023+0000", 2024-02-05T04:53:56.273 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.2416999999999997e-05 2024-02-05T04:53:56.273 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:56.273 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:56.273 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:56.273 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.273 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.273 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.5s5 63/62 ECSubWrite(tid=62, reqid=osd.15.0:62, at_version=63'42, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:53:56.273 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:54.998181+0000", 2024-02-05T04:53:56.273 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 1.1890666809999999, 2024-02-05T04:53:56.274 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0064704999999999997, 2024-02-05T04:53:56.274 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:56.274 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:56.274 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:56.274 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.274 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:56.274 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:54.998181+0000", 2024-02-05T04:53:56.274 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:56.274 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.274 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.274 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:56.274 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:54.998181+0000", 2024-02-05T04:53:56.274 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:56.275 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.275 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.275 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:56.275 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:54.998181+0000", 2024-02-05T04:53:56.275 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.3799999999999998e-07 2024-02-05T04:53:56.275 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.275 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.275 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:56.275 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:54.998201+0000", 2024-02-05T04:53:56.275 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.0049000000000001e-05 2024-02-05T04:53:56.275 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.275 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.275 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:56.275 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:54.998210+0000", 2024-02-05T04:53:56.276 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.0280000000000006e-06 2024-02-05T04:53:56.276 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.276 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.276 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:56.276 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:54.998219+0000", 2024-02-05T04:53:56.276 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.6789999999999994e-06 2024-02-05T04:53:56.276 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.276 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.276 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:56.276 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:54.998271+0000", 2024-02-05T04:53:56.276 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.2667999999999998e-05 2024-02-05T04:53:56.276 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.276 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.276 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:56.276 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:54.998281+0000", 2024-02-05T04:53:56.277 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.0729999999999994e-06 2024-02-05T04:53:56.277 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.277 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.277 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:56.277 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:55.004614+0000", 2024-02-05T04:53:56.277 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0063335279999999997 2024-02-05T04:53:56.277 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.277 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.277 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:56.277 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:55.004651+0000", 2024-02-05T04:53:56.277 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.7036999999999998e-05 2024-02-05T04:53:56.277 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:56.277 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:56.277 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:56.277 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.278 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.278 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.bs1 63/57 ECSubWrite(tid=264, reqid=osd.13.0:263, at_version=0'0, trim_to=0'0, roll_forward_to=63'58))", 2024-02-05T04:53:56.278 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:55.000320+0000", 2024-02-05T04:53:56.278 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 1.1869271219999999, 2024-02-05T04:53:56.278 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0069582419999999999, 2024-02-05T04:53:56.278 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:56.278 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:56.278 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:56.278 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.278 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:56.278 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:55.000320+0000", 2024-02-05T04:53:56.278 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:56.278 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.278 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.278 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:56.279 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:55.000320+0000", 2024-02-05T04:53:56.279 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:56.279 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.279 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.279 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:56.279 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:55.000320+0000", 2024-02-05T04:53:56.279 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.0499999999999999e-07 2024-02-05T04:53:56.279 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.279 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.279 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:56.279 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:55.000328+0000", 2024-02-05T04:53:56.279 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.9489999999999996e-06 2024-02-05T04:53:56.279 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.279 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.279 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:56.280 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:55.000339+0000", 2024-02-05T04:53:56.280 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.0200999999999999e-05 2024-02-05T04:53:56.280 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.280 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.280 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:56.280 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:55.000347+0000", 2024-02-05T04:53:56.280 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.6379999999999993e-06 2024-02-05T04:53:56.280 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.280 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.280 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:56.280 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:55.000407+0000", 2024-02-05T04:53:56.280 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.0006000000000001e-05 2024-02-05T04:53:56.280 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.280 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.281 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:56.281 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:55.000416+0000", 2024-02-05T04:53:56.281 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.4919999999999993e-06 2024-02-05T04:53:56.281 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.281 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.281 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:56.281 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:55.007235+0000", 2024-02-05T04:53:56.281 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0068194390000000001 2024-02-05T04:53:56.281 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.281 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.281 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:56.281 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:55.007278+0000", 2024-02-05T04:53:56.281 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.3211999999999999e-05 2024-02-05T04:53:56.281 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:56.282 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:56.282 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:56.282 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.282 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.282 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.cs5 63/31 ECSubWrite(tid=123, reqid=osd.11.0:120, at_version=0'0, trim_to=0'0, roll_forward_to=63'25))", 2024-02-05T04:53:56.282 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:55.000427+0000", 2024-02-05T04:53:56.282 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 1.186820451, 2024-02-05T04:53:56.282 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0068580860000000002, 2024-02-05T04:53:56.282 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:53:56.282 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:53:56.282 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:53:56.282 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.283 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:53:56.283 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:55.000427+0000", 2024-02-05T04:53:56.283 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:56.283 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.283 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.283 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:53:56.283 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:55.000427+0000", 2024-02-05T04:53:56.283 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:53:56.283 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.283 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.283 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:53:56.283 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:55.000427+0000", 2024-02-05T04:53:56.283 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.9599999999999999e-07 2024-02-05T04:53:56.283 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.283 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.284 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:53:56.284 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:55.000437+0000", 2024-02-05T04:53:56.284 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.1749999999999994e-06 2024-02-05T04:53:56.284 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.284 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.284 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:53:56.284 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:55.000448+0000", 2024-02-05T04:53:56.284 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.0985000000000001e-05 2024-02-05T04:53:56.284 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.284 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.284 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:53:56.284 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:55.000455+0000", 2024-02-05T04:53:56.284 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.7409999999999992e-06 2024-02-05T04:53:56.284 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.285 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.285 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:53:56.285 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:55.000668+0000", 2024-02-05T04:53:56.285 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00021302300000000001 2024-02-05T04:53:56.285 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.285 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.285 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:53:56.285 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:55.000679+0000", 2024-02-05T04:53:56.285 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.0454e-05 2024-02-05T04:53:56.285 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.285 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.285 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:53:56.285 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:55.007258+0000", 2024-02-05T04:53:56.285 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0065793939999999997 2024-02-05T04:53:56.285 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:53:56.286 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:53:56.286 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:53:56.286 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:55.007285+0000", 2024-02-05T04:53:56.286 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.6718000000000001e-05 2024-02-05T04:53:56.286 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:56.286 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:56.286 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:56.286 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:53:56.286 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:53:56.286 INFO:teuthology.orchestra.run.smithi091.stdout:} 2024-02-05T04:53:56.288 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-02-05T04:53:56.288 INFO:tasks.ceph.osd.1.smithi091.stderr:2024-02-05T04:53:56.287+0000 7ff9dc283640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34275) UID: 0 2024-02-05T04:53:56.388 INFO:tasks.ceph.osd.3.smithi196.stderr:2024-02-05T04:53:56.387+0000 7fc8817c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34605) UID: 0 2024-02-05T04:53:56.405 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T04:53:56.435 INFO:teuthology.orchestra.run.smithi091.stdout:{ 2024-02-05T04:53:56.435 INFO:teuthology.orchestra.run.smithi091.stdout: "ops": [], 2024-02-05T04:53:56.435 INFO:teuthology.orchestra.run.smithi091.stdout: "num_ops": 0 2024-02-05T04:53:56.435 INFO:teuthology.orchestra.run.smithi091.stdout:} 2024-02-05T04:53:56.443 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-02-05T04:53:56.489 INFO:tasks.ceph.osd.13.smithi091.stderr:2024-02-05T04:53:56.488+0000 7f526fd4d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 34281) UID: 0 2024-02-05T04:53:56.528 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0: {} 2024-02-05T04:53:56.529 INFO:teuthology.orchestra.run.smithi062.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T04:53:56.547 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1: {} 2024-02-05T04:53:56.547 INFO:teuthology.orchestra.run.smithi062.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T04:53:56.549 INFO:teuthology.orchestra.run.smithi091.stdout:{ 2024-02-05T04:53:56.549 INFO:teuthology.orchestra.run.smithi091.stdout: "ops": [], 2024-02-05T04:53:56.549 INFO:teuthology.orchestra.run.smithi091.stdout: "complaint_time": 30, 2024-02-05T04:53:56.549 INFO:teuthology.orchestra.run.smithi091.stdout: "num_blocked_ops": 0 2024-02-05T04:53:56.549 INFO:teuthology.orchestra.run.smithi091.stdout:} 2024-02-05T04:53:56.557 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-02-05T04:53:56.566 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2: {} 2024-02-05T04:53:56.566 INFO:teuthology.orchestra.run.smithi062.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T04:53:56.585 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3: {} 2024-02-05T04:53:56.585 INFO:teuthology.orchestra.run.smithi062.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T04:53:56.589 INFO:tasks.ceph.osd.10.smithi150.stderr:2024-02-05T04:53:56.588+0000 7f3075a4d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 34018) UID: 0 2024-02-05T04:53:56.602 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4: {} 2024-02-05T04:53:56.603 INFO:teuthology.orchestra.run.smithi062.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T04:53:56.620 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5: {} 2024-02-05T04:53:56.620 INFO:teuthology.orchestra.run.smithi062.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T04:53:56.639 INFO:teuthology.orchestra.run.smithi062.stdout:osd.6: {} 2024-02-05T04:53:56.639 INFO:teuthology.orchestra.run.smithi062.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T04:53:56.657 INFO:teuthology.orchestra.run.smithi062.stdout:osd.7: {} 2024-02-05T04:53:56.657 INFO:teuthology.orchestra.run.smithi062.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T04:53:56.674 INFO:teuthology.orchestra.run.smithi062.stdout:osd.8: {} 2024-02-05T04:53:56.674 INFO:teuthology.orchestra.run.smithi062.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-05T04:53:56.690 INFO:tasks.ceph.osd.13.smithi091.stderr:2024-02-05T04:53:56.690+0000 7f526fd4d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 34281) UID: 0 2024-02-05T04:53:56.692 INFO:teuthology.orchestra.run.smithi062.stdout:osd.9: {} 2024-02-05T04:53:56.692 INFO:teuthology.orchestra.run.smithi062.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-05T04:53:56.704 INFO:teuthology.orchestra.run.smithi091.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:56.704 INFO:teuthology.orchestra.run.smithi091.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:56.709 INFO:teuthology.orchestra.run.smithi062.stdout:osd.10: {} 2024-02-05T04:53:56.710 INFO:teuthology.orchestra.run.smithi062.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-05T04:53:56.713 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-02-05T04:53:56.727 INFO:teuthology.orchestra.run.smithi062.stdout:osd.11: {} 2024-02-05T04:53:56.727 INFO:teuthology.orchestra.run.smithi062.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-05T04:53:56.744 INFO:teuthology.orchestra.run.smithi062.stdout:osd.12: {} 2024-02-05T04:53:56.744 INFO:teuthology.orchestra.run.smithi062.stderr:osd.12: osd_enable_op_tracker = '' 2024-02-05T04:53:56.762 INFO:teuthology.orchestra.run.smithi062.stdout:osd.13: {} 2024-02-05T04:53:56.762 INFO:teuthology.orchestra.run.smithi062.stderr:osd.13: osd_enable_op_tracker = '' 2024-02-05T04:53:56.779 INFO:teuthology.orchestra.run.smithi062.stdout:osd.14: {} 2024-02-05T04:53:56.780 INFO:teuthology.orchestra.run.smithi062.stderr:osd.14: osd_enable_op_tracker = '' 2024-02-05T04:53:56.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-02-05T04:53:56.779+0000 7f08867d6640 0 -- 172.21.15.62:0/1917615148 >> [v2:172.21.15.196:6826/3187639683,v1:172.21.15.196:6827/3187639683] conn(0x7f088021fcd0 msgr2=0x7f08802220c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-02-05T04:53:56.791 INFO:tasks.ceph.osd.14.smithi150.stderr:2024-02-05T04:53:56.790+0000 7f5f81083640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 34020) UID: 0 2024-02-05T04:53:56.860 INFO:teuthology.orchestra.run.smithi091.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:56.860 INFO:teuthology.orchestra.run.smithi091.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:56.869 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-02-05T04:53:56.892 INFO:tasks.ceph.osd.4.smithi062.stderr:2024-02-05T04:53:56.892+0000 7f708c308640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34689) UID: 0 2024-02-05T04:53:56.969 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 12, 13, 14, 15] out_osds: [8] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-02-05T04:53:56.969 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-02-05T04:53:56.972 INFO:teuthology.orchestra.run.smithi091.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:56.973 INFO:teuthology.orchestra.run.smithi091.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:56.978 INFO:teuthology.orchestra.run.smithi062.stderr:noscrub is set 2024-02-05T04:53:56.981 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-02-05T04:53:56.987 INFO:tasks.rados.rados.0.smithi196.stdout:240: done (0 left) 2024-02-05T04:53:56.987 INFO:tasks.rados.rados.0.smithi196.stdout:241: read oid 15 snap -1 2024-02-05T04:53:56.987 INFO:tasks.rados.rados.0.smithi196.stdout:241: expect deleted 2024-02-05T04:53:56.988 INFO:tasks.rados.rados.0.smithi196.stdout:242: read oid 21 snap -1 2024-02-05T04:53:56.988 INFO:tasks.rados.rados.0.smithi196.stdout:242: expect (ObjNum 114 snap 18 seq_num 114) 2024-02-05T04:53:56.988 INFO:tasks.rados.rados.0.smithi196.stdout:243: rmattr oid 49 current snap is 19 2024-02-05T04:53:56.988 INFO:tasks.rados.rados.0.smithi196.stdout:244: read oid 8 snap -1 2024-02-05T04:53:56.988 INFO:tasks.rados.rados.0.smithi196.stdout:244: expect (ObjNum 60 snap 2 seq_num 60) 2024-02-05T04:53:56.988 INFO:tasks.rados.rados.0.smithi196.stdout:245: snap_remove snap 11 2024-02-05T04:53:57.010 INFO:teuthology.orchestra.run.smithi062.stdout:osd.15: {} 2024-02-05T04:53:57.010 INFO:teuthology.orchestra.run.smithi062.stderr:osd.15: osd_enable_op_tracker = '' 2024-02-05T04:53:57.026 INFO:tasks.ceph.osd.13.smithi091.stderr:2024-02-05T04:53:57.026+0000 7f526fd4d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 34281) UID: 0 2024-02-05T04:53:57.094 INFO:teuthology.orchestra.run.smithi091.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:57.094 INFO:teuthology.orchestra.run.smithi091.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:57.094 INFO:tasks.ceph.osd.0.smithi062.stderr:2024-02-05T04:53:57.093+0000 7fece8e03640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34688) UID: 0 2024-02-05T04:53:57.102 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-02-05T04:53:57.195 INFO:tasks.ceph.osd.3.smithi196.stderr:2024-02-05T04:53:57.194+0000 7fc8817c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34605) UID: 0 2024-02-05T04:53:57.249 INFO:teuthology.orchestra.run.smithi091.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:57.249 INFO:teuthology.orchestra.run.smithi091.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:57.257 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-02-05T04:53:57.296 INFO:tasks.ceph.osd.6.smithi150.stderr:2024-02-05T04:53:57.295+0000 7f6bea9f0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34015) UID: 0 2024-02-05T04:53:57.362 INFO:teuthology.orchestra.run.smithi091.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:57.362 INFO:teuthology.orchestra.run.smithi091.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:57.370 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-02-05T04:53:57.397 INFO:tasks.ceph.osd.6.smithi150.stderr:2024-02-05T04:53:57.396+0000 7f6bea9f0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34015) UID: 0 2024-02-05T04:53:57.474 INFO:teuthology.orchestra.run.smithi091.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:57.475 INFO:teuthology.orchestra.run.smithi091.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:57.483 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-02-05T04:53:57.497 INFO:tasks.ceph.osd.9.smithi091.stderr:2024-02-05T04:53:57.497+0000 7efc18827640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 34279) UID: 0 2024-02-05T04:53:57.598 INFO:tasks.ceph.osd.1.smithi091.stderr:2024-02-05T04:53:57.597+0000 7ff9dc283640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34275) UID: 0 2024-02-05T04:53:57.601 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:57.601 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:57.610 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-02-05T04:53:57.700 INFO:tasks.ceph.osd.15.smithi196.stderr:2024-02-05T04:53:57.698+0000 7f1facc45640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 34611) UID: 0 2024-02-05T04:53:57.715 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:57.716 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:57.724 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-02-05T04:53:57.801 INFO:tasks.ceph.osd.7.smithi196.stderr:2024-02-05T04:53:57.800+0000 7fb281419640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34606) UID: 0 2024-02-05T04:53:57.830 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:57.830 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:57.839 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-02-05T04:53:57.902 INFO:tasks.ceph.osd.1.smithi091.stderr:2024-02-05T04:53:57.901+0000 7ff9dc283640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34275) UID: 0 2024-02-05T04:53:57.944 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:57.944 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:57.953 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-02-05T04:53:57.980 INFO:teuthology.orchestra.run.smithi062.stderr:reweighted osd.0 to 1 (10000) 2024-02-05T04:53:57.988 INFO:tasks.rados.rados.0.smithi196.stdout:245: done (4 left) 2024-02-05T04:53:57.989 INFO:tasks.rados.rados.0.smithi196.stdout:246: snap_create 2024-02-05T04:53:57.989 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 49 v 0 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-02-05T04:53:57.989 INFO:tasks.rados.rados.0.smithi196.stdout:242: expect (ObjNum 114 snap 18 seq_num 114) 2024-02-05T04:53:58.004 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-02-05T04:53:58.004 INFO:tasks.ceph.osd.7.smithi196.stderr:2024-02-05T04:53:58.002+0000 7fb281419640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34606) UID: 0 2024-02-05T04:53:58.090 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:58.090 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:58.098 INFO:tasks.rados.rados.0.smithi196.stdout:244: expect (ObjNum 60 snap 2 seq_num 60) 2024-02-05T04:53:58.099 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-02-05T04:53:58.103 INFO:tasks.ceph.osd.4.smithi062.stderr:2024-02-05T04:53:58.102+0000 7f708c308640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34689) UID: 0 2024-02-05T04:53:58.204 INFO:tasks.ceph.osd.11.smithi196.stderr:2024-02-05T04:53:58.203+0000 7f85a8d86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 34609) UID: 0 2024-02-05T04:53:58.206 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:58.300 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:58.302 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-02-05T04:53:58.305 INFO:tasks.ceph.osd.1.smithi091.stderr:2024-02-05T04:53:58.304+0000 7ff9dc283640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34275) UID: 0 2024-02-05T04:53:58.405 INFO:tasks.ceph.osd.0.smithi062.stderr:2024-02-05T04:53:58.405+0000 7fece8e03640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34688) UID: 0 2024-02-05T04:53:58.408 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:58.408 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:58.416 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-02-05T04:53:58.506 INFO:tasks.ceph.osd.0.smithi062.stderr:2024-02-05T04:53:58.505+0000 7fece8e03640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34688) UID: 0 2024-02-05T04:53:58.522 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:58.523 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:58.531 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-02-05T04:53:58.607 INFO:tasks.ceph.osd.4.smithi062.stderr:2024-02-05T04:53:58.606+0000 7f708c308640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34689) UID: 0 2024-02-05T04:53:58.637 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:58.637 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:58.646 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-02-05T04:53:58.708 INFO:tasks.ceph.osd.12.smithi062.stderr:2024-02-05T04:53:58.708+0000 7f5e5847e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 34695) UID: 0 2024-02-05T04:53:58.751 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:58.751 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:58.761 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-02-05T04:53:58.809 INFO:tasks.ceph.osd.4.smithi062.stderr:2024-02-05T04:53:58.809+0000 7f708c308640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34689) UID: 0 2024-02-05T04:53:58.867 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:58.867 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:58.875 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-02-05T04:53:58.911 INFO:tasks.ceph.osd.5.smithi091.stderr:2024-02-05T04:53:58.910+0000 7f63983b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34277) UID: 0 2024-02-05T04:53:58.942 INFO:teuthology.orchestra.run.smithi062.stderr:reweighted osd.1 to 1 (10000) 2024-02-05T04:53:58.949 INFO:tasks.rados.rados.0.smithi196.stdout:241: done (4 left) 2024-02-05T04:53:58.950 INFO:tasks.rados.rados.0.smithi196.stdout:242: done (3 left) 2024-02-05T04:53:58.950 INFO:tasks.rados.rados.0.smithi196.stdout:243: done (2 left) 2024-02-05T04:53:58.950 INFO:tasks.rados.rados.0.smithi196.stdout:244: done (1 left) 2024-02-05T04:53:58.950 INFO:tasks.rados.rados.0.smithi196.stdout:246: done (0 left) 2024-02-05T04:53:58.950 INFO:tasks.rados.rados.0.smithi196.stdout:247: snap_create 2024-02-05T04:53:58.966 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-02-05T04:53:58.988 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:58.989 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:58.998 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-02-05T04:53:59.000 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-02-05T04:53:59.022 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T04:53:59.050 INFO:tasks.ceph.osd.12.smithi062.stderr:2024-02-05T04:53:59.049+0000 7f5e5847e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 34695) UID: 0 2024-02-05T04:53:59.109 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:59.109 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:59.111 INFO:tasks.ceph.osd.13.smithi091.stderr:2024-02-05T04:53:59.111+0000 7f526fd4d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 34281) UID: 0 2024-02-05T04:53:59.118 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-02-05T04:53:59.208 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0: {} 2024-02-05T04:53:59.208 INFO:teuthology.orchestra.run.smithi062.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T04:53:59.212 INFO:tasks.ceph.osd.11.smithi196.stderr:2024-02-05T04:53:59.211+0000 7f85a8d86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 34609) UID: 0 2024-02-05T04:53:59.224 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T04:53:59.224 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:53:59.233 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-02-05T04:53:59.233 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1: {} 2024-02-05T04:53:59.234 INFO:teuthology.orchestra.run.smithi062.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T04:53:59.270 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2: {} 2024-02-05T04:53:59.271 INFO:teuthology.orchestra.run.smithi062.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T04:53:59.295 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3: {} 2024-02-05T04:53:59.295 INFO:teuthology.orchestra.run.smithi062.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T04:53:59.311 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4: {} 2024-02-05T04:53:59.312 INFO:teuthology.orchestra.run.smithi062.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T04:53:59.313 INFO:tasks.ceph.osd.14.smithi150.stderr:2024-02-05T04:53:59.312+0000 7f5f81083640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 34020) UID: 0 2024-02-05T04:53:59.330 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5: {} 2024-02-05T04:53:59.347 INFO:teuthology.orchestra.run.smithi062.stdout:osd.6: {} 2024-02-05T04:53:59.348 INFO:teuthology.orchestra.run.smithi062.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T04:53:59.364 INFO:teuthology.orchestra.run.smithi062.stdout:osd.7: {} 2024-02-05T04:53:59.364 INFO:teuthology.orchestra.run.smithi062.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T04:53:59.380 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T04:53:59.380 INFO:teuthology.orchestra.run.smithi196.stdout: "size": 20, 2024-02-05T04:53:59.380 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 600, 2024-02-05T04:53:59.380 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-02-05T04:53:59.380 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.380 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.3s6 37/31 ECSubWrite(tid=31, reqid=client.4634.0:220, at_version=37'14, trim_to=0'0, roll_forward_to=37'10))", 2024-02-05T04:53:59.380 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:29.138137+0000", 2024-02-05T04:53:59.381 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 30.240121961, 2024-02-05T04:53:59.381 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0041763130000000001, 2024-02-05T04:53:59.381 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:59.381 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:59.381 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:59.381 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.381 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:59.381 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.138137+0000", 2024-02-05T04:53:59.381 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:59.381 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.381 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.381 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:59.381 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.138137+0000", 2024-02-05T04:53:59.381 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:59.382 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.382 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.382 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:59.382 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.138138+0000", 2024-02-05T04:53:59.382 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.37e-07 2024-02-05T04:53:59.382 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.382 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.382 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:59.382 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.138156+0000", 2024-02-05T04:53:59.382 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8448999999999999e-05 2024-02-05T04:53:59.382 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.382 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.382 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:59.382 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.138167+0000", 2024-02-05T04:53:59.382 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0930999999999999e-05 2024-02-05T04:53:59.382 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.383 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.383 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:59.383 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.138176+0000", 2024-02-05T04:53:59.383 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1600000000000004e-06 2024-02-05T04:53:59.383 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.383 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.383 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:59.383 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.138835+0000", 2024-02-05T04:53:59.383 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00065867499999999995 2024-02-05T04:53:59.383 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.383 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.383 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:59.383 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.138845+0000", 2024-02-05T04:53:59.383 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8109999999999997e-06 2024-02-05T04:53:59.384 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.384 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.384 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:59.384 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.142268+0000", 2024-02-05T04:53:59.384 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.003423059 2024-02-05T04:53:59.384 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.384 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.384 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:59.384 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.142313+0000", 2024-02-05T04:53:59.384 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5491000000000002e-05 2024-02-05T04:53:59.384 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.384 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:59.384 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.384 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.385 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.385 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.2s2 37/31 ECSubWrite(tid=24, reqid=client.4634.0:257, at_version=37'18, trim_to=0'0, roll_forward_to=37'12))", 2024-02-05T04:53:59.385 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:29.211387+0000", 2024-02-05T04:53:59.385 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 30.166871746999998, 2024-02-05T04:53:59.385 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0043575100000000002, 2024-02-05T04:53:59.385 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:59.385 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:59.385 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:59.385 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.385 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:59.385 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.211387+0000", 2024-02-05T04:53:59.385 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:59.385 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.386 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.386 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:59.386 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.211387+0000", 2024-02-05T04:53:59.386 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:59.386 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.386 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.386 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:59.386 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.211388+0000", 2024-02-05T04:53:59.386 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2200000000000004e-07 2024-02-05T04:53:59.386 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.386 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.386 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:59.386 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.211406+0000", 2024-02-05T04:53:59.386 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7796999999999999e-05 2024-02-05T04:53:59.386 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.386 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.386 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:59.387 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.211416+0000", 2024-02-05T04:53:59.387 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0234000000000001e-05 2024-02-05T04:53:59.387 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.387 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.387 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:59.387 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.211421+0000", 2024-02-05T04:53:59.387 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.609e-06 2024-02-05T04:53:59.387 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.387 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.387 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:59.387 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.211467+0000", 2024-02-05T04:53:59.387 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5271000000000001e-05 2024-02-05T04:53:59.387 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.387 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.387 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:59.387 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.211473+0000", 2024-02-05T04:53:59.388 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5549999999999999e-06 2024-02-05T04:53:59.388 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.388 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.388 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:59.388 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.215717+0000", 2024-02-05T04:53:59.388 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.004243592 2024-02-05T04:53:59.388 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.388 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.388 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:59.388 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.215745+0000", 2024-02-05T04:53:59.388 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.7929999999999999e-05 2024-02-05T04:53:59.388 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.388 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:59.388 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.388 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.389 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.389 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.2s2 38/31 ECSubWrite(tid=28, reqid=osd.12.0:28, at_version=38'22, trim_to=0'0, roll_forward_to=37'18))", 2024-02-05T04:53:59.389 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:30.144310+0000", 2024-02-05T04:53:59.389 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 29.233948617999999, 2024-02-05T04:53:59.389 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0042662150000000003, 2024-02-05T04:53:59.389 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:59.389 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:59.389 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:59.389 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.389 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:59.389 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.144310+0000", 2024-02-05T04:53:59.389 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:59.389 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.389 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.390 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:59.390 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.144310+0000", 2024-02-05T04:53:59.390 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:59.390 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.390 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.390 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:59.390 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.144311+0000", 2024-02-05T04:53:59.390 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4499999999999996e-07 2024-02-05T04:53:59.390 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.390 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.390 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:59.390 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.144332+0000", 2024-02-05T04:53:59.390 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0954999999999998e-05 2024-02-05T04:53:59.390 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.390 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.391 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:59.391 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.144345+0000", 2024-02-05T04:53:59.391 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.259e-05 2024-02-05T04:53:59.391 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.391 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.391 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:59.391 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.144354+0000", 2024-02-05T04:53:59.391 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.7110000000000001e-06 2024-02-05T04:53:59.391 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.391 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.391 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:59.391 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.144438+0000", 2024-02-05T04:53:59.391 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4466000000000001e-05 2024-02-05T04:53:59.391 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.391 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.391 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:59.392 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.144448+0000", 2024-02-05T04:53:59.392 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0419e-05 2024-02-05T04:53:59.392 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.395 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.395 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:59.395 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.148538+0000", 2024-02-05T04:53:59.396 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0040892769999999997 2024-02-05T04:53:59.396 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.396 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.396 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:59.396 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.148577+0000", 2024-02-05T04:53:59.396 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8952000000000002e-05 2024-02-05T04:53:59.396 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.396 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:59.396 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.396 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.396 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.396 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.2s2 38/31 ECSubWrite(tid=30, reqid=osd.12.0:30, at_version=38'24, trim_to=0'0, roll_forward_to=38'22))", 2024-02-05T04:53:59.396 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:30.155268+0000", 2024-02-05T04:53:59.397 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 29.222991197999999, 2024-02-05T04:53:59.397 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0043229849999999997, 2024-02-05T04:53:59.397 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:59.397 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:59.397 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:59.397 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.397 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:59.397 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.155268+0000", 2024-02-05T04:53:59.397 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:59.397 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.397 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.397 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:59.398 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.155268+0000", 2024-02-05T04:53:59.398 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:59.398 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.398 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.398 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:59.398 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.155269+0000", 2024-02-05T04:53:59.398 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.33e-07 2024-02-05T04:53:59.398 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.398 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.398 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:59.398 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.155291+0000", 2024-02-05T04:53:59.398 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2708e-05 2024-02-05T04:53:59.398 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.398 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.399 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:59.399 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.155308+0000", 2024-02-05T04:53:59.399 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6996000000000001e-05 2024-02-05T04:53:59.399 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.399 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.399 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:59.399 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.155318+0000", 2024-02-05T04:53:59.399 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8849999999999994e-06 2024-02-05T04:53:59.399 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.399 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.399 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:59.399 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.155388+0000", 2024-02-05T04:53:59.399 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9704999999999998e-05 2024-02-05T04:53:59.400 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.400 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.400 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:59.400 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.155399+0000", 2024-02-05T04:53:59.400 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0473e-05 2024-02-05T04:53:59.400 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.400 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.400 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:59.400 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.159532+0000", 2024-02-05T04:53:59.400 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0041339180000000003 2024-02-05T04:53:59.400 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.400 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.400 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:59.400 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.159591+0000", 2024-02-05T04:53:59.401 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8366999999999999e-05 2024-02-05T04:53:59.401 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.401 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:59.401 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.401 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.401 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.401 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.2s2 43/31 ECSubWrite(tid=48, reqid=osd.12.0:48, at_version=43'38, trim_to=0'0, roll_forward_to=42'35))", 2024-02-05T04:53:59.401 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:35.164524+0000", 2024-02-05T04:53:59.401 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 24.213734947999999, 2024-02-05T04:53:59.401 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0049387320000000004, 2024-02-05T04:53:59.401 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:59.401 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:59.401 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:59.401 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.401 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:59.402 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.164524+0000", 2024-02-05T04:53:59.402 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:59.402 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.402 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.402 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:59.402 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.164524+0000", 2024-02-05T04:53:59.402 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:59.402 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.402 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.402 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:59.402 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.164525+0000", 2024-02-05T04:53:59.402 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.2200000000000003e-07 2024-02-05T04:53:59.402 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.402 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.403 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:59.403 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.164556+0000", 2024-02-05T04:53:59.403 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.1050000000000003e-05 2024-02-05T04:53:59.403 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.403 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.403 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:59.403 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.164572+0000", 2024-02-05T04:53:59.403 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6387000000000001e-05 2024-02-05T04:53:59.403 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.403 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.403 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:59.403 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.164582+0000", 2024-02-05T04:53:59.403 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2050000000000008e-06 2024-02-05T04:53:59.403 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.403 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.403 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:59.404 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.164651+0000", 2024-02-05T04:53:59.404 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9219000000000003e-05 2024-02-05T04:53:59.404 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.404 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.404 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:59.404 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.164661+0000", 2024-02-05T04:53:59.404 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0210999999999999e-05 2024-02-05T04:53:59.404 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.404 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.404 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:59.404 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.169414+0000", 2024-02-05T04:53:59.404 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0047528470000000001 2024-02-05T04:53:59.404 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.404 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.404 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:59.405 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.169463+0000", 2024-02-05T04:53:59.405 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8990999999999999e-05 2024-02-05T04:53:59.405 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.405 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:59.405 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.405 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.405 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.405 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.2s2 43/31 ECSubWrite(tid=49, reqid=osd.12.0:48, at_version=0'0, trim_to=0'0, roll_forward_to=43'38))", 2024-02-05T04:53:59.405 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:35.170040+0000", 2024-02-05T04:53:59.405 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 24.208219445000001, 2024-02-05T04:53:59.405 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.004631652, 2024-02-05T04:53:59.405 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:59.405 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:59.405 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:59.405 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.406 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:59.406 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.170040+0000", 2024-02-05T04:53:59.406 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:59.406 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.406 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.406 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:59.406 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.170040+0000", 2024-02-05T04:53:59.406 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:59.406 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.406 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.406 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:59.406 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.170040+0000", 2024-02-05T04:53:59.406 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.0599999999999999e-07 2024-02-05T04:53:59.406 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.406 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.406 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:59.407 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.170053+0000", 2024-02-05T04:53:59.407 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2179e-05 2024-02-05T04:53:59.407 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.407 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.407 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:59.407 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.170066+0000", 2024-02-05T04:53:59.407 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2875e-05 2024-02-05T04:53:59.407 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.407 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.407 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:59.407 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.170074+0000", 2024-02-05T04:53:59.407 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.7760000000000003e-06 2024-02-05T04:53:59.407 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.407 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.407 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:59.407 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.170139+0000", 2024-02-05T04:53:59.408 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5086000000000007e-05 2024-02-05T04:53:59.408 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.408 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.408 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:59.408 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.170149+0000", 2024-02-05T04:53:59.408 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0045e-05 2024-02-05T04:53:59.408 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.408 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.408 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:59.408 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.174632+0000", 2024-02-05T04:53:59.408 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0044826570000000001 2024-02-05T04:53:59.408 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.408 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.408 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:59.408 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.174671+0000", 2024-02-05T04:53:59.409 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9128000000000002e-05 2024-02-05T04:53:59.409 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.409 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:59.409 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.409 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.409 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.409 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.ds1 43/31 ECSubWrite(tid=44, reqid=client.4634.0:401, at_version=43'24, trim_to=0'0, roll_forward_to=38'22))", 2024-02-05T04:53:59.409 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:35.183587+0000", 2024-02-05T04:53:59.409 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 24.194671946, 2024-02-05T04:53:59.409 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0048306419999999996, 2024-02-05T04:53:59.409 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:59.409 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:59.409 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:59.409 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.409 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:59.410 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.183587+0000", 2024-02-05T04:53:59.410 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:59.410 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.410 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.410 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:59.410 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.183587+0000", 2024-02-05T04:53:59.410 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:59.410 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.410 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.410 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:59.410 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.183588+0000", 2024-02-05T04:53:59.410 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5999999999999999e-07 2024-02-05T04:53:59.410 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.410 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.410 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:59.410 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.183610+0000", 2024-02-05T04:53:59.411 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2606e-05 2024-02-05T04:53:59.411 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.411 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.411 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:59.411 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.183623+0000", 2024-02-05T04:53:59.411 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2461e-05 2024-02-05T04:53:59.411 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.411 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.411 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:59.411 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.183631+0000", 2024-02-05T04:53:59.411 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8249999999999994e-06 2024-02-05T04:53:59.411 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.411 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.411 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:59.411 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.183703+0000", 2024-02-05T04:53:59.412 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1551999999999995e-05 2024-02-05T04:53:59.412 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.412 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.412 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:59.412 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.183713+0000", 2024-02-05T04:53:59.412 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0472000000000001e-05 2024-02-05T04:53:59.412 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.412 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.412 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:59.412 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.188379+0000", 2024-02-05T04:53:59.412 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0046652819999999998 2024-02-05T04:53:59.412 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.412 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.412 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:59.412 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.188418+0000", 2024-02-05T04:53:59.413 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8983999999999998e-05 2024-02-05T04:53:59.413 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.413 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:59.413 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.413 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.413 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.413 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.3s6 47/31 ECSubWrite(tid=73, reqid=client.4634.0:419, at_version=47'33, trim_to=0'0, roll_forward_to=44'31))", 2024-02-05T04:53:59.413 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:38.944689+0000", 2024-02-05T04:53:59.413 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 20.433570439, 2024-02-05T04:53:59.413 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.004725796, 2024-02-05T04:53:59.413 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:59.413 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:59.413 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:59.413 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.413 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:59.413 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:38.944689+0000", 2024-02-05T04:53:59.414 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:59.414 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.414 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.414 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:59.414 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:38.944689+0000", 2024-02-05T04:53:59.414 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:59.414 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.414 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.414 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:59.414 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:38.944689+0000", 2024-02-05T04:53:59.414 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5699999999999999e-07 2024-02-05T04:53:59.414 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.414 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.414 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:59.414 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:38.944911+0000", 2024-02-05T04:53:59.415 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00022261199999999999 2024-02-05T04:53:59.415 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.415 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.415 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:59.415 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:38.944925+0000", 2024-02-05T04:53:59.415 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4098e-05 2024-02-05T04:53:59.415 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.415 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.415 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:59.415 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:38.944939+0000", 2024-02-05T04:53:59.415 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3424000000000001e-05 2024-02-05T04:53:59.415 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.415 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.415 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:59.415 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:38.945110+0000", 2024-02-05T04:53:59.415 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00017102 2024-02-05T04:53:59.416 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.416 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.416 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:59.416 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:38.945119+0000", 2024-02-05T04:53:59.416 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2220000000000008e-06 2024-02-05T04:53:59.416 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.416 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.416 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:59.416 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:38.949379+0000", 2024-02-05T04:53:59.416 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0042596719999999999 2024-02-05T04:53:59.416 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.416 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.416 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:59.416 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:38.949414+0000", 2024-02-05T04:53:59.416 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5590999999999999e-05 2024-02-05T04:53:59.417 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.417 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:59.417 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.417 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.417 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.417 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.9s3 52/31 ECSubWrite(tid=44, reqid=osd.14.0:44, at_version=52'19, trim_to=0'0, roll_forward_to=51'17))", 2024-02-05T04:53:59.417 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:43.957868+0000", 2024-02-05T04:53:59.417 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 15.420390926, 2024-02-05T04:53:59.417 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0035345699999999999, 2024-02-05T04:53:59.417 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:59.417 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:59.417 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:59.417 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.417 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:59.417 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:43.957868+0000", 2024-02-05T04:53:59.417 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:59.418 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.418 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.418 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:59.418 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:43.957868+0000", 2024-02-05T04:53:59.418 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:59.418 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.418 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.418 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:59.418 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:43.957868+0000", 2024-02-05T04:53:59.418 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.7399999999999999e-07 2024-02-05T04:53:59.418 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.418 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.419 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:59.419 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:43.957889+0000", 2024-02-05T04:53:59.419 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0489999999999999e-05 2024-02-05T04:53:59.419 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.419 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.419 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:59.419 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:43.957900+0000", 2024-02-05T04:53:59.419 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1114999999999999e-05 2024-02-05T04:53:59.419 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.419 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.419 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:59.419 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:43.957908+0000", 2024-02-05T04:53:59.419 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9009999999999994e-06 2024-02-05T04:53:59.419 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.419 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.419 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:59.420 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:43.957982+0000", 2024-02-05T04:53:59.420 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4196999999999995e-05 2024-02-05T04:53:59.420 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.420 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.420 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:59.420 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:43.957995+0000", 2024-02-05T04:53:59.420 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2435e-05 2024-02-05T04:53:59.420 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.420 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.420 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:59.420 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:43.961350+0000", 2024-02-05T04:53:59.420 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.003355846 2024-02-05T04:53:59.420 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.420 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.420 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:59.421 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:43.961403+0000", 2024-02-05T04:53:59.421 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2312000000000003e-05 2024-02-05T04:53:59.421 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.421 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:59.421 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.421 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.421 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.421 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.ds1 57/31 ECSubWrite(tid=54, reqid=client.4634.0:525, at_version=57'34, trim_to=0'0, roll_forward_to=57'32))", 2024-02-05T04:53:59.421 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:48.959354+0000", 2024-02-05T04:53:59.421 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 10.418904639999999, 2024-02-05T04:53:59.421 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0066299729999999999, 2024-02-05T04:53:59.421 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:59.421 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:59.421 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:59.421 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.421 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:59.422 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.959354+0000", 2024-02-05T04:53:59.422 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:59.422 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.422 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.422 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:59.422 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.959354+0000", 2024-02-05T04:53:59.422 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:59.422 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.422 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.422 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:59.422 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.959355+0000", 2024-02-05T04:53:59.422 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5600000000000005e-07 2024-02-05T04:53:59.422 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.422 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.422 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:59.422 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.959379+0000", 2024-02-05T04:53:59.423 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3366e-05 2024-02-05T04:53:59.423 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.423 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.423 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:59.423 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.959391+0000", 2024-02-05T04:53:59.423 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2672e-05 2024-02-05T04:53:59.423 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.423 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.423 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:59.423 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.959398+0000", 2024-02-05T04:53:59.423 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3450000000000002e-06 2024-02-05T04:53:59.423 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.423 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.423 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:59.423 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.959467+0000", 2024-02-05T04:53:59.424 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9665000000000005e-05 2024-02-05T04:53:59.424 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.424 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.424 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:59.424 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.959477+0000", 2024-02-05T04:53:59.424 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6930000000000002e-06 2024-02-05T04:53:59.424 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.424 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.424 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:59.424 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.965940+0000", 2024-02-05T04:53:59.424 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.006463112 2024-02-05T04:53:59.424 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.424 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.424 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:59.424 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.965984+0000", 2024-02-05T04:53:59.425 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4363999999999997e-05 2024-02-05T04:53:59.425 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.425 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:59.425 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.425 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.425 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.425 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.3s6 57/31 ECSubWrite(tid=77, reqid=osd.4.0:76, at_version=0'0, trim_to=0'0, roll_forward_to=57'35))", 2024-02-05T04:53:59.425 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:48.960703+0000", 2024-02-05T04:53:59.425 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 10.417556307, 2024-02-05T04:53:59.425 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.014641594000000001, 2024-02-05T04:53:59.425 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:59.425 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:59.425 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:59.425 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.425 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:59.425 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.960703+0000", 2024-02-05T04:53:59.426 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:59.426 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.426 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.426 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:59.426 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.960703+0000", 2024-02-05T04:53:59.426 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:59.426 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.426 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.426 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:59.426 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.960704+0000", 2024-02-05T04:53:59.426 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1399999999999995e-07 2024-02-05T04:53:59.426 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.426 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.426 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:59.426 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.960715+0000", 2024-02-05T04:53:59.426 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1534e-05 2024-02-05T04:53:59.427 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.427 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.427 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:59.427 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.960734+0000", 2024-02-05T04:53:59.427 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9264999999999999e-05 2024-02-05T04:53:59.427 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.427 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.427 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:59.427 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.960744+0000", 2024-02-05T04:53:59.427 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.7610000000000002e-06 2024-02-05T04:53:59.427 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.427 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.427 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:59.427 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.960808+0000", 2024-02-05T04:53:59.427 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4257000000000005e-05 2024-02-05T04:53:59.428 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.428 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.428 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:59.428 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.960819+0000", 2024-02-05T04:53:59.428 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0158e-05 2024-02-05T04:53:59.428 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.428 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.428 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:59.428 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.975292+0000", 2024-02-05T04:53:59.428 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.014473803 2024-02-05T04:53:59.428 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.428 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.428 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:59.428 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.975344+0000", 2024-02-05T04:53:59.428 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1901999999999999e-05 2024-02-05T04:53:59.428 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.429 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:59.429 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.429 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.429 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.429 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.ds1 57/31 ECSubWrite(tid=55, reqid=client.4634.0:525, at_version=0'0, trim_to=0'0, roll_forward_to=57'34))", 2024-02-05T04:53:59.429 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:48.966615+0000", 2024-02-05T04:53:59.429 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 10.411644075, 2024-02-05T04:53:59.429 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0093695359999999995, 2024-02-05T04:53:59.429 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:59.429 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:59.429 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:59.429 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.429 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:59.429 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.966615+0000", 2024-02-05T04:53:59.429 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:59.430 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.430 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.430 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:59.430 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.966615+0000", 2024-02-05T04:53:59.431 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:59.431 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.431 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.431 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:59.431 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.966616+0000", 2024-02-05T04:53:59.431 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.7000000000000003e-07 2024-02-05T04:53:59.431 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.432 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.432 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:59.432 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.966638+0000", 2024-02-05T04:53:59.432 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1980999999999998e-05 2024-02-05T04:53:59.432 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.432 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.432 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:59.432 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.966656+0000", 2024-02-05T04:53:59.432 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8550999999999999e-05 2024-02-05T04:53:59.432 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.432 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.432 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:59.432 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.966674+0000", 2024-02-05T04:53:59.433 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7510999999999999e-05 2024-02-05T04:53:59.433 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.433 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.433 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:59.433 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.966728+0000", 2024-02-05T04:53:59.433 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4001e-05 2024-02-05T04:53:59.433 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.433 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.433 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:59.433 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.966737+0000", 2024-02-05T04:53:59.433 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.6759999999999996e-06 2024-02-05T04:53:59.433 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.433 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.433 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:59.433 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.975954+0000", 2024-02-05T04:53:59.434 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0092173919999999996 2024-02-05T04:53:59.434 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.434 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.434 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:59.434 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.975984+0000", 2024-02-05T04:53:59.434 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.0454e-05 2024-02-05T04:53:59.434 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.434 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:59.434 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.435 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.435 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.435 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.5s3 58/57 ECSubWrite(tid=57, reqid=osd.15.0:57, at_version=58'36, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:53:59.435 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:49.955234+0000", 2024-02-05T04:53:59.435 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 9.4230254589999998, 2024-02-05T04:53:59.435 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0090458080000000007, 2024-02-05T04:53:59.435 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:59.435 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:59.435 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:59.435 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.435 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:59.435 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.955234+0000", 2024-02-05T04:53:59.435 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:59.436 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.436 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.436 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:59.436 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.955234+0000", 2024-02-05T04:53:59.436 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:59.437 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.437 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.437 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:59.437 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.955234+0000", 2024-02-05T04:53:59.437 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1699999999999997e-07 2024-02-05T04:53:59.437 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.437 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.437 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:59.437 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.955259+0000", 2024-02-05T04:53:59.437 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.4264e-05 2024-02-05T04:53:59.437 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.438 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.438 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:59.438 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.955275+0000", 2024-02-05T04:53:59.438 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5896000000000002e-05 2024-02-05T04:53:59.438 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.438 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.438 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:59.438 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.955285+0000", 2024-02-05T04:53:59.438 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0643e-05 2024-02-05T04:53:59.438 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.438 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.438 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:59.438 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.955388+0000", 2024-02-05T04:53:59.438 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010303599999999999 2024-02-05T04:53:59.438 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.438 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.439 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:59.439 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.955400+0000", 2024-02-05T04:53:59.439 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1582e-05 2024-02-05T04:53:59.439 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.439 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.439 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:59.439 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.964224+0000", 2024-02-05T04:53:59.439 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0088238989999999996 2024-02-05T04:53:59.439 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.439 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.439 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:59.439 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.964279+0000", 2024-02-05T04:53:59.440 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5571e-05 2024-02-05T04:53:59.440 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.440 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:59.440 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.440 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.441 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.441 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.5s3 58/57 ECSubWrite(tid=58, reqid=osd.15.0:58, at_version=58'38, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:53:59.441 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:49.956373+0000", 2024-02-05T04:53:59.441 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 9.4218857370000002, 2024-02-05T04:53:59.441 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.010807686, 2024-02-05T04:53:59.441 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:59.441 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:59.441 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:59.441 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.441 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:59.441 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.956373+0000", 2024-02-05T04:53:59.441 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:59.441 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.441 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.441 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:59.442 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.956373+0000", 2024-02-05T04:53:59.442 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:59.442 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.442 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.442 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:59.442 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.956374+0000", 2024-02-05T04:53:59.442 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.9899999999999999e-07 2024-02-05T04:53:59.442 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.442 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.442 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:59.442 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.956401+0000", 2024-02-05T04:53:59.442 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.6404e-05 2024-02-05T04:53:59.442 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.442 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.442 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:59.442 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.956417+0000", 2024-02-05T04:53:59.443 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.613e-05 2024-02-05T04:53:59.443 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.443 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.443 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:59.443 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.956441+0000", 2024-02-05T04:53:59.443 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3836e-05 2024-02-05T04:53:59.443 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.443 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.443 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:59.443 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.956524+0000", 2024-02-05T04:53:59.443 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3902999999999999e-05 2024-02-05T04:53:59.443 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.443 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.443 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:59.443 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.956536+0000", 2024-02-05T04:53:59.444 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.118e-05 2024-02-05T04:53:59.445 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.445 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.445 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:59.445 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.967117+0000", 2024-02-05T04:53:59.445 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.010581620999999999 2024-02-05T04:53:59.445 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.445 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.445 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:59.445 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.967181+0000", 2024-02-05T04:53:59.446 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3713000000000005e-05 2024-02-05T04:53:59.446 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.446 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:59.446 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.446 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.446 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.446 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.5s3 58/57 ECSubWrite(tid=59, reqid=osd.15.0:58, at_version=0'0, trim_to=0'0, roll_forward_to=58'38))", 2024-02-05T04:53:59.446 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:49.967928+0000", 2024-02-05T04:53:59.446 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 9.41033075, 2024-02-05T04:53:59.446 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0048870759999999997, 2024-02-05T04:53:59.447 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:59.447 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:59.447 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:59.447 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.447 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:59.447 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.967928+0000", 2024-02-05T04:53:59.447 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:59.447 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.447 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.447 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:59.447 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.967928+0000", 2024-02-05T04:53:59.447 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:59.447 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.447 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.448 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:59.448 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.967930+0000", 2024-02-05T04:53:59.448 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5e-06 2024-02-05T04:53:59.448 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.448 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.448 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:59.448 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.967948+0000", 2024-02-05T04:53:59.448 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8573e-05 2024-02-05T04:53:59.448 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.448 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.448 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:59.448 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.967965+0000", 2024-02-05T04:53:59.448 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6888000000000002e-05 2024-02-05T04:53:59.448 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.448 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.449 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:59.449 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.967978+0000", 2024-02-05T04:53:59.449 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.257e-05 2024-02-05T04:53:59.449 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.449 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.449 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:59.449 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.968058+0000", 2024-02-05T04:53:59.449 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0210999999999995e-05 2024-02-05T04:53:59.449 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.449 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.449 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:59.449 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.968070+0000", 2024-02-05T04:53:59.449 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2264e-05 2024-02-05T04:53:59.449 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.449 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.450 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:59.450 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.972768+0000", 2024-02-05T04:53:59.450 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0046979709999999996 2024-02-05T04:53:59.450 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.450 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.450 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:59.450 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.972815+0000", 2024-02-05T04:53:59.450 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7098999999999997e-05 2024-02-05T04:53:59.450 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.450 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:59.450 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.450 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.450 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.450 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.5s3 58/57 ECSubWrite(tid=60, reqid=osd.15.0:60, at_version=58'40, trim_to=0'0, roll_forward_to=58'38))", 2024-02-05T04:53:59.450 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:49.969331+0000", 2024-02-05T04:53:59.450 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 9.4089280399999993, 2024-02-05T04:53:59.451 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0094108439999999998, 2024-02-05T04:53:59.451 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:59.451 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:59.451 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:59.451 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.451 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:59.451 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.969331+0000", 2024-02-05T04:53:59.451 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:59.451 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.451 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.451 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:59.451 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.969331+0000", 2024-02-05T04:53:59.451 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:59.451 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.452 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.452 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:59.452 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.969332+0000", 2024-02-05T04:53:59.452 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2419999999999999e-06 2024-02-05T04:53:59.452 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.452 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.452 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:59.452 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.969362+0000", 2024-02-05T04:53:59.452 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.9696000000000002e-05 2024-02-05T04:53:59.452 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.452 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.452 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:59.453 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.969382+0000", 2024-02-05T04:53:59.453 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9945999999999998e-05 2024-02-05T04:53:59.453 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.453 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.453 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:59.453 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.969393+0000", 2024-02-05T04:53:59.453 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1015999999999999e-05 2024-02-05T04:53:59.453 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.453 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.453 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:59.453 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.969485+0000", 2024-02-05T04:53:59.453 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2377e-05 2024-02-05T04:53:59.453 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.453 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.453 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:59.454 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.969496+0000", 2024-02-05T04:53:59.454 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.065e-05 2024-02-05T04:53:59.454 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.454 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.454 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:59.454 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.978687+0000", 2024-02-05T04:53:59.454 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0091914130000000007 2024-02-05T04:53:59.454 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.454 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.454 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:59.454 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.978742+0000", 2024-02-05T04:53:59.454 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4503999999999999e-05 2024-02-05T04:53:59.454 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.454 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:59.455 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.455 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.455 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.455 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.ds1 58/31 ECSubWrite(tid=56, reqid=client.4634.0:526, at_version=58'35, trim_to=0'0, roll_forward_to=57'34))", 2024-02-05T04:53:59.455 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:49.970064+0000", 2024-02-05T04:53:59.455 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 9.4081952750000006, 2024-02-05T04:53:59.455 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.011474428, 2024-02-05T04:53:59.455 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:59.455 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:59.455 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:59.455 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.455 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:59.455 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.970064+0000", 2024-02-05T04:53:59.456 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:59.456 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.456 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.456 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:59.456 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.970064+0000", 2024-02-05T04:53:59.456 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:59.456 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.456 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.456 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:59.456 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.970064+0000", 2024-02-05T04:53:59.456 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4099999999999998e-07 2024-02-05T04:53:59.456 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.456 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.456 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:59.457 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.970616+0000", 2024-02-05T04:53:59.457 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00055171500000000004 2024-02-05T04:53:59.457 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.457 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.457 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:59.457 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.970636+0000", 2024-02-05T04:53:59.457 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0244000000000001e-05 2024-02-05T04:53:59.457 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.457 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.457 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:59.457 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.970648+0000", 2024-02-05T04:53:59.457 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1678000000000001e-05 2024-02-05T04:53:59.457 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.457 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.457 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:59.458 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.970755+0000", 2024-02-05T04:53:59.458 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000107188 2024-02-05T04:53:59.458 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.458 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.458 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:59.458 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.970769+0000", 2024-02-05T04:53:59.458 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3346000000000001e-05 2024-02-05T04:53:59.458 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.458 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.458 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:59.458 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.981495+0000", 2024-02-05T04:53:59.458 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.010726352999999999 2024-02-05T04:53:59.458 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.458 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.458 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:59.459 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.981538+0000", 2024-02-05T04:53:59.459 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3163e-05 2024-02-05T04:53:59.459 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.459 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:59.459 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.459 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.459 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.459 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.5s3 58/57 ECSubWrite(tid=61, reqid=osd.15.0:60, at_version=0'0, trim_to=0'0, roll_forward_to=58'40))", 2024-02-05T04:53:59.459 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:49.979590+0000", 2024-02-05T04:53:59.459 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 9.3986694449999995, 2024-02-05T04:53:59.459 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0046492290000000004, 2024-02-05T04:53:59.459 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:59.459 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:59.459 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:59.459 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.460 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:59.460 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.979590+0000", 2024-02-05T04:53:59.460 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:59.460 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.460 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.460 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:59.460 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.979590+0000", 2024-02-05T04:53:59.460 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:59.460 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.460 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.460 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:59.460 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.979590+0000", 2024-02-05T04:53:59.460 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8599999999999997e-07 2024-02-05T04:53:59.460 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.460 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.461 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:59.461 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.979603+0000", 2024-02-05T04:53:59.461 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2153e-05 2024-02-05T04:53:59.461 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.461 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.461 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:59.461 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.979617+0000", 2024-02-05T04:53:59.461 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.431e-05 2024-02-05T04:53:59.461 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.461 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.461 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:59.461 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.979629+0000", 2024-02-05T04:53:59.461 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1877e-05 2024-02-05T04:53:59.461 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.461 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.462 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:59.462 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.979700+0000", 2024-02-05T04:53:59.462 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0937000000000002e-05 2024-02-05T04:53:59.462 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.462 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.462 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:59.462 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.979710+0000", 2024-02-05T04:53:59.462 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0332999999999999e-05 2024-02-05T04:53:59.462 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.462 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.462 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:59.462 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.984201+0000", 2024-02-05T04:53:59.462 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0044911150000000004 2024-02-05T04:53:59.462 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.462 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.462 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:59.463 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.984239+0000", 2024-02-05T04:53:59.463 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7617999999999998e-05 2024-02-05T04:53:59.463 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.463 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:59.463 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.463 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.463 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.463 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.ds1 63/31 ECSubWrite(tid=61, reqid=osd.7.0:61, at_version=63'39, trim_to=0'0, roll_forward_to=61'37))", 2024-02-05T04:53:59.463 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:54.975339+0000", 2024-02-05T04:53:59.463 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 4.4029203790000002, 2024-02-05T04:53:59.463 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0039558989999999997, 2024-02-05T04:53:59.463 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:59.463 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:59.463 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:59.463 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.464 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:59.464 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:54.975339+0000", 2024-02-05T04:53:59.464 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:59.464 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.464 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.464 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:59.464 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:54.975339+0000", 2024-02-05T04:53:59.464 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:59.464 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.464 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.464 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:59.464 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:54.975339+0000", 2024-02-05T04:53:59.464 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8500000000000002e-07 2024-02-05T04:53:59.464 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.464 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.465 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:59.465 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:54.975361+0000", 2024-02-05T04:53:59.465 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1928000000000001e-05 2024-02-05T04:53:59.465 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.465 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.465 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:59.465 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:54.975375+0000", 2024-02-05T04:53:59.465 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.453e-05 2024-02-05T04:53:59.465 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.465 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.465 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:59.465 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:54.975388+0000", 2024-02-05T04:53:59.465 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2296000000000001e-05 2024-02-05T04:53:59.465 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.465 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.465 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:59.466 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:54.975465+0000", 2024-02-05T04:53:59.466 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7070000000000006e-05 2024-02-05T04:53:59.466 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.466 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.466 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:59.466 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:54.975479+0000", 2024-02-05T04:53:59.466 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4647999999999999e-05 2024-02-05T04:53:59.466 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.466 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.466 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:59.466 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:54.979266+0000", 2024-02-05T04:53:59.466 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.003786743 2024-02-05T04:53:59.466 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.466 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.466 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:59.467 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:54.979295+0000", 2024-02-05T04:53:59.467 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8399e-05 2024-02-05T04:53:59.467 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.467 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:59.467 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.467 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.467 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.467 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.5s3 63/62 ECSubWrite(tid=63, reqid=osd.15.0:62, at_version=0'0, trim_to=0'0, roll_forward_to=63'42))", 2024-02-05T04:53:59.467 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:55.010573+0000", 2024-02-05T04:53:59.467 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 4.3676856949999996, 2024-02-05T04:53:59.467 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0044064509999999996, 2024-02-05T04:53:59.467 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:59.467 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:59.467 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:59.467 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.468 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:59.468 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.010573+0000", 2024-02-05T04:53:59.468 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:59.468 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.468 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.468 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:59.468 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.010573+0000", 2024-02-05T04:53:59.468 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:59.468 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.468 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.468 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:59.468 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.010574+0000", 2024-02-05T04:53:59.468 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1100000000000004e-07 2024-02-05T04:53:59.468 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.469 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.469 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:59.469 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.010597+0000", 2024-02-05T04:53:59.469 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2807999999999999e-05 2024-02-05T04:53:59.469 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.469 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.469 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:59.469 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.010613+0000", 2024-02-05T04:53:59.469 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5724999999999999e-05 2024-02-05T04:53:59.469 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.469 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.469 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:59.469 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.010622+0000", 2024-02-05T04:53:59.469 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6390000000000004e-06 2024-02-05T04:53:59.469 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.470 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.470 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:59.470 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.010685+0000", 2024-02-05T04:53:59.470 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.2607999999999994e-05 2024-02-05T04:53:59.470 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.470 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.470 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:59.470 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.010697+0000", 2024-02-05T04:53:59.470 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1756999999999999e-05 2024-02-05T04:53:59.470 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.470 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.470 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:59.470 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.014938+0000", 2024-02-05T04:53:59.470 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0042409179999999998 2024-02-05T04:53:59.470 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.471 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.471 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:59.471 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.014980+0000", 2024-02-05T04:53:59.471 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2085000000000001e-05 2024-02-05T04:53:59.471 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.471 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:59.471 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.471 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.471 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:59.471 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T04:53:59.472 INFO:teuthology.orchestra.run.smithi062.stdout:osd.8: {} 2024-02-05T04:53:59.473 INFO:tasks.ceph.osd.2.smithi150.stderr:2024-02-05T04:53:59.471+0000 7f7337bf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 34014) UID: 0 2024-02-05T04:53:59.473 INFO:teuthology.orchestra.run.smithi062.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-05T04:53:59.474 INFO:teuthology.orchestra.run.smithi062.stdout:osd.9: {} 2024-02-05T04:53:59.474 INFO:teuthology.orchestra.run.smithi062.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-05T04:53:59.474 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-02-05T04:53:59.475 INFO:teuthology.orchestra.run.smithi062.stdout:osd.10: {} 2024-02-05T04:53:59.475 INFO:teuthology.orchestra.run.smithi062.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-05T04:53:59.475 INFO:teuthology.orchestra.run.smithi062.stdout:osd.11: {} 2024-02-05T04:53:59.476 INFO:teuthology.orchestra.run.smithi062.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-05T04:53:59.476 INFO:teuthology.orchestra.run.smithi062.stdout:osd.12: {} 2024-02-05T04:53:59.476 INFO:teuthology.orchestra.run.smithi062.stderr:osd.12: osd_enable_op_tracker = '' 2024-02-05T04:53:59.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-02-05T04:53:59.468+0000 7feec9281640 0 -- 172.21.15.62:0/175288668 >> [v2:172.21.15.91:6809/3656255795,v1:172.21.15.91:6812/3656255795] conn(0x7feec4217160 msgr2=0x7feec4219550 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-02-05T04:53:59.477 INFO:teuthology.orchestra.run.smithi062.stdout:osd.13: {} 2024-02-05T04:53:59.488 INFO:teuthology.orchestra.run.smithi062.stdout:osd.14: {} 2024-02-05T04:53:59.488 INFO:teuthology.orchestra.run.smithi062.stderr:osd.14: osd_enable_op_tracker = '' 2024-02-05T04:53:59.505 INFO:teuthology.orchestra.run.smithi062.stdout:osd.15: {} 2024-02-05T04:53:59.506 INFO:teuthology.orchestra.run.smithi062.stderr:osd.15: osd_enable_op_tracker = '' 2024-02-05T04:53:59.573 INFO:tasks.ceph.osd.14.smithi150.stderr:2024-02-05T04:53:59.572+0000 7f5f81083640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 34020) UID: 0 2024-02-05T04:53:59.582 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T04:53:59.582 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-02-05T04:53:59.583 INFO:teuthology.orchestra.run.smithi196.stdout: "num_ops": 0 2024-02-05T04:53:59.583 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T04:53:59.591 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-02-05T04:53:59.674 INFO:tasks.ceph.osd.1.smithi091.stderr:2024-02-05T04:53:59.673+0000 7ff9dc283640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34275) UID: 0 2024-02-05T04:53:59.696 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T04:53:59.697 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-02-05T04:53:59.697 INFO:teuthology.orchestra.run.smithi196.stdout: "complaint_time": 30, 2024-02-05T04:53:59.697 INFO:teuthology.orchestra.run.smithi196.stdout: "num_blocked_ops": 0 2024-02-05T04:53:59.697 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T04:53:59.705 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-02-05T04:53:59.774 INFO:tasks.ceph.osd.8.smithi062.stderr:2024-02-05T04:53:59.774+0000 7f9385bf2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 34693) UID: 0 2024-02-05T04:53:59.813 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T04:53:59.813 INFO:teuthology.orchestra.run.smithi196.stdout: "size": 20, 2024-02-05T04:53:59.813 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 600, 2024-02-05T04:53:59.813 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-02-05T04:53:59.814 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.814 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:225 3.ds0 3:b4e8e82c:::smithi19639362-25:head [copy-from ver 6] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:53:59.814 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:29.135146+0000", 2024-02-05T04:53:59.814 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 30.676505514999999, 2024-02-05T04:53:59.814 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.016925534999999998, 2024-02-05T04:53:59.814 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:59.814 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:59.814 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:53:59.814 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:53:59.814 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:59.814 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 225 2024-02-05T04:53:59.814 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.814 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:59.814 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.814 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:59.815 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135146+0000", 2024-02-05T04:53:59.815 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:59.815 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.815 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.815 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:59.815 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135146+0000", 2024-02-05T04:53:59.815 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:59.815 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.815 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.815 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:59.815 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135148+0000", 2024-02-05T04:53:59.815 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5170000000000002e-06 2024-02-05T04:53:59.815 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.815 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.815 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:59.816 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135158+0000", 2024-02-05T04:53:59.816 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.3819999999999993e-06 2024-02-05T04:53:59.816 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.816 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.816 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:59.816 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135177+0000", 2024-02-05T04:53:59.816 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9513000000000001e-05 2024-02-05T04:53:59.816 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.816 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.816 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:59.816 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135187+0000", 2024-02-05T04:53:59.816 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0054000000000001e-05 2024-02-05T04:53:59.816 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.817 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.817 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:59.817 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135292+0000", 2024-02-05T04:53:59.817 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000104651 2024-02-05T04:53:59.817 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.817 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.817 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:53:59.817 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135405+0000", 2024-02-05T04:53:59.817 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011277099999999999 2024-02-05T04:53:59.817 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.817 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.817 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:59.817 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.146446+0000", 2024-02-05T04:53:59.817 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.011041535 2024-02-05T04:53:59.817 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.817 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.818 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:59.818 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.148233+0000", 2024-02-05T04:53:59.818 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001787089 2024-02-05T04:53:59.818 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.818 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.818 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T04:53:59.818 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.151960+0000", 2024-02-05T04:53:59.818 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0037264720000000002 2024-02-05T04:53:59.818 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.818 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.818 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:59.818 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.152071+0000", 2024-02-05T04:53:59.818 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000111551 2024-02-05T04:53:59.818 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.819 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:59.819 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.819 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.819 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.819 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:226 3.ds0 3:b4e8e82c:::smithi19639362-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:53:59.819 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:29.135222+0000", 2024-02-05T04:53:59.819 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 30.676428994999998, 2024-02-05T04:53:59.819 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.012833986, 2024-02-05T04:53:59.819 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:59.819 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-02-05T04:53:59.819 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:53:59.819 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:53:59.819 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:59.819 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 226 2024-02-05T04:53:59.819 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.819 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:59.820 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.820 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:59.820 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135222+0000", 2024-02-05T04:53:59.820 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:59.820 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.820 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.820 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:59.820 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135222+0000", 2024-02-05T04:53:59.820 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:59.820 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.820 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.820 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:59.820 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135223+0000", 2024-02-05T04:53:59.820 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4099999999999998e-07 2024-02-05T04:53:59.821 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.821 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.821 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:59.821 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135227+0000", 2024-02-05T04:53:59.821 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8399999999999997e-06 2024-02-05T04:53:59.821 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.821 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.821 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:59.821 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135237+0000", 2024-02-05T04:53:59.821 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8290000000000003e-06 2024-02-05T04:53:59.821 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.821 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.821 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:59.821 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135244+0000", 2024-02-05T04:53:59.821 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4989999999999997e-06 2024-02-05T04:53:59.821 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.822 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.822 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:59.822 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135624+0000", 2024-02-05T04:53:59.822 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00037992100000000002 2024-02-05T04:53:59.822 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.822 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.822 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for blocked object", 2024-02-05T04:53:59.822 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135672+0000", 2024-02-05T04:53:59.822 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7861999999999997e-05 2024-02-05T04:53:59.822 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.822 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.822 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:59.822 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.147862+0000", 2024-02-05T04:53:59.822 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.012189847 2024-02-05T04:53:59.823 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.823 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.823 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:53:59.823 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.147947+0000", 2024-02-05T04:53:59.823 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4862000000000002e-05 2024-02-05T04:53:59.823 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.823 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.823 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:59.823 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.148056+0000", 2024-02-05T04:53:59.823 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000109585 2024-02-05T04:53:59.823 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.823 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:59.823 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.823 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.823 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.824 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:388 3.ds0 3:b4e8e82c:::smithi19639362-25:head [read 0~638976,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-02-05T04:53:59.824 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:34.171871+0000", 2024-02-05T04:53:59.824 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 25.639780569999999, 2024-02-05T04:53:59.824 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0058757660000000001, 2024-02-05T04:53:59.824 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:59.824 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-02-05T04:53:59.824 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:53:59.824 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:53:59.824 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:59.824 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 388 2024-02-05T04:53:59.824 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.824 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:59.824 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.824 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:59.824 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.171871+0000", 2024-02-05T04:53:59.825 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:59.825 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.825 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.825 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:59.825 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.171871+0000", 2024-02-05T04:53:59.825 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:59.825 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.825 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.825 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:59.825 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.171871+0000", 2024-02-05T04:53:59.825 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8899999999999999e-07 2024-02-05T04:53:59.825 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.825 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.825 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:59.825 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.171875+0000", 2024-02-05T04:53:59.826 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.777e-06 2024-02-05T04:53:59.826 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.826 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.826 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:59.826 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.171884+0000", 2024-02-05T04:53:59.826 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8559999999999996e-06 2024-02-05T04:53:59.826 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.826 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.826 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:59.826 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.171892+0000", 2024-02-05T04:53:59.826 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6000000000000001e-06 2024-02-05T04:53:59.826 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.826 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.826 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:59.827 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.171986+0000", 2024-02-05T04:53:59.827 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.4010000000000003e-05 2024-02-05T04:53:59.827 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.827 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.827 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting_for_map not empty", 2024-02-05T04:53:59.827 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.171990+0000", 2024-02-05T04:53:59.827 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2719999999999999e-06 2024-02-05T04:53:59.827 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.827 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.827 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:59.827 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.175603+0000", 2024-02-05T04:53:59.827 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0036129080000000002 2024-02-05T04:53:59.827 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.827 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.827 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:53:59.828 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.175679+0000", 2024-02-05T04:53:59.828 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6552999999999997e-05 2024-02-05T04:53:59.828 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.828 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.828 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:59.828 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.177747+0000", 2024-02-05T04:53:59.828 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0020672009999999998 2024-02-05T04:53:59.828 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.828 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:59.828 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.828 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.828 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.828 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:389 3.ds0 3:b4e8e82c:::smithi19639362-25:head [read 0~638976,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-02-05T04:53:59.828 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:34.171905+0000", 2024-02-05T04:53:59.828 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 25.639746095, 2024-02-05T04:53:59.829 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0072757830000000001, 2024-02-05T04:53:59.829 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:59.829 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-02-05T04:53:59.829 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:53:59.829 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:53:59.829 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:59.829 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 389 2024-02-05T04:53:59.829 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.829 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:59.829 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.829 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:59.829 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.171905+0000", 2024-02-05T04:53:59.829 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:59.829 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.829 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.830 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:59.830 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.171905+0000", 2024-02-05T04:53:59.830 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:59.830 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.830 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.830 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:59.830 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.171906+0000", 2024-02-05T04:53:59.830 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2499999999999995e-07 2024-02-05T04:53:59.830 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.830 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.830 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:59.830 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.171910+0000", 2024-02-05T04:53:59.830 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4490000000000001e-06 2024-02-05T04:53:59.830 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.830 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.830 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:59.831 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.171918+0000", 2024-02-05T04:53:59.831 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.2449999999999998e-06 2024-02-05T04:53:59.831 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.831 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.831 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:59.831 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.171924+0000", 2024-02-05T04:53:59.831 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6350000000000003e-06 2024-02-05T04:53:59.831 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.831 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.831 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:59.831 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.172022+0000", 2024-02-05T04:53:59.831 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8004000000000006e-05 2024-02-05T04:53:59.831 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.832 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.832 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting_for_map not empty", 2024-02-05T04:53:59.832 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.172026+0000", 2024-02-05T04:53:59.832 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8500000000000004e-06 2024-02-05T04:53:59.832 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.833 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.833 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:59.833 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.177805+0000", 2024-02-05T04:53:59.833 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0057789419999999996 2024-02-05T04:53:59.833 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.833 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.833 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:53:59.833 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.177900+0000", 2024-02-05T04:53:59.833 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5529e-05 2024-02-05T04:53:59.833 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.833 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.833 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:59.833 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.179181+0000", 2024-02-05T04:53:59.833 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0012806040000000001 2024-02-05T04:53:59.834 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.834 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:59.834 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.834 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.834 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.834 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.bs5 43/31 ECSubWrite(tid=137, reqid=osd.13.0:137, at_version=43'28, trim_to=0'0, roll_forward_to=41'26))", 2024-02-05T04:53:59.834 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:35.171816+0000", 2024-02-05T04:53:59.834 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 24.639835095999999, 2024-02-05T04:53:59.834 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0057279920000000003, 2024-02-05T04:53:59.834 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:59.834 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:59.834 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:59.835 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.835 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:59.835 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.171816+0000", 2024-02-05T04:53:59.835 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:59.835 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.835 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.835 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:59.835 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.171816+0000", 2024-02-05T04:53:59.835 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:59.835 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.835 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.835 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:59.835 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.171817+0000", 2024-02-05T04:53:59.835 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.89e-07 2024-02-05T04:53:59.835 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.836 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.836 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:59.836 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.171845+0000", 2024-02-05T04:53:59.836 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.7804e-05 2024-02-05T04:53:59.836 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.836 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.836 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:59.836 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.171855+0000", 2024-02-05T04:53:59.836 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0227999999999999e-05 2024-02-05T04:53:59.836 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.836 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.836 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:59.836 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.171865+0000", 2024-02-05T04:53:59.836 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.026e-05 2024-02-05T04:53:59.837 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.837 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.837 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:59.837 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.171958+0000", 2024-02-05T04:53:59.837 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2785e-05 2024-02-05T04:53:59.837 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.837 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.837 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:59.837 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.171969+0000", 2024-02-05T04:53:59.837 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0998e-05 2024-02-05T04:53:59.837 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.837 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.837 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:59.837 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.177502+0000", 2024-02-05T04:53:59.837 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0055336270000000002 2024-02-05T04:53:59.838 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.838 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.838 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:59.838 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.177544+0000", 2024-02-05T04:53:59.838 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1801000000000002e-05 2024-02-05T04:53:59.838 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.838 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:59.838 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.838 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.838 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.838 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.6s5 43/31 ECSubWrite(tid=138, reqid=osd.13.0:138, at_version=43'24, trim_to=0'0, roll_forward_to=40'22))", 2024-02-05T04:53:59.838 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:35.172477+0000", 2024-02-05T04:53:59.838 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 24.639174392000001, 2024-02-05T04:53:59.838 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0079411910000000002, 2024-02-05T04:53:59.838 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:59.839 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:59.839 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:59.839 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.839 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:59.839 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.172477+0000", 2024-02-05T04:53:59.839 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:59.839 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.839 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.839 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:59.839 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.172477+0000", 2024-02-05T04:53:59.839 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:59.839 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.839 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.839 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:59.840 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.172478+0000", 2024-02-05T04:53:59.840 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9299999999999997e-07 2024-02-05T04:53:59.840 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.840 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.840 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:59.840 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.172502+0000", 2024-02-05T04:53:59.840 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.4071e-05 2024-02-05T04:53:59.840 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.840 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.840 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:59.840 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.172525+0000", 2024-02-05T04:53:59.840 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3666e-05 2024-02-05T04:53:59.840 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.841 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.841 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:59.841 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.172534+0000", 2024-02-05T04:53:59.841 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.7749999999999998e-06 2024-02-05T04:53:59.841 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.841 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.841 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:59.841 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.172612+0000", 2024-02-05T04:53:59.841 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7486999999999999e-05 2024-02-05T04:53:59.841 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.841 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.841 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:59.841 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.172622+0000", 2024-02-05T04:53:59.841 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0040999999999999e-05 2024-02-05T04:53:59.842 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.842 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.842 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:59.842 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.180380+0000", 2024-02-05T04:53:59.842 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0077582739999999999 2024-02-05T04:53:59.842 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.842 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.842 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:59.842 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.180418+0000", 2024-02-05T04:53:59.842 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8183999999999999e-05 2024-02-05T04:53:59.842 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.842 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:59.842 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.842 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.843 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.843 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.3s1 43/31 ECSubWrite(tid=63, reqid=osd.4.0:63, at_version=43'27, trim_to=0'0, roll_forward_to=40'25))", 2024-02-05T04:53:59.843 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:35.173843+0000", 2024-02-05T04:53:59.843 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 24.637808449000001, 2024-02-05T04:53:59.843 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.006542473, 2024-02-05T04:53:59.843 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:59.843 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:59.843 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:59.843 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.843 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:59.843 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.173843+0000", 2024-02-05T04:53:59.843 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:59.843 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.843 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.844 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:59.844 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.173843+0000", 2024-02-05T04:53:59.844 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:59.844 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.844 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.844 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:59.844 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.173843+0000", 2024-02-05T04:53:59.844 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.18e-07 2024-02-05T04:53:59.844 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.844 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.844 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:59.844 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.173862+0000", 2024-02-05T04:53:59.844 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8770000000000002e-05 2024-02-05T04:53:59.844 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.845 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.845 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:59.845 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.173883+0000", 2024-02-05T04:53:59.845 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1072000000000001e-05 2024-02-05T04:53:59.845 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.846 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.846 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:59.846 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.173890+0000", 2024-02-05T04:53:59.846 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9569999999999996e-06 2024-02-05T04:53:59.846 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.846 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.846 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:59.846 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.173952+0000", 2024-02-05T04:53:59.846 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.2553999999999998e-05 2024-02-05T04:53:59.846 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.847 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.847 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:59.847 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.173960+0000", 2024-02-05T04:53:59.847 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7600000000000002e-06 2024-02-05T04:53:59.847 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.847 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.847 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:59.847 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.180350+0000", 2024-02-05T04:53:59.847 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0063900290000000002 2024-02-05T04:53:59.847 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.847 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.847 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:59.847 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.180385+0000", 2024-02-05T04:53:59.847 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5213000000000003e-05 2024-02-05T04:53:59.848 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.848 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:59.848 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.848 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.848 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.848 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:401 3.ds0 3:b9c88094:::smithi19639362-50:head [delete] snapc 7=[7,6,4] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e43)", 2024-02-05T04:53:59.848 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:35.182518+0000", 2024-02-05T04:53:59.848 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 24.629133251999999, 2024-02-05T04:53:59.848 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0087143140000000008, 2024-02-05T04:53:59.848 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:59.848 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:59.848 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:53:59.848 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:53:59.848 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:59.849 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 401 2024-02-05T04:53:59.849 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.849 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:59.849 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.849 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:59.849 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.182518+0000", 2024-02-05T04:53:59.849 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:59.849 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.849 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.849 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:59.849 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.182518+0000", 2024-02-05T04:53:59.849 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:59.849 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.849 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.849 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:59.849 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.182519+0000", 2024-02-05T04:53:59.850 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.313e-06 2024-02-05T04:53:59.850 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.850 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.850 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:59.850 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.182524+0000", 2024-02-05T04:53:59.850 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9150000000000004e-06 2024-02-05T04:53:59.850 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.850 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.850 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:59.850 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.182544+0000", 2024-02-05T04:53:59.850 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9327999999999999e-05 2024-02-05T04:53:59.850 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.850 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.850 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:59.850 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.182572+0000", 2024-02-05T04:53:59.850 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8180000000000001e-05 2024-02-05T04:53:59.851 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.851 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.851 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:59.851 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.182643+0000", 2024-02-05T04:53:59.851 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1598000000000001e-05 2024-02-05T04:53:59.851 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.851 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.851 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:53:59.851 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.182744+0000", 2024-02-05T04:53:59.851 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000100881 2024-02-05T04:53:59.851 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.851 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.851 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:59.851 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.183600+0000", 2024-02-05T04:53:59.851 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00085556200000000003 2024-02-05T04:53:59.852 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.852 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.852 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:59.852 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.188714+0000", 2024-02-05T04:53:59.852 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.005113828 2024-02-05T04:53:59.852 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.852 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.852 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T04:53:59.852 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.191180+0000", 2024-02-05T04:53:59.852 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0024662130000000001 2024-02-05T04:53:59.852 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.852 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.852 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:59.852 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.191232+0000", 2024-02-05T04:53:59.852 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2496e-05 2024-02-05T04:53:59.853 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.853 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:59.853 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.853 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.853 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.853 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.6s5 43/31 ECSubWrite(tid=141, reqid=osd.13.0:138, at_version=0'0, trim_to=0'0, roll_forward_to=43'24))", 2024-02-05T04:53:59.853 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:35.183142+0000", 2024-02-05T04:53:59.853 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 24.628509573999999, 2024-02-05T04:53:59.853 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0055687339999999997, 2024-02-05T04:53:59.853 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:59.853 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:59.853 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:59.853 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.853 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:59.853 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.183142+0000", 2024-02-05T04:53:59.854 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:59.854 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.854 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.854 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:59.854 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.183142+0000", 2024-02-05T04:53:59.854 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:59.854 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.854 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.854 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:59.854 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.183142+0000", 2024-02-05T04:53:59.854 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3799999999999998e-07 2024-02-05T04:53:59.854 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.854 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.854 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:59.854 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.183153+0000", 2024-02-05T04:53:59.855 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0924000000000001e-05 2024-02-05T04:53:59.855 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.855 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.855 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:59.855 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.183171+0000", 2024-02-05T04:53:59.855 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8199e-05 2024-02-05T04:53:59.855 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.855 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.855 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:59.855 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.183185+0000", 2024-02-05T04:53:59.855 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3594999999999999e-05 2024-02-05T04:53:59.855 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.855 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.855 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:59.856 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.183244+0000", 2024-02-05T04:53:59.856 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8714000000000001e-05 2024-02-05T04:53:59.856 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.856 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.856 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:59.856 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.183253+0000", 2024-02-05T04:53:59.856 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5340000000000005e-06 2024-02-05T04:53:59.856 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.856 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.856 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:59.856 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.188665+0000", 2024-02-05T04:53:59.856 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0054115350000000003 2024-02-05T04:53:59.856 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.856 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.856 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:59.857 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.188711+0000", 2024-02-05T04:53:59.857 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5794999999999998e-05 2024-02-05T04:53:59.857 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.857 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:59.857 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.857 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.857 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.857 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.5s4 52/31 ECSubWrite(tid=42, reqid=osd.15.0:42, at_version=52'32, trim_to=0'0, roll_forward_to=51'30))", 2024-02-05T04:53:59.857 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:43.966072+0000", 2024-02-05T04:53:59.857 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 15.845578975, 2024-02-05T04:53:59.857 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0059367960000000003, 2024-02-05T04:53:59.857 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:59.857 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:59.857 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:59.857 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.858 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:59.858 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:43.966072+0000", 2024-02-05T04:53:59.858 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:59.858 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.858 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.858 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:59.858 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:43.966072+0000", 2024-02-05T04:53:59.858 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:59.858 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.858 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.858 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:59.858 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:43.966073+0000", 2024-02-05T04:53:59.859 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7199999999999999e-07 2024-02-05T04:53:59.859 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.859 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.859 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:59.859 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:43.966102+0000", 2024-02-05T04:53:59.860 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.9249000000000001e-05 2024-02-05T04:53:59.860 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.860 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.860 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:59.860 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:43.966113+0000", 2024-02-05T04:53:59.860 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0925e-05 2024-02-05T04:53:59.860 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.860 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.860 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:59.860 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:43.966122+0000", 2024-02-05T04:53:59.860 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1649999999999995e-06 2024-02-05T04:53:59.860 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.860 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.860 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:59.861 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:43.966195+0000", 2024-02-05T04:53:59.861 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.3046999999999994e-05 2024-02-05T04:53:59.861 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.861 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.861 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:59.861 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:43.966205+0000", 2024-02-05T04:53:59.861 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0179e-05 2024-02-05T04:53:59.861 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.861 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.861 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:59.861 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:43.971929+0000", 2024-02-05T04:53:59.861 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0057237190000000004 2024-02-05T04:53:59.861 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.861 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.862 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:59.862 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:43.972009+0000", 2024-02-05T04:53:59.862 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0039999999999999e-05 2024-02-05T04:53:59.862 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.862 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:59.862 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.862 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.862 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.862 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:525 3.ds0 3:be3a9027:::smithi19639362-44:head [delete] snapc f=[f,e,d,b,8,7] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e57)", 2024-02-05T04:53:59.862 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:48.958273+0000", 2024-02-05T04:53:59.862 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 10.853378804, 2024-02-05T04:53:59.862 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0081252289999999994, 2024-02-05T04:53:59.862 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:59.862 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:59.862 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:53:59.863 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:53:59.863 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:59.863 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 525 2024-02-05T04:53:59.863 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.863 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:59.863 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.863 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:59.863 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.958273+0000", 2024-02-05T04:53:59.863 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:59.863 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.863 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.863 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:59.863 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.958273+0000", 2024-02-05T04:53:59.863 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:59.863 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.863 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.864 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:59.864 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.958274+0000", 2024-02-05T04:53:59.864 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8870000000000001e-06 2024-02-05T04:53:59.864 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.864 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.864 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:59.864 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.958282+0000", 2024-02-05T04:53:59.864 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2599999999999999e-06 2024-02-05T04:53:59.864 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.864 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.864 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:59.864 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.958296+0000", 2024-02-05T04:53:59.864 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4157e-05 2024-02-05T04:53:59.864 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.865 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.865 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:59.865 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.958308+0000", 2024-02-05T04:53:59.865 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2075999999999999e-05 2024-02-05T04:53:59.865 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.865 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.865 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:59.865 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.958677+0000", 2024-02-05T04:53:59.865 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00036947599999999998 2024-02-05T04:53:59.865 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.865 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.865 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:53:59.865 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.958775+0000", 2024-02-05T04:53:59.865 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8039000000000001e-05 2024-02-05T04:53:59.866 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.866 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.866 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:59.866 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.959380+0000", 2024-02-05T04:53:59.866 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00060500599999999995 2024-02-05T04:53:59.866 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.866 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.866 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:59.866 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.966178+0000", 2024-02-05T04:53:59.866 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0067973230000000001 2024-02-05T04:53:59.866 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.866 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.866 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T04:53:59.866 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.966328+0000", 2024-02-05T04:53:59.867 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000149918 2024-02-05T04:53:59.867 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.867 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.867 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:59.867 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.966398+0000", 2024-02-05T04:53:59.867 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0086999999999995e-05 2024-02-05T04:53:59.867 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.867 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:59.867 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.867 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.867 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.867 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.3s1 57/31 ECSubWrite(tid=77, reqid=osd.4.0:76, at_version=0'0, trim_to=0'0, roll_forward_to=57'35))", 2024-02-05T04:53:59.867 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:48.960701+0000", 2024-02-05T04:53:59.867 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 10.850950191000001, 2024-02-05T04:53:59.868 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0059086820000000002, 2024-02-05T04:53:59.868 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:59.868 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:59.868 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:59.868 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.868 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:59.868 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.960701+0000", 2024-02-05T04:53:59.868 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:59.868 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.868 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.868 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:59.868 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.960701+0000", 2024-02-05T04:53:59.868 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:59.868 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.869 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.869 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:59.869 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.960702+0000", 2024-02-05T04:53:59.869 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.3099999999999996e-07 2024-02-05T04:53:59.869 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.869 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.869 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:59.869 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.960718+0000", 2024-02-05T04:53:59.869 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6365999999999999e-05 2024-02-05T04:53:59.869 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.869 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.869 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:59.869 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.960771+0000", 2024-02-05T04:53:59.869 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2689000000000003e-05 2024-02-05T04:53:59.869 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.870 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.870 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:59.870 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.960785+0000", 2024-02-05T04:53:59.870 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4107999999999999e-05 2024-02-05T04:53:59.870 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.870 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.870 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:59.870 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.960894+0000", 2024-02-05T04:53:59.870 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000109184 2024-02-05T04:53:59.870 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.870 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.870 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:59.870 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.960904+0000", 2024-02-05T04:53:59.870 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.7480000000000005e-06 2024-02-05T04:53:59.871 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.871 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.871 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:59.871 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.966566+0000", 2024-02-05T04:53:59.871 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0056615700000000003 2024-02-05T04:53:59.871 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.871 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.871 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:59.871 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.966610+0000", 2024-02-05T04:53:59.871 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4085999999999998e-05 2024-02-05T04:53:59.871 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.871 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:59.871 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.871 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.871 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.872 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:526 3.ds0 3:b7ad4782:::smithi19639362-40:head [copy-from ver 29] snapc f=[f,e,d,b,8,7] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-02-05T04:53:59.872 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:48.966782+0000", 2024-02-05T04:53:59.872 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 10.844869323999999, 2024-02-05T04:53:59.872 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0165904020000001, 2024-02-05T04:53:59.872 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:59.872 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:53:59.872 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:53:59.872 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:53:59.872 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:59.872 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 526 2024-02-05T04:53:59.872 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.872 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:59.872 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.872 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:59.873 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.966782+0000", 2024-02-05T04:53:59.873 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:59.873 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.873 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.873 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:59.873 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.966782+0000", 2024-02-05T04:53:59.873 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:59.873 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.873 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.873 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:59.873 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.966784+0000", 2024-02-05T04:53:59.873 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5510000000000001e-06 2024-02-05T04:53:59.873 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.873 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.873 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:59.874 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.966790+0000", 2024-02-05T04:53:59.874 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7390000000000002e-06 2024-02-05T04:53:59.874 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.874 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.874 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:59.874 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.966805+0000", 2024-02-05T04:53:59.874 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4209999999999999e-05 2024-02-05T04:53:59.874 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.874 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.874 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:59.874 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.966813+0000", 2024-02-05T04:53:59.874 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0509999999999997e-06 2024-02-05T04:53:59.874 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.874 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.874 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:59.875 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.966917+0000", 2024-02-05T04:53:59.875 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010419799999999999 2024-02-05T04:53:59.875 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.875 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.875 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:53:59.875 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.967039+0000", 2024-02-05T04:53:59.875 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012211500000000001 2024-02-05T04:53:59.875 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.875 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.875 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:59.875 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.970041+0000", 2024-02-05T04:53:59.875 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0030016580000001 2024-02-05T04:53:59.875 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.875 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.876 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:59.876 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.978954+0000", 2024-02-05T04:53:59.876 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.008913397 2024-02-05T04:53:59.876 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.876 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.876 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T04:53:59.876 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.983312+0000", 2024-02-05T04:53:59.876 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0043576969999999998 2024-02-05T04:53:59.876 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.876 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.876 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:59.876 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.983372+0000", 2024-02-05T04:53:59.876 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0785999999999997e-05 2024-02-05T04:53:59.876 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.876 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:59.877 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.877 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.877 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.877 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:527 3.ds0 3:b7ad4782:::smithi19639362-40:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e57)", 2024-02-05T04:53:59.877 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:48.966876+0000", 2024-02-05T04:53:59.877 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 10.844775592, 2024-02-05T04:53:59.877 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.006645641, 2024-02-05T04:53:59.877 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:59.877 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-02-05T04:53:59.877 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:53:59.877 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:53:59.877 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:53:59.877 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 527 2024-02-05T04:53:59.877 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.877 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:59.878 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.878 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:59.878 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.966876+0000", 2024-02-05T04:53:59.878 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:59.878 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.878 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.878 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:59.878 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.966876+0000", 2024-02-05T04:53:59.878 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:59.878 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.878 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.878 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:59.878 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.966876+0000", 2024-02-05T04:53:59.878 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.68e-07 2024-02-05T04:53:59.878 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.878 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.879 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:59.879 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.966880+0000", 2024-02-05T04:53:59.879 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.1690000000000001e-06 2024-02-05T04:53:59.879 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.879 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.879 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:59.879 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.966889+0000", 2024-02-05T04:53:59.879 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.38e-06 2024-02-05T04:53:59.879 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.879 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.879 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:59.879 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.966896+0000", 2024-02-05T04:53:59.879 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2130000000000002e-06 2024-02-05T04:53:59.879 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.879 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.879 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:59.880 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.967519+0000", 2024-02-05T04:53:59.880 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00062232499999999996 2024-02-05T04:53:59.880 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.880 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.880 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for blocked object", 2024-02-05T04:53:59.880 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.967582+0000", 2024-02-05T04:53:59.880 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3142e-05 2024-02-05T04:53:59.880 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.880 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.880 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:59.880 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.973321+0000", 2024-02-05T04:53:59.880 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.005739114 2024-02-05T04:53:59.880 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.880 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.880 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:53:59.881 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.973429+0000", 2024-02-05T04:53:59.881 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000108207 2024-02-05T04:53:59.881 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.881 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.881 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:59.881 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.973521+0000", 2024-02-05T04:53:59.881 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2423000000000006e-05 2024-02-05T04:53:59.881 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.881 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:59.881 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.881 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.881 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.881 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.es2 58/57 ECSubWrite(tid=64, reqid=osd.2.0:64, at_version=58'50, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:53:59.881 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:49.953060+0000", 2024-02-05T04:53:59.881 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 9.8585912980000003, 2024-02-05T04:53:59.881 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.006033882, 2024-02-05T04:53:59.882 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:59.882 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:59.882 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:59.882 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.882 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:59.882 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.953060+0000", 2024-02-05T04:53:59.882 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:59.882 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.882 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.882 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:59.882 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.953060+0000", 2024-02-05T04:53:59.882 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:59.882 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.882 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.883 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:59.883 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.953061+0000", 2024-02-05T04:53:59.883 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4099999999999997e-07 2024-02-05T04:53:59.883 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.883 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.883 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:59.883 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.953084+0000", 2024-02-05T04:53:59.883 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3507999999999999e-05 2024-02-05T04:53:59.883 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.883 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.883 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:59.883 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.953105+0000", 2024-02-05T04:53:59.883 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0585999999999999e-05 2024-02-05T04:53:59.883 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.883 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.883 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:59.884 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.953115+0000", 2024-02-05T04:53:59.884 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0144e-05 2024-02-05T04:53:59.884 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.884 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.884 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:59.884 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.953179+0000", 2024-02-05T04:53:59.884 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3456999999999999e-05 2024-02-05T04:53:59.884 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.884 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.884 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:59.884 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.953190+0000", 2024-02-05T04:53:59.884 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1253000000000001e-05 2024-02-05T04:53:59.884 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.884 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.884 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:59.885 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.959048+0000", 2024-02-05T04:53:59.885 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0058580519999999999 2024-02-05T04:53:59.885 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.885 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.885 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:59.885 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.959094+0000", 2024-02-05T04:53:59.885 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6041000000000002e-05 2024-02-05T04:53:59.885 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.885 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:59.885 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.885 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.885 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.885 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.5s4 58/57 ECSubWrite(tid=57, reqid=osd.15.0:57, at_version=58'36, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:53:59.885 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:49.955263+0000", 2024-02-05T04:53:59.885 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 9.8563881430000002, 2024-02-05T04:53:59.886 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0065309849999999996, 2024-02-05T04:53:59.886 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:59.886 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:59.886 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:59.886 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.886 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:59.886 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.955263+0000", 2024-02-05T04:53:59.886 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:59.886 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.886 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.886 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:59.886 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.955263+0000", 2024-02-05T04:53:59.886 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:59.886 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.886 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.887 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:59.887 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.955264+0000", 2024-02-05T04:53:59.887 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6400000000000001e-07 2024-02-05T04:53:59.887 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.887 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.887 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:59.887 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.955285+0000", 2024-02-05T04:53:59.887 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0883999999999999e-05 2024-02-05T04:53:59.887 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.887 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.887 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:59.887 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.955300+0000", 2024-02-05T04:53:59.887 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4769999999999999e-05 2024-02-05T04:53:59.887 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.888 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.888 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:59.888 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.955310+0000", 2024-02-05T04:53:59.888 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0190000000000001e-05 2024-02-05T04:53:59.888 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.888 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.888 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:59.888 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.955383+0000", 2024-02-05T04:53:59.888 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.3349000000000005e-05 2024-02-05T04:53:59.888 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.888 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.888 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:59.888 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.955395+0000", 2024-02-05T04:53:59.888 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2298e-05 2024-02-05T04:53:59.888 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.889 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.889 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:59.889 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.961733+0000", 2024-02-05T04:53:59.889 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0063372020000000001 2024-02-05T04:53:59.889 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.889 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.889 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:59.889 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.961794+0000", 2024-02-05T04:53:59.889 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1527999999999998e-05 2024-02-05T04:53:59.889 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.889 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:59.889 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.889 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.889 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.889 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.5s4 58/57 ECSubWrite(tid=58, reqid=osd.15.0:58, at_version=58'38, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:53:59.890 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:49.956419+0000", 2024-02-05T04:53:59.890 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 9.855232204, 2024-02-05T04:53:59.890 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0080420990000000005, 2024-02-05T04:53:59.890 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:59.890 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:59.890 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:59.890 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.890 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:59.890 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.956419+0000", 2024-02-05T04:53:59.890 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:59.890 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.890 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.890 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:59.890 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.956419+0000", 2024-02-05T04:53:59.890 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:59.891 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.891 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.891 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:59.891 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.956420+0000", 2024-02-05T04:53:59.891 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6499999999999998e-07 2024-02-05T04:53:59.891 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.891 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.891 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:59.891 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.956443+0000", 2024-02-05T04:53:59.891 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2725999999999999e-05 2024-02-05T04:53:59.891 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.891 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.891 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:59.891 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.956457+0000", 2024-02-05T04:53:59.891 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.451e-05 2024-02-05T04:53:59.891 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.892 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.892 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:59.892 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.956466+0000", 2024-02-05T04:53:59.892 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1279999999999997e-06 2024-02-05T04:53:59.892 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.892 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.892 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:59.892 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.957077+0000", 2024-02-05T04:53:59.892 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00061054900000000005 2024-02-05T04:53:59.892 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.892 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.892 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:59.892 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.957092+0000", 2024-02-05T04:53:59.892 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5242000000000001e-05 2024-02-05T04:53:59.892 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.893 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.893 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:59.893 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.964399+0000", 2024-02-05T04:53:59.893 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0073072420000000003 2024-02-05T04:53:59.893 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.893 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.893 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:59.893 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.964461+0000", 2024-02-05T04:53:59.893 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1937e-05 2024-02-05T04:53:59.893 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.893 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:59.893 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.893 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.893 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.893 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.bs5 58/57 ECSubWrite(tid=252, reqid=osd.13.0:252, at_version=58'54, trim_to=0'0, roll_forward_to=58'52))", 2024-02-05T04:53:59.893 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:49.968805+0000", 2024-02-05T04:53:59.894 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 9.8428463209999997, 2024-02-05T04:53:59.894 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0074122190000000003, 2024-02-05T04:53:59.894 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:59.894 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:59.894 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:59.894 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.894 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:59.894 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.968805+0000", 2024-02-05T04:53:59.894 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:59.894 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.894 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.894 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:59.894 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.968805+0000", 2024-02-05T04:53:59.894 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:59.895 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.895 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.895 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:59.895 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.968806+0000", 2024-02-05T04:53:59.895 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0500000000000002e-07 2024-02-05T04:53:59.895 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.895 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.895 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:59.895 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.968831+0000", 2024-02-05T04:53:59.895 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5119000000000001e-05 2024-02-05T04:53:59.895 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.895 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.895 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:59.895 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.968843+0000", 2024-02-05T04:53:59.895 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2169e-05 2024-02-05T04:53:59.896 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.896 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.896 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:59.896 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.968852+0000", 2024-02-05T04:53:59.896 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.3729999999999999e-06 2024-02-05T04:53:59.896 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.896 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.896 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:59.896 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.968910+0000", 2024-02-05T04:53:59.896 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7883000000000001e-05 2024-02-05T04:53:59.896 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.896 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.896 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:59.896 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.968921+0000", 2024-02-05T04:53:59.897 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0473e-05 2024-02-05T04:53:59.897 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.897 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.897 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:59.897 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.976171+0000", 2024-02-05T04:53:59.897 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0072504680000000004 2024-02-05T04:53:59.897 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.897 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.897 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:59.897 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.976217+0000", 2024-02-05T04:53:59.897 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5929e-05 2024-02-05T04:53:59.897 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.897 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:59.897 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.898 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.898 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.898 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.5s4 58/57 ECSubWrite(tid=60, reqid=osd.15.0:60, at_version=58'40, trim_to=0'0, roll_forward_to=58'38))", 2024-02-05T04:53:59.898 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:49.969258+0000", 2024-02-05T04:53:59.898 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 9.8423929399999999, 2024-02-05T04:53:59.898 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.009848295, 2024-02-05T04:53:59.898 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:59.898 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:59.898 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:59.898 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.898 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:59.898 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.969258+0000", 2024-02-05T04:53:59.898 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:59.898 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.898 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.899 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:59.899 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.969258+0000", 2024-02-05T04:53:59.899 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:59.899 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.899 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.899 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:59.899 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.969259+0000", 2024-02-05T04:53:59.899 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9899999999999999e-07 2024-02-05T04:53:59.899 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.899 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.899 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:59.899 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.969282+0000", 2024-02-05T04:53:59.899 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2297999999999999e-05 2024-02-05T04:53:59.899 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.899 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.900 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:59.900 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.969295+0000", 2024-02-05T04:53:59.900 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3447e-05 2024-02-05T04:53:59.900 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.900 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.900 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:59.900 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.969303+0000", 2024-02-05T04:53:59.900 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4090000000000003e-06 2024-02-05T04:53:59.900 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.900 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.900 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:59.900 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.969393+0000", 2024-02-05T04:53:59.900 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.9324000000000001e-05 2024-02-05T04:53:59.900 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.901 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.901 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:59.901 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.969403+0000", 2024-02-05T04:53:59.901 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0244000000000001e-05 2024-02-05T04:53:59.901 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.901 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.901 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:59.901 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.979067+0000", 2024-02-05T04:53:59.901 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0096643059999999992 2024-02-05T04:53:59.901 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.901 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.901 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:59.901 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.979107+0000", 2024-02-05T04:53:59.901 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9468000000000002e-05 2024-02-05T04:53:59.901 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.901 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:59.902 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.902 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.902 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.902 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.bs5 63/57 ECSubWrite(tid=270, reqid=client.4634.0:581, at_version=63'60, trim_to=0'0, roll_forward_to=63'58))", 2024-02-05T04:53:59.902 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:55.016300+0000", 2024-02-05T04:53:59.902 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 4.7953515199999996, 2024-02-05T04:53:59.902 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.005560204, 2024-02-05T04:53:59.902 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:53:59.902 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:53:59.902 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:53:59.902 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.902 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:53:59.902 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.016300+0000", 2024-02-05T04:53:59.902 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:59.902 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.903 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.903 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:53:59.903 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.016300+0000", 2024-02-05T04:53:59.903 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:53:59.903 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.903 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.903 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:53:59.903 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.016300+0000", 2024-02-05T04:53:59.903 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1099999999999996e-07 2024-02-05T04:53:59.903 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.903 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.903 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:53:59.903 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.017300+0000", 2024-02-05T04:53:59.903 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0010001389999999999 2024-02-05T04:53:59.903 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.904 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.904 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:53:59.904 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.017316+0000", 2024-02-05T04:53:59.904 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5824999999999998e-05 2024-02-05T04:53:59.904 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.904 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.904 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:53:59.904 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.017326+0000", 2024-02-05T04:53:59.904 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2699999999999993e-06 2024-02-05T04:53:59.904 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.904 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.904 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:53:59.904 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.017386+0000", 2024-02-05T04:53:59.904 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0510999999999997e-05 2024-02-05T04:53:59.904 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.905 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.905 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:53:59.905 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.017394+0000", 2024-02-05T04:53:59.905 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3599999999999996e-06 2024-02-05T04:53:59.905 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.905 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.905 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:53:59.905 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.021824+0000", 2024-02-05T04:53:59.905 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0044293240000000001 2024-02-05T04:53:59.905 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:53:59.905 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:53:59.905 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:53:59.905 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.021860+0000", 2024-02-05T04:53:59.905 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6263999999999997e-05 2024-02-05T04:53:59.905 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.905 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:59.906 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.906 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:53:59.906 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:53:59.906 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T04:53:59.907 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-02-05T04:53:59.908 INFO:tasks.ceph.osd.7.smithi196.stderr:2024-02-05T04:53:59.906+0000 7fb281419640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34606) UID: 0 2024-02-05T04:53:59.943 INFO:teuthology.orchestra.run.smithi062.stderr:reweighted osd.2 to 1 (10000) 2024-02-05T04:53:59.943 INFO:teuthology.orchestra.run.smithi062.stderr:nodeep-scrub is set 2024-02-05T04:53:59.951 INFO:tasks.rados.rados.0.smithi196.stdout:247: done (0 left) 2024-02-05T04:53:59.951 INFO:tasks.rados.rados.0.smithi196.stdout:248: rollback oid 21 current snap is 21 2024-02-05T04:53:59.951 INFO:tasks.rados.rados.0.smithi196.stdout:rollback oid 21 to 15 2024-02-05T04:53:59.951 INFO:tasks.rados.rados.0.smithi196.stdout:249: snap_remove snap 17 2024-02-05T04:53:59.969 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-02-05T04:54:00.012 INFO:tasks.ceph.osd.14.smithi150.stderr:2024-02-05T04:54:00.007+0000 7f5f81083640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 34020) UID: 0 2024-02-05T04:54:00.063 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T04:54:00.063 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-02-05T04:54:00.063 INFO:teuthology.orchestra.run.smithi196.stdout: "num_ops": 0 2024-02-05T04:54:00.063 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T04:54:00.072 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-02-05T04:54:00.109 INFO:tasks.ceph.osd.10.smithi150.stderr:2024-02-05T04:54:00.108+0000 7f3075a4d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 34018) UID: 0 2024-02-05T04:54:00.178 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T04:54:00.178 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-02-05T04:54:00.178 INFO:teuthology.orchestra.run.smithi196.stdout: "complaint_time": 30, 2024-02-05T04:54:00.178 INFO:teuthology.orchestra.run.smithi196.stdout: "num_blocked_ops": 0 2024-02-05T04:54:00.179 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T04:54:00.187 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-02-05T04:54:00.210 INFO:tasks.ceph.osd.10.smithi150.stderr:2024-02-05T04:54:00.209+0000 7f3075a4d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 34018) UID: 0 2024-02-05T04:54:00.294 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T04:54:00.294 INFO:teuthology.orchestra.run.smithi196.stdout: "size": 20, 2024-02-05T04:54:00.294 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 600, 2024-02-05T04:54:00.294 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-02-05T04:54:00.294 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.294 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:222 3.cs0 3:30950ab7:::smithi19639362-4:3 [read 0~65536,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:54:00.294 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:29.134965+0000", 2024-02-05T04:54:00.295 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 31.157172591999998, 2024-02-05T04:54:00.295 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0084490469999999995, 2024-02-05T04:54:00.295 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:00.295 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-02-05T04:54:00.295 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:54:00.295 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:54:00.295 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:00.295 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 222 2024-02-05T04:54:00.295 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.295 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:00.295 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.295 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:00.295 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.134965+0000", 2024-02-05T04:54:00.295 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:00.296 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.296 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.296 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:00.296 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.134965+0000", 2024-02-05T04:54:00.296 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:00.296 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.296 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.296 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:00.296 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.134991+0000", 2024-02-05T04:54:00.296 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.6359999999999998e-05 2024-02-05T04:54:00.296 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.296 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.296 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:00.296 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135011+0000", 2024-02-05T04:54:00.296 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9874e-05 2024-02-05T04:54:00.297 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.297 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.297 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:00.297 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135044+0000", 2024-02-05T04:54:00.297 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2778999999999999e-05 2024-02-05T04:54:00.297 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.297 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.297 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:00.297 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135057+0000", 2024-02-05T04:54:00.297 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2582000000000001e-05 2024-02-05T04:54:00.297 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.297 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.297 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:00.297 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135163+0000", 2024-02-05T04:54:00.297 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010616900000000001 2024-02-05T04:54:00.297 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.298 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.298 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op must wait for map", 2024-02-05T04:54:00.298 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135171+0000", 2024-02-05T04:54:00.298 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.7709999999999995e-06 2024-02-05T04:54:00.298 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.298 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.298 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:00.298 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.139041+0000", 2024-02-05T04:54:00.298 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0038695769999999999 2024-02-05T04:54:00.298 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.298 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.298 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:54:00.298 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.139163+0000", 2024-02-05T04:54:00.298 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012187499999999999 2024-02-05T04:54:00.298 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.299 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.299 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:00.299 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.143414+0000", 2024-02-05T04:54:00.299 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0042510600000000001 2024-02-05T04:54:00.299 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.299 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:00.299 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.299 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.299 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.299 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:223 3.cs0 3:30950ab7:::smithi19639362-4:3 [read 0~65536,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:54:00.299 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:29.135093+0000", 2024-02-05T04:54:00.299 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 31.157044655, 2024-02-05T04:54:00.299 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0089544459999999996, 2024-02-05T04:54:00.299 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:00.299 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-02-05T04:54:00.299 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:54:00.300 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:54:00.300 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:00.300 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 223 2024-02-05T04:54:00.300 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.300 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:00.300 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.300 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:00.300 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135093+0000", 2024-02-05T04:54:00.300 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:00.300 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.300 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.300 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:00.300 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135093+0000", 2024-02-05T04:54:00.300 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:00.301 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.301 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.301 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:00.301 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135094+0000", 2024-02-05T04:54:00.301 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0100000000000004e-07 2024-02-05T04:54:00.301 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.301 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.301 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:00.301 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135100+0000", 2024-02-05T04:54:00.301 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9950000000000002e-06 2024-02-05T04:54:00.301 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.301 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.301 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:00.301 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135114+0000", 2024-02-05T04:54:00.301 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4831000000000001e-05 2024-02-05T04:54:00.301 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.302 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.302 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:00.302 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135123+0000", 2024-02-05T04:54:00.302 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0730000000000005e-06 2024-02-05T04:54:00.302 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.302 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.302 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:00.302 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135265+0000", 2024-02-05T04:54:00.302 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000142423 2024-02-05T04:54:00.302 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.302 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.302 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting_for_map not empty", 2024-02-05T04:54:00.302 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135271+0000", 2024-02-05T04:54:00.302 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.2689999999999996e-06 2024-02-05T04:54:00.302 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.303 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.303 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:00.303 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.140216+0000", 2024-02-05T04:54:00.303 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0049449699999999999 2024-02-05T04:54:00.303 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.303 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.303 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:54:00.303 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.140326+0000", 2024-02-05T04:54:00.303 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000109412 2024-02-05T04:54:00.303 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.303 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.303 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:00.303 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.144047+0000", 2024-02-05T04:54:00.303 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0037216720000000001 2024-02-05T04:54:00.303 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.303 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:00.304 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.304 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.304 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.304 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:224 3.cs0 3:30950ab7:::smithi19639362-4:3 [read 0~65536,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:54:00.304 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:29.135196+0000", 2024-02-05T04:54:00.304 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 31.156941418999999, 2024-02-05T04:54:00.304 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0091200659999999996, 2024-02-05T04:54:00.304 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:00.304 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-02-05T04:54:00.304 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:54:00.304 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:54:00.304 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:00.304 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 224 2024-02-05T04:54:00.304 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.304 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:00.305 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.305 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:00.305 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135196+0000", 2024-02-05T04:54:00.305 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:00.305 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.305 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.305 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:00.305 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135196+0000", 2024-02-05T04:54:00.305 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:00.305 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.305 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.305 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:00.305 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135197+0000", 2024-02-05T04:54:00.305 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1790000000000001e-06 2024-02-05T04:54:00.305 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.305 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.306 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:00.306 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135204+0000", 2024-02-05T04:54:00.306 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5309999999999998e-06 2024-02-05T04:54:00.306 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.306 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.306 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:00.306 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135216+0000", 2024-02-05T04:54:00.306 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2573e-05 2024-02-05T04:54:00.306 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.306 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.306 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:00.306 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135227+0000", 2024-02-05T04:54:00.306 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1029000000000001e-05 2024-02-05T04:54:00.306 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.306 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.306 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:00.307 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135925+0000", 2024-02-05T04:54:00.307 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00069716299999999999 2024-02-05T04:54:00.307 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.307 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.307 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting_for_map not empty", 2024-02-05T04:54:00.307 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135930+0000", 2024-02-05T04:54:00.307 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5799999999999999e-06 2024-02-05T04:54:00.307 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.307 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.307 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:00.307 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.142325+0000", 2024-02-05T04:54:00.307 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0063945950000000003 2024-02-05T04:54:00.307 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.307 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.308 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:54:00.308 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.142420+0000", 2024-02-05T04:54:00.308 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5747000000000003e-05 2024-02-05T04:54:00.308 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.308 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.308 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:00.308 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.144316+0000", 2024-02-05T04:54:00.308 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001895669 2024-02-05T04:54:00.308 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.308 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:00.308 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.308 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.308 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.308 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:267 3.cs0 3:30950ab7:::smithi19639362-4:head [append 0~753664] snapc 3=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-02-05T04:54:00.308 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:30.179941+0000", 2024-02-05T04:54:00.309 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 30.112196822000001, 2024-02-05T04:54:00.309 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0094095010000000007, 2024-02-05T04:54:00.309 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:00.309 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:00.309 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:54:00.309 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:54:00.309 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:00.309 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 267 2024-02-05T04:54:00.309 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.309 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:00.309 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.309 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:00.309 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.179941+0000", 2024-02-05T04:54:00.309 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:00.309 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.310 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.310 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:00.310 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.179941+0000", 2024-02-05T04:54:00.310 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:00.310 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.310 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.310 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:00.310 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.179943+0000", 2024-02-05T04:54:00.310 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3939999999999999e-06 2024-02-05T04:54:00.310 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.310 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.310 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:00.310 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.180152+0000", 2024-02-05T04:54:00.310 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000208791 2024-02-05T04:54:00.311 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.311 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.311 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:00.311 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.180171+0000", 2024-02-05T04:54:00.311 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9001e-05 2024-02-05T04:54:00.311 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.311 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.311 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:00.311 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.180178+0000", 2024-02-05T04:54:00.311 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4309999999999997e-06 2024-02-05T04:54:00.311 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.311 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.311 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:00.311 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.180243+0000", 2024-02-05T04:54:00.311 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4887000000000004e-05 2024-02-05T04:54:00.311 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.312 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.312 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:54:00.312 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.180307+0000", 2024-02-05T04:54:00.312 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3909000000000001e-05 2024-02-05T04:54:00.312 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.312 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.312 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:00.312 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.181056+0000", 2024-02-05T04:54:00.312 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00074879799999999995 2024-02-05T04:54:00.312 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.312 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.312 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:00.312 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.182113+0000", 2024-02-05T04:54:00.312 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001057555 2024-02-05T04:54:00.312 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.313 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.313 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T04:54:00.313 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.189310+0000", 2024-02-05T04:54:00.313 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0071969490000000002 2024-02-05T04:54:00.313 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.313 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.313 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:00.313 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.189350+0000", 2024-02-05T04:54:00.313 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9786000000000002e-05 2024-02-05T04:54:00.313 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.313 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:00.313 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.313 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.314 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.314 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:291 3.fs0 3:f8d01528:::smithi19639362-17:head [append 0~573440] snapc 3=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-02-05T04:54:00.314 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:30.384804+0000", 2024-02-05T04:54:00.314 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 29.907333374, 2024-02-05T04:54:00.314 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.009396738, 2024-02-05T04:54:00.314 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:00.314 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:00.314 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:54:00.314 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:54:00.314 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:00.314 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 291 2024-02-05T04:54:00.314 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.314 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:00.315 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.315 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:00.315 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.384804+0000", 2024-02-05T04:54:00.315 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:00.315 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.315 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.315 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:00.315 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.384804+0000", 2024-02-05T04:54:00.315 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:00.315 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.315 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.315 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:00.315 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.384805+0000", 2024-02-05T04:54:00.315 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5400000000000002e-07 2024-02-05T04:54:00.315 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.316 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.316 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:00.316 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.385086+0000", 2024-02-05T04:54:00.316 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00028146300000000003 2024-02-05T04:54:00.316 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.316 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.316 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:00.316 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.385095+0000", 2024-02-05T04:54:00.316 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.6449999999999994e-06 2024-02-05T04:54:00.316 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.316 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.316 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:00.316 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.385101+0000", 2024-02-05T04:54:00.316 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7209999999999997e-06 2024-02-05T04:54:00.317 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.317 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.317 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:00.317 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.388360+0000", 2024-02-05T04:54:00.317 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0032590010000000001 2024-02-05T04:54:00.317 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.317 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.317 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:54:00.317 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.388427+0000", 2024-02-05T04:54:00.317 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6960999999999998e-05 2024-02-05T04:54:00.317 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.317 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.317 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:00.317 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.388961+0000", 2024-02-05T04:54:00.317 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00053408100000000005 2024-02-05T04:54:00.318 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.318 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.318 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:00.318 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.390213+0000", 2024-02-05T04:54:00.318 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001251807 2024-02-05T04:54:00.318 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.318 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.318 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T04:54:00.318 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.394146+0000", 2024-02-05T04:54:00.318 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.003933219 2024-02-05T04:54:00.318 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.318 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.318 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:00.318 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.394201+0000", 2024-02-05T04:54:00.318 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4486e-05 2024-02-05T04:54:00.318 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.319 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:00.319 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.319 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.319 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.319 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:292 3.fs0 3:f8d01528:::smithi19639362-17:head [append 0~196608] snapc 3=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-02-05T04:54:00.319 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:30.385112+0000", 2024-02-05T04:54:00.319 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 29.907025150999999, 2024-02-05T04:54:00.319 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0092752849999999994, 2024-02-05T04:54:00.319 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:00.319 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:00.319 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:54:00.319 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:54:00.319 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:00.319 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 292 2024-02-05T04:54:00.319 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.319 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:00.320 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.320 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:00.320 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.385112+0000", 2024-02-05T04:54:00.320 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:00.320 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.320 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.320 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:00.320 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.385112+0000", 2024-02-05T04:54:00.320 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:00.320 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.320 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.320 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:00.320 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.385113+0000", 2024-02-05T04:54:00.320 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7199999999999999e-07 2024-02-05T04:54:00.320 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.321 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.321 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:00.321 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.385165+0000", 2024-02-05T04:54:00.321 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1607000000000002e-05 2024-02-05T04:54:00.321 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.321 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.321 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:00.321 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.385172+0000", 2024-02-05T04:54:00.321 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5939999999999996e-06 2024-02-05T04:54:00.321 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.321 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.321 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:00.321 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.385177+0000", 2024-02-05T04:54:00.321 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1760000000000001e-06 2024-02-05T04:54:00.321 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.322 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.322 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:00.322 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.389345+0000", 2024-02-05T04:54:00.322 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0041681640000000002 2024-02-05T04:54:00.322 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.322 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.322 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:54:00.322 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.389413+0000", 2024-02-05T04:54:00.322 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7166e-05 2024-02-05T04:54:00.322 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.322 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.322 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:00.322 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.389814+0000", 2024-02-05T04:54:00.322 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00040106400000000002 2024-02-05T04:54:00.322 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.322 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.323 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:00.323 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.390855+0000", 2024-02-05T04:54:00.323 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001041518 2024-02-05T04:54:00.323 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.323 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.323 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T04:54:00.323 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.394348+0000", 2024-02-05T04:54:00.323 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0034927159999999999 2024-02-05T04:54:00.323 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.323 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.323 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:00.323 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.394388+0000", 2024-02-05T04:54:00.323 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9708e-05 2024-02-05T04:54:00.323 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.323 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:00.324 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.324 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.324 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.324 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:293 3.fs0 3:f8d01528:::smithi19639362-17:head [setxattr _header (56) in=7b] snapc 3=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-02-05T04:54:00.324 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:30.385185+0000", 2024-02-05T04:54:00.324 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 29.906952112999999, 2024-02-05T04:54:00.324 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.010325663000000001, 2024-02-05T04:54:00.324 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:00.324 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:00.324 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:54:00.324 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:54:00.324 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:00.324 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 293 2024-02-05T04:54:00.324 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.324 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:00.325 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.325 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:00.325 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.385185+0000", 2024-02-05T04:54:00.325 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:00.325 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.325 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.325 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:00.325 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.385185+0000", 2024-02-05T04:54:00.325 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:00.325 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.325 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.325 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:00.325 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.385186+0000", 2024-02-05T04:54:00.325 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9400000000000005e-07 2024-02-05T04:54:00.325 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.326 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.326 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:00.326 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.385189+0000", 2024-02-05T04:54:00.326 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.6809999999999998e-06 2024-02-05T04:54:00.326 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.326 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.326 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:00.326 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.385195+0000", 2024-02-05T04:54:00.326 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0630000000000002e-06 2024-02-05T04:54:00.326 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.326 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.326 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:00.326 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.385200+0000", 2024-02-05T04:54:00.326 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0560000000000004e-06 2024-02-05T04:54:00.326 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.327 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.327 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:00.327 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.390293+0000", 2024-02-05T04:54:00.327 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0050930180000000004 2024-02-05T04:54:00.327 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.327 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.327 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:54:00.327 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.390337+0000", 2024-02-05T04:54:00.327 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3721999999999999e-05 2024-02-05T04:54:00.327 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.327 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.327 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:00.327 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.390566+0000", 2024-02-05T04:54:00.327 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00022925199999999999 2024-02-05T04:54:00.327 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.328 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.328 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:00.328 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.395220+0000", 2024-02-05T04:54:00.328 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0046545919999999999 2024-02-05T04:54:00.328 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.328 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.328 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T04:54:00.328 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.395254+0000", 2024-02-05T04:54:00.328 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3522000000000002e-05 2024-02-05T04:54:00.328 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.328 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.328 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:00.328 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.395511+0000", 2024-02-05T04:54:00.328 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00025716299999999998 2024-02-05T04:54:00.328 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.328 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:00.329 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.329 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.329 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.329 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:294 3.fs0 3:f8d01528:::smithi19639362-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-02-05T04:54:00.329 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:30.385209+0000", 2024-02-05T04:54:00.329 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 29.906928511, 2024-02-05T04:54:00.329 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.011163398999999999, 2024-02-05T04:54:00.329 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:00.329 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-02-05T04:54:00.329 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:54:00.329 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:54:00.329 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:00.329 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 294 2024-02-05T04:54:00.329 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.329 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:00.330 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.330 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:00.330 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.385209+0000", 2024-02-05T04:54:00.330 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:00.330 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.330 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.330 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:00.330 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.385209+0000", 2024-02-05T04:54:00.330 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:00.330 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.330 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.330 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:00.330 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.385209+0000", 2024-02-05T04:54:00.330 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.32e-07 2024-02-05T04:54:00.330 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.330 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.331 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:00.331 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.385212+0000", 2024-02-05T04:54:00.331 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1900000000000002e-06 2024-02-05T04:54:00.331 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.331 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.331 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:00.331 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.385218+0000", 2024-02-05T04:54:00.331 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4169999999999997e-06 2024-02-05T04:54:00.331 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.331 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.331 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:00.331 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.385229+0000", 2024-02-05T04:54:00.331 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0616e-05 2024-02-05T04:54:00.331 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.331 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.331 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:00.332 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.390742+0000", 2024-02-05T04:54:00.332 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0055133980000000001 2024-02-05T04:54:00.332 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.332 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.332 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for rw locks", 2024-02-05T04:54:00.332 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.390788+0000", 2024-02-05T04:54:00.332 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5627000000000002e-05 2024-02-05T04:54:00.332 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.332 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.332 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:00.332 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.395549+0000", 2024-02-05T04:54:00.332 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0047614520000000002 2024-02-05T04:54:00.332 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.332 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.332 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:54:00.333 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.395622+0000", 2024-02-05T04:54:00.333 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2916000000000005e-05 2024-02-05T04:54:00.333 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.333 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.333 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:00.333 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.396372+0000", 2024-02-05T04:54:00.333 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00075035100000000001 2024-02-05T04:54:00.333 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.333 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:00.333 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.333 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.333 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.333 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.es3 52/31 ECSubWrite(tid=55, reqid=osd.2.0:54, at_version=0'0, trim_to=0'0, roll_forward_to=52'48))", 2024-02-05T04:54:00.333 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:43.956685+0000", 2024-02-05T04:54:00.333 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 16.33545213, 2024-02-05T04:54:00.333 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0075937330000000001, 2024-02-05T04:54:00.334 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:00.334 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:00.334 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:00.334 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.334 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:00.334 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:43.956685+0000", 2024-02-05T04:54:00.334 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:00.334 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.334 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.334 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:00.334 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:43.956685+0000", 2024-02-05T04:54:00.334 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:00.334 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.334 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.335 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:00.335 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:43.956686+0000", 2024-02-05T04:54:00.335 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6800000000000001e-07 2024-02-05T04:54:00.335 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.335 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.335 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:00.335 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:43.956696+0000", 2024-02-05T04:54:00.335 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0594000000000001e-05 2024-02-05T04:54:00.335 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.335 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.335 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:00.335 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:43.956706+0000", 2024-02-05T04:54:00.335 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6460000000000005e-06 2024-02-05T04:54:00.335 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.335 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.335 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:00.336 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:43.956714+0000", 2024-02-05T04:54:00.336 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6410000000000002e-06 2024-02-05T04:54:00.336 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.336 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.336 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:00.336 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:43.956752+0000", 2024-02-05T04:54:00.336 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.782e-05 2024-02-05T04:54:00.336 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.336 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.336 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:00.336 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:43.956756+0000", 2024-02-05T04:54:00.336 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7199999999999997e-06 2024-02-05T04:54:00.336 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.336 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.336 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:00.337 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:43.964235+0000", 2024-02-05T04:54:00.337 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0074785019999999997 2024-02-05T04:54:00.337 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.337 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.337 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:00.337 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:43.964279+0000", 2024-02-05T04:54:00.337 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4342000000000003e-05 2024-02-05T04:54:00.337 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.337 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:00.337 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.337 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.337 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.337 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.1s1 52/31 ECSubWrite(tid=203, reqid=osd.13.0:203, at_version=52'45, trim_to=0'0, roll_forward_to=51'43))", 2024-02-05T04:54:00.337 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:43.961552+0000", 2024-02-05T04:54:00.337 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 16.330585417999998, 2024-02-05T04:54:00.337 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0078738820000000005, 2024-02-05T04:54:00.338 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:00.338 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:00.338 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:00.338 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.338 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:00.338 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:43.961552+0000", 2024-02-05T04:54:00.338 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:00.338 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.338 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.338 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:00.338 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:43.961552+0000", 2024-02-05T04:54:00.338 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:00.338 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.338 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.338 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:00.339 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:43.961553+0000", 2024-02-05T04:54:00.339 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.47e-07 2024-02-05T04:54:00.339 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.339 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.339 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:00.339 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:43.961576+0000", 2024-02-05T04:54:00.339 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3085e-05 2024-02-05T04:54:00.339 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.339 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.339 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:00.339 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:43.961587+0000", 2024-02-05T04:54:00.339 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1317e-05 2024-02-05T04:54:00.339 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.339 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.339 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:00.339 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:43.961600+0000", 2024-02-05T04:54:00.340 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2325e-05 2024-02-05T04:54:00.340 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.340 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.340 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:00.340 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:43.961672+0000", 2024-02-05T04:54:00.340 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2171999999999999e-05 2024-02-05T04:54:00.340 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.340 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.340 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:00.340 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:43.961681+0000", 2024-02-05T04:54:00.340 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1540000000000008e-06 2024-02-05T04:54:00.340 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.340 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.340 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:00.340 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:43.969355+0000", 2024-02-05T04:54:00.341 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0076740130000000004 2024-02-05T04:54:00.341 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.341 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.341 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:00.341 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:43.969426+0000", 2024-02-05T04:54:00.341 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0869000000000002e-05 2024-02-05T04:54:00.341 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.341 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:00.341 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.341 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.341 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.341 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.7s6 52/31 ECSubWrite(tid=204, reqid=osd.13.0:201, at_version=0'0, trim_to=0'0, roll_forward_to=52'21))", 2024-02-05T04:54:00.341 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:43.962273+0000", 2024-02-05T04:54:00.341 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 16.329864956000002, 2024-02-05T04:54:00.341 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0072166289999999996, 2024-02-05T04:54:00.341 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:00.342 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:00.342 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:00.342 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.342 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:00.342 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:43.962273+0000", 2024-02-05T04:54:00.342 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:00.342 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.342 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.342 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:00.342 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:43.962273+0000", 2024-02-05T04:54:00.342 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:00.342 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.342 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.342 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:00.342 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:43.962273+0000", 2024-02-05T04:54:00.342 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5799999999999999e-07 2024-02-05T04:54:00.343 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.343 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.343 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:00.343 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:43.962286+0000", 2024-02-05T04:54:00.343 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.27e-05 2024-02-05T04:54:00.343 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.343 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.343 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:00.343 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:43.962298+0000", 2024-02-05T04:54:00.343 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2323999999999999e-05 2024-02-05T04:54:00.343 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.343 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.343 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:00.343 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:43.962311+0000", 2024-02-05T04:54:00.343 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3193999999999999e-05 2024-02-05T04:54:00.344 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.344 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.344 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:00.344 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:43.962361+0000", 2024-02-05T04:54:00.344 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9530000000000002e-05 2024-02-05T04:54:00.344 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.344 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.344 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:00.344 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:43.962379+0000", 2024-02-05T04:54:00.344 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7972e-05 2024-02-05T04:54:00.344 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.344 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.344 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:00.344 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:43.969463+0000", 2024-02-05T04:54:00.344 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0070844699999999998 2024-02-05T04:54:00.345 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.345 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.345 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:00.345 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:43.969489+0000", 2024-02-05T04:54:00.345 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5780999999999999e-05 2024-02-05T04:54:00.345 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.345 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:00.345 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.345 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.345 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.345 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:507 3.fs0 3:fdb87099:::smithi19639362-41:head [stat,delete] snapc e=[e,d,b,9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2024-02-05T04:54:00.345 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:45.956619+0000", 2024-02-05T04:54:00.345 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 14.335518495000001, 2024-02-05T04:54:00.345 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.010281007999999999, 2024-02-05T04:54:00.345 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:00.346 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:00.346 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:54:00.346 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:54:00.346 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:00.346 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 507 2024-02-05T04:54:00.346 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.346 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:00.346 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.346 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:00.346 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:45.956619+0000", 2024-02-05T04:54:00.346 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:00.346 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.346 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.346 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:00.346 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:45.956619+0000", 2024-02-05T04:54:00.346 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:00.347 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.347 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.347 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:00.347 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:45.956621+0000", 2024-02-05T04:54:00.347 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.685e-06 2024-02-05T04:54:00.347 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.347 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.347 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:00.347 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:45.956627+0000", 2024-02-05T04:54:00.347 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.691e-06 2024-02-05T04:54:00.347 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.347 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.347 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:00.347 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:45.956640+0000", 2024-02-05T04:54:00.347 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2843999999999999e-05 2024-02-05T04:54:00.347 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.348 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.348 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:00.348 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:45.956650+0000", 2024-02-05T04:54:00.348 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.9340000000000001e-06 2024-02-05T04:54:00.348 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.348 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.348 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:00.348 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:45.956719+0000", 2024-02-05T04:54:00.348 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8733999999999996e-05 2024-02-05T04:54:00.348 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.348 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.348 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op must wait for map", 2024-02-05T04:54:00.348 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:45.956727+0000", 2024-02-05T04:54:00.348 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8150000000000006e-06 2024-02-05T04:54:00.348 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.349 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.349 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:00.349 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:45.958183+0000", 2024-02-05T04:54:00.349 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00145615 2024-02-05T04:54:00.349 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.349 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.349 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:54:00.349 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:45.958287+0000", 2024-02-05T04:54:00.349 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000103897 2024-02-05T04:54:00.349 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.349 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.349 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:00.349 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:45.961364+0000", 2024-02-05T04:54:00.349 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0030776459999999999 2024-02-05T04:54:00.349 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.350 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.350 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:00.350 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:45.962769+0000", 2024-02-05T04:54:00.350 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001404948 2024-02-05T04:54:00.350 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.350 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.350 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T04:54:00.350 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:45.966839+0000", 2024-02-05T04:54:00.350 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0040691579999999998 2024-02-05T04:54:00.350 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.350 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.350 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:00.350 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:45.966900+0000", 2024-02-05T04:54:00.350 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1506000000000004e-05 2024-02-05T04:54:00.350 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.350 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:00.351 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.351 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.351 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.351 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:531 3.cs0 3:30950ab7:::smithi19639362-4:head [append 0~606208] snapc f=[f,e,d,b,8,7] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-02-05T04:54:00.351 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:48.973323+0000", 2024-02-05T04:54:00.351 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 11.318814315000001, 2024-02-05T04:54:00.351 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.011870864, 2024-02-05T04:54:00.351 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:00.351 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:00.351 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:54:00.351 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:54:00.351 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:00.351 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 531 2024-02-05T04:54:00.351 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.351 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:00.351 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.352 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:00.352 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.973323+0000", 2024-02-05T04:54:00.352 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:00.352 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.352 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.352 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:00.352 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.973323+0000", 2024-02-05T04:54:00.352 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:00.352 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.352 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.352 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:00.352 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.973325+0000", 2024-02-05T04:54:00.352 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.092e-06 2024-02-05T04:54:00.352 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.352 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.353 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:00.353 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.973678+0000", 2024-02-05T04:54:00.353 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00035291599999999999 2024-02-05T04:54:00.353 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.353 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.353 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:00.353 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.973691+0000", 2024-02-05T04:54:00.353 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3217000000000001e-05 2024-02-05T04:54:00.353 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.353 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.353 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:00.353 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.973697+0000", 2024-02-05T04:54:00.353 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6729999999999997e-06 2024-02-05T04:54:00.353 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.353 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.353 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:00.354 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.973747+0000", 2024-02-05T04:54:00.354 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9583000000000003e-05 2024-02-05T04:54:00.354 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.354 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.354 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:54:00.354 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.973824+0000", 2024-02-05T04:54:00.354 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7082000000000005e-05 2024-02-05T04:54:00.354 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.354 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.354 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:00.354 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.974743+0000", 2024-02-05T04:54:00.354 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00091901600000000004 2024-02-05T04:54:00.354 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.354 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.354 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:00.354 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.976151+0000", 2024-02-05T04:54:00.355 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001408124 2024-02-05T04:54:00.355 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.355 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.355 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T04:54:00.355 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.985102+0000", 2024-02-05T04:54:00.355 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0089515270000000008 2024-02-05T04:54:00.355 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.355 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.355 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:00.355 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.985194+0000", 2024-02-05T04:54:00.355 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1633999999999997e-05 2024-02-05T04:54:00.355 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.355 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:00.355 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.355 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.355 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.356 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:532 3.cs0 3:30950ab7:::smithi19639362-4:head [append 0~688128] snapc f=[f,e,d,b,8,7] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-02-05T04:54:00.356 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:48.977248+0000", 2024-02-05T04:54:00.356 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 11.314889317, 2024-02-05T04:54:00.356 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0088660609999999997, 2024-02-05T04:54:00.356 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:00.356 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:00.356 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:54:00.356 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:54:00.356 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:00.356 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 532 2024-02-05T04:54:00.356 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.356 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:00.356 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.356 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:00.356 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.977248+0000", 2024-02-05T04:54:00.356 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:00.357 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.357 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.357 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:00.357 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.977248+0000", 2024-02-05T04:54:00.357 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:00.357 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.357 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.357 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:00.357 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.977251+0000", 2024-02-05T04:54:00.357 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5100000000000001e-06 2024-02-05T04:54:00.357 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.357 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.357 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:00.357 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.977641+0000", 2024-02-05T04:54:00.357 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000389859 2024-02-05T04:54:00.358 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.358 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.358 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:00.358 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.977652+0000", 2024-02-05T04:54:00.358 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0985000000000001e-05 2024-02-05T04:54:00.358 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.358 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.358 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:00.358 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.977657+0000", 2024-02-05T04:54:00.358 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.789e-06 2024-02-05T04:54:00.358 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.358 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.358 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:00.358 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.977697+0000", 2024-02-05T04:54:00.358 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9237999999999999e-05 2024-02-05T04:54:00.358 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.359 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.359 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:54:00.359 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.977765+0000", 2024-02-05T04:54:00.359 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8887999999999995e-05 2024-02-05T04:54:00.359 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.359 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.359 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:00.359 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.978429+0000", 2024-02-05T04:54:00.359 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00066370800000000005 2024-02-05T04:54:00.359 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.359 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.359 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:00.359 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.979498+0000", 2024-02-05T04:54:00.359 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0010690459999999999 2024-02-05T04:54:00.359 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.360 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.360 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T04:54:00.360 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.986021+0000", 2024-02-05T04:54:00.360 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0065229529999999997 2024-02-05T04:54:00.360 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.360 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.360 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:00.360 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.986114+0000", 2024-02-05T04:54:00.360 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.3084999999999994e-05 2024-02-05T04:54:00.360 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.360 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:00.360 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.360 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.360 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.360 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:535 3.cs0 3:30950ab7:::smithi19639362-4:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e57)", 2024-02-05T04:54:00.360 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:48.980316+0000", 2024-02-05T04:54:00.361 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 11.311821599, 2024-02-05T04:54:00.361 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0077787969999999996, 2024-02-05T04:54:00.361 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:00.361 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-02-05T04:54:00.361 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:54:00.361 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:54:00.361 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:00.361 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 535 2024-02-05T04:54:00.361 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.361 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:00.361 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.361 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:00.361 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.980316+0000", 2024-02-05T04:54:00.361 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:00.361 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.362 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.362 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:00.362 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.980316+0000", 2024-02-05T04:54:00.362 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:00.362 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.362 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.362 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:00.362 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.980316+0000", 2024-02-05T04:54:00.362 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8900000000000001e-07 2024-02-05T04:54:00.362 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.362 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.362 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:00.362 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.980318+0000", 2024-02-05T04:54:00.362 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.827e-06 2024-02-05T04:54:00.362 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.362 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.363 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:00.363 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.980323+0000", 2024-02-05T04:54:00.363 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.989e-06 2024-02-05T04:54:00.363 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.363 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.363 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:00.363 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.980326+0000", 2024-02-05T04:54:00.363 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.455e-06 2024-02-05T04:54:00.363 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.363 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.363 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:00.363 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.982343+0000", 2024-02-05T04:54:00.363 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0020166889999999999 2024-02-05T04:54:00.363 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.363 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.363 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for rw locks", 2024-02-05T04:54:00.364 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.982442+0000", 2024-02-05T04:54:00.364 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8972e-05 2024-02-05T04:54:00.364 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.364 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.364 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:00.364 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.987025+0000", 2024-02-05T04:54:00.364 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0045831530000000004 2024-02-05T04:54:00.364 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.364 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.364 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:54:00.364 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.987120+0000", 2024-02-05T04:54:00.364 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.4480000000000003e-05 2024-02-05T04:54:00.364 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.364 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.364 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:00.365 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.988095+0000", 2024-02-05T04:54:00.365 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00097494300000000003 2024-02-05T04:54:00.365 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.365 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:00.365 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.365 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.365 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.365 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.es3 58/57 ECSubWrite(tid=65, reqid=osd.2.0:64, at_version=0'0, trim_to=0'0, roll_forward_to=58'50))", 2024-02-05T04:54:00.365 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:49.959825+0000", 2024-02-05T04:54:00.365 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 10.332312655000001, 2024-02-05T04:54:00.365 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0074697970000000002, 2024-02-05T04:54:00.365 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:00.365 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:00.365 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:00.365 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.365 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:00.366 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.959825+0000", 2024-02-05T04:54:00.366 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:00.366 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.366 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.366 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:00.366 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.959825+0000", 2024-02-05T04:54:00.366 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:00.366 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.366 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.366 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:00.366 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.959826+0000", 2024-02-05T04:54:00.366 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.6199999999999996e-07 2024-02-05T04:54:00.366 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.366 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.366 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:00.366 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.959838+0000", 2024-02-05T04:54:00.367 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2660000000000001e-05 2024-02-05T04:54:00.367 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.367 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.367 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:00.367 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.959851+0000", 2024-02-05T04:54:00.367 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2357e-05 2024-02-05T04:54:00.367 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.367 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.367 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:00.367 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.959858+0000", 2024-02-05T04:54:00.367 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1219999999999997e-06 2024-02-05T04:54:00.367 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.367 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.367 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:00.367 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.959929+0000", 2024-02-05T04:54:00.367 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0974000000000002e-05 2024-02-05T04:54:00.368 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.368 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.368 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:00.368 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.959941+0000", 2024-02-05T04:54:00.368 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.239e-05 2024-02-05T04:54:00.368 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.368 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.368 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:00.368 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.967232+0000", 2024-02-05T04:54:00.368 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0072903559999999996 2024-02-05T04:54:00.368 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.368 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.368 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:00.368 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.967295+0000", 2024-02-05T04:54:00.368 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3076000000000004e-05 2024-02-05T04:54:00.368 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.369 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:00.369 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.369 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.369 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.369 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.7s6 58/31 ECSubWrite(tid=247, reqid=client.4634.0:520, at_version=58'30, trim_to=0'0, roll_forward_to=57'29))", 2024-02-05T04:54:00.369 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:49.962138+0000", 2024-02-05T04:54:00.369 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 10.329999764, 2024-02-05T04:54:00.369 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0097069860000000008, 2024-02-05T04:54:00.369 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:00.369 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:00.369 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:00.369 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.369 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:00.369 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.962138+0000", 2024-02-05T04:54:00.369 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:00.370 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.370 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.370 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:00.370 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.962138+0000", 2024-02-05T04:54:00.370 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:00.370 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.370 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.370 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:00.370 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.962138+0000", 2024-02-05T04:54:00.370 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9100000000000004e-07 2024-02-05T04:54:00.370 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.370 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.370 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:00.370 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.962957+0000", 2024-02-05T04:54:00.370 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00081820799999999996 2024-02-05T04:54:00.371 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.371 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.371 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:00.371 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.962970+0000", 2024-02-05T04:54:00.371 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3910000000000001e-05 2024-02-05T04:54:00.371 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.371 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.371 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:00.371 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.962979+0000", 2024-02-05T04:54:00.371 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4489999999999999e-06 2024-02-05T04:54:00.371 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.371 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.371 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:00.371 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.963059+0000", 2024-02-05T04:54:00.371 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0224999999999998e-05 2024-02-05T04:54:00.371 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.372 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.372 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:00.372 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.963069+0000", 2024-02-05T04:54:00.372 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6369999999999994e-06 2024-02-05T04:54:00.372 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.372 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.372 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:00.372 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.971785+0000", 2024-02-05T04:54:00.372 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0087162209999999997 2024-02-05T04:54:00.372 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.372 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.372 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:00.372 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.971845+0000", 2024-02-05T04:54:00.372 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9744999999999998e-05 2024-02-05T04:54:00.372 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.373 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:00.373 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.373 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.373 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.373 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.1s1 58/57 ECSubWrite(tid=248, reqid=osd.13.0:248, at_version=58'49, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:00.373 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:49.963004+0000", 2024-02-05T04:54:00.373 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 10.329133797000001, 2024-02-05T04:54:00.373 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0087081989999999998, 2024-02-05T04:54:00.373 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:00.373 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:00.373 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:00.373 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.373 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:00.373 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.963004+0000", 2024-02-05T04:54:00.373 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:00.373 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.374 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.374 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:00.374 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.963004+0000", 2024-02-05T04:54:00.374 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:00.374 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.374 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.374 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:00.374 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.963004+0000", 2024-02-05T04:54:00.374 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9200000000000001e-07 2024-02-05T04:54:00.374 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.374 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.374 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:00.374 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.963024+0000", 2024-02-05T04:54:00.374 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0183999999999999e-05 2024-02-05T04:54:00.374 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.375 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.375 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:00.375 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.963043+0000", 2024-02-05T04:54:00.375 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8213e-05 2024-02-05T04:54:00.375 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.375 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.375 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:00.375 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.963050+0000", 2024-02-05T04:54:00.375 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8560000000000007e-06 2024-02-05T04:54:00.375 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.375 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.375 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:00.375 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.963119+0000", 2024-02-05T04:54:00.375 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8858999999999999e-05 2024-02-05T04:54:00.375 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.375 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.376 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:00.376 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.963127+0000", 2024-02-05T04:54:00.376 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.1580000000000005e-06 2024-02-05T04:54:00.376 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.376 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.376 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:00.376 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.971647+0000", 2024-02-05T04:54:00.376 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0085198770000000004 2024-02-05T04:54:00.376 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.376 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.376 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:00.376 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.971712+0000", 2024-02-05T04:54:00.376 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4560000000000005e-05 2024-02-05T04:54:00.376 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.376 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:00.376 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.377 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.377 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.377 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.1s1 63/62 ECSubWrite(tid=265, reqid=client.4634.0:594, at_version=63'53, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:00.377 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:55.004443+0000", 2024-02-05T04:54:00.377 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 5.2876945759999998, 2024-02-05T04:54:00.377 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0077481909999999998, 2024-02-05T04:54:00.377 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:00.377 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:00.377 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:00.377 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.377 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:00.377 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.004443+0000", 2024-02-05T04:54:00.377 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:00.377 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.377 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.377 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:00.378 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.004443+0000", 2024-02-05T04:54:00.378 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:00.378 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.378 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.378 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:00.378 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.004444+0000", 2024-02-05T04:54:00.378 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7000000000000005e-07 2024-02-05T04:54:00.378 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.378 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.378 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:00.378 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.004924+0000", 2024-02-05T04:54:00.378 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00048008099999999999 2024-02-05T04:54:00.378 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.378 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.378 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:00.378 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.004939+0000", 2024-02-05T04:54:00.379 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5699999999999999e-05 2024-02-05T04:54:00.379 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.379 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.379 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:00.379 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.004947+0000", 2024-02-05T04:54:00.379 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8429999999999993e-06 2024-02-05T04:54:00.379 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.379 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.379 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:00.379 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.004999+0000", 2024-02-05T04:54:00.379 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2181000000000001e-05 2024-02-05T04:54:00.379 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.379 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.379 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:00.379 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.005006+0000", 2024-02-05T04:54:00.379 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7850000000000003e-06 2024-02-05T04:54:00.380 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.380 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.380 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:00.380 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.012155+0000", 2024-02-05T04:54:00.380 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0071493629999999997 2024-02-05T04:54:00.380 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.380 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.380 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:00.380 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.012191+0000", 2024-02-05T04:54:00.380 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5667999999999998e-05 2024-02-05T04:54:00.380 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.380 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:00.380 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.380 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.380 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.381 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.12s4 68/67 ECSubWrite(tid=85, reqid=osd.2.0:85, at_version=68'57, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:00.381 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:59.961627+0000", 2024-02-05T04:54:00.381 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.33051016599999999, 2024-02-05T04:54:00.381 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0075150820000000002, 2024-02-05T04:54:00.381 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:00.381 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:00.381 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:00.381 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.381 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:00.381 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.961627+0000", 2024-02-05T04:54:00.381 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:00.381 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.381 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.381 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:00.381 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.961627+0000", 2024-02-05T04:54:00.381 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:00.382 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.382 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.382 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:00.382 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.961628+0000", 2024-02-05T04:54:00.382 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7300000000000002e-07 2024-02-05T04:54:00.382 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.382 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.382 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:00.382 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.961651+0000", 2024-02-05T04:54:00.382 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3359000000000001e-05 2024-02-05T04:54:00.382 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.382 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.382 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:00.382 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.961660+0000", 2024-02-05T04:54:00.382 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1570000000000006e-06 2024-02-05T04:54:00.383 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.383 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.383 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:00.383 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.961666+0000", 2024-02-05T04:54:00.383 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8730000000000004e-06 2024-02-05T04:54:00.383 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.383 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.383 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:00.383 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.962056+0000", 2024-02-05T04:54:00.383 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00038990500000000002 2024-02-05T04:54:00.383 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.383 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.383 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:00.383 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.962068+0000", 2024-02-05T04:54:00.383 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2016e-05 2024-02-05T04:54:00.383 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.384 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.384 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:00.384 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.969107+0000", 2024-02-05T04:54:00.384 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0070391910000000002 2024-02-05T04:54:00.384 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.384 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.384 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:00.384 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.969142+0000", 2024-02-05T04:54:00.384 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5207999999999999e-05 2024-02-05T04:54:00.384 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.384 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:00.384 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.384 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.384 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:00.384 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T04:54:00.386 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-02-05T04:54:00.386 INFO:tasks.ceph.osd.15.smithi196.stderr:2024-02-05T04:54:00.385+0000 7f1facc45640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 34611) UID: 0 2024-02-05T04:54:00.487 INFO:tasks.ceph.osd.13.smithi091.stderr:2024-02-05T04:54:00.486+0000 7f526fd4d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 34281) UID: 0 2024-02-05T04:54:00.532 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T04:54:00.532 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-02-05T04:54:00.533 INFO:teuthology.orchestra.run.smithi196.stdout: "num_ops": 0 2024-02-05T04:54:00.533 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T04:54:00.541 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-02-05T04:54:00.588 INFO:tasks.ceph.osd.11.smithi196.stderr:2024-02-05T04:54:00.587+0000 7f85a8d86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 34609) UID: 0 2024-02-05T04:54:00.647 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T04:54:00.647 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-02-05T04:54:00.647 INFO:teuthology.orchestra.run.smithi196.stdout: "complaint_time": 30, 2024-02-05T04:54:00.647 INFO:teuthology.orchestra.run.smithi196.stdout: "num_blocked_ops": 0 2024-02-05T04:54:00.647 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T04:54:00.655 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-02-05T04:54:00.689 INFO:tasks.ceph.osd.6.smithi150.stderr:2024-02-05T04:54:00.688+0000 7f6bea9f0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34015) UID: 0 2024-02-05T04:54:00.789 INFO:tasks.ceph.osd.12.smithi062.stderr:2024-02-05T04:54:00.789+0000 7f5e5847e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 34695) UID: 0 2024-02-05T04:54:00.804 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T04:54:00.804 INFO:teuthology.orchestra.run.smithi196.stdout: "size": 20, 2024-02-05T04:54:00.804 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 600, 2024-02-05T04:54:00.804 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-02-05T04:54:00.805 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.805 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.3s3 43/31 ECSubWrite(tid=63, reqid=osd.4.0:63, at_version=43'27, trim_to=0'0, roll_forward_to=40'25))", 2024-02-05T04:54:00.805 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:35.173378+0000", 2024-02-05T04:54:00.805 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 25.629183050000002, 2024-02-05T04:54:00.805 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0066975339999999998, 2024-02-05T04:54:00.805 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:00.805 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:00.805 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:00.805 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.805 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:00.805 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.173378+0000", 2024-02-05T04:54:00.805 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:00.805 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.805 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.806 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:00.806 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.173378+0000", 2024-02-05T04:54:00.806 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:00.806 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.806 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.806 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:00.806 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.173378+0000", 2024-02-05T04:54:00.806 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9599999999999999e-07 2024-02-05T04:54:00.806 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.806 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.806 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:00.806 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.173394+0000", 2024-02-05T04:54:00.806 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6269e-05 2024-02-05T04:54:00.806 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.806 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.807 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:00.807 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.173405+0000", 2024-02-05T04:54:00.807 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0372e-05 2024-02-05T04:54:00.807 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.807 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.807 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:00.807 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.173411+0000", 2024-02-05T04:54:00.807 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0549999999999996e-06 2024-02-05T04:54:00.807 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.807 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.807 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:00.807 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.173465+0000", 2024-02-05T04:54:00.807 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3665000000000002e-05 2024-02-05T04:54:00.807 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.807 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.807 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:00.808 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.173471+0000", 2024-02-05T04:54:00.808 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1079999999999998e-06 2024-02-05T04:54:00.808 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.808 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.808 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:00.808 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.180035+0000", 2024-02-05T04:54:00.808 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0065645019999999998 2024-02-05T04:54:00.808 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.808 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.808 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:00.808 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.180075+0000", 2024-02-05T04:54:00.808 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9966999999999999e-05 2024-02-05T04:54:00.808 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.808 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:00.809 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.809 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.809 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.809 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.6s3 43/31 ECSubWrite(tid=141, reqid=osd.13.0:138, at_version=0'0, trim_to=0'0, roll_forward_to=43'24))", 2024-02-05T04:54:00.809 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:35.183155+0000", 2024-02-05T04:54:00.809 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 25.619405549, 2024-02-05T04:54:00.809 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0055237740000000004, 2024-02-05T04:54:00.809 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:00.809 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:00.809 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:00.809 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.809 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:00.809 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.183155+0000", 2024-02-05T04:54:00.809 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:00.809 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.809 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.810 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:00.810 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.183155+0000", 2024-02-05T04:54:00.810 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:00.810 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.810 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.810 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:00.810 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.183156+0000", 2024-02-05T04:54:00.810 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.7000000000000003e-07 2024-02-05T04:54:00.810 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.810 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.810 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:00.810 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.183168+0000", 2024-02-05T04:54:00.810 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1644999999999999e-05 2024-02-05T04:54:00.810 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.810 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.811 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:00.811 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.183184+0000", 2024-02-05T04:54:00.811 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.63e-05 2024-02-05T04:54:00.811 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.811 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.811 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:00.811 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.183195+0000", 2024-02-05T04:54:00.811 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1456e-05 2024-02-05T04:54:00.811 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.811 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.811 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:00.811 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.183257+0000", 2024-02-05T04:54:00.811 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1734000000000002e-05 2024-02-05T04:54:00.811 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.812 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.812 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:00.812 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.183267+0000", 2024-02-05T04:54:00.812 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8379999999999997e-06 2024-02-05T04:54:00.812 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.812 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.812 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:00.812 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.188639+0000", 2024-02-05T04:54:00.812 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0053717189999999996 2024-02-05T04:54:00.812 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.812 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.812 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:00.812 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.188679+0000", 2024-02-05T04:54:00.812 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0212000000000001e-05 2024-02-05T04:54:00.812 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.812 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:00.813 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.813 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.813 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.813 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.12s4 63/62 ECSubWrite(tid=73, reqid=osd.12.0:73, at_version=63'55, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:00.813 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:54.999240+0000", 2024-02-05T04:54:00.813 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 5.8033211360000001, 2024-02-05T04:54:00.813 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0059916839999999997, 2024-02-05T04:54:00.813 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:00.813 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:00.813 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:00.813 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.814 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:00.814 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:54.999240+0000", 2024-02-05T04:54:00.814 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:00.814 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.814 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.814 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:00.814 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:54.999240+0000", 2024-02-05T04:54:00.814 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:00.814 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.814 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.814 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:00.814 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:54.999240+0000", 2024-02-05T04:54:00.814 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4300000000000002e-07 2024-02-05T04:54:00.814 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.814 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.815 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:00.815 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:54.999261+0000", 2024-02-05T04:54:00.815 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0831000000000001e-05 2024-02-05T04:54:00.815 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.815 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.815 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:00.815 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:54.999273+0000", 2024-02-05T04:54:00.815 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1798e-05 2024-02-05T04:54:00.815 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.815 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.815 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:00.815 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:54.999282+0000", 2024-02-05T04:54:00.815 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.4280000000000002e-06 2024-02-05T04:54:00.815 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.815 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.816 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:00.816 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:54.999339+0000", 2024-02-05T04:54:00.816 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6168999999999997e-05 2024-02-05T04:54:00.816 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.816 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.816 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:00.816 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:54.999349+0000", 2024-02-05T04:54:00.816 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.9289999999999993e-06 2024-02-05T04:54:00.816 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.816 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.816 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:00.816 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.005194+0000", 2024-02-05T04:54:00.816 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0058456999999999997 2024-02-05T04:54:00.816 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.816 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.817 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:00.817 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.005231+0000", 2024-02-05T04:54:00.817 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6986000000000001e-05 2024-02-05T04:54:00.817 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.817 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:00.817 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.817 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.817 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.817 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.0s2 63/62 ECSubWrite(tid=52, reqid=client.4634.0:576, at_version=63'6, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:00.817 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:55.000860+0000", 2024-02-05T04:54:00.817 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 5.8017002020000001, 2024-02-05T04:54:00.817 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.010870894000000001, 2024-02-05T04:54:00.817 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:00.817 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:00.817 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:00.818 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.818 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:00.818 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.000860+0000", 2024-02-05T04:54:00.818 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:00.818 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.818 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.818 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:00.818 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.000860+0000", 2024-02-05T04:54:00.818 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:00.818 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.818 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.818 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:00.818 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.000861+0000", 2024-02-05T04:54:00.818 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.61e-07 2024-02-05T04:54:00.818 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.819 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.819 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:00.819 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.001743+0000", 2024-02-05T04:54:00.819 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00088195999999999999 2024-02-05T04:54:00.819 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.819 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.819 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:00.819 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.001753+0000", 2024-02-05T04:54:00.819 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0370000000000001e-05 2024-02-05T04:54:00.819 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.819 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.819 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:00.819 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.001762+0000", 2024-02-05T04:54:00.819 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.6649999999999992e-06 2024-02-05T04:54:00.819 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.820 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.820 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:00.820 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.002420+0000", 2024-02-05T04:54:00.820 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00065761699999999999 2024-02-05T04:54:00.820 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.820 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.820 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:00.820 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.002430+0000", 2024-02-05T04:54:00.820 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0366e-05 2024-02-05T04:54:00.820 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.820 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.820 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:00.820 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.011697+0000", 2024-02-05T04:54:00.820 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0092668270000000001 2024-02-05T04:54:00.820 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.821 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.821 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:00.821 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.011731+0000", 2024-02-05T04:54:00.821 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4428000000000003e-05 2024-02-05T04:54:00.821 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.821 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:00.821 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.821 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.821 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.821 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.0s2 63/62 ECSubWrite(tid=53, reqid=client.4634.0:577, at_version=63'7, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:00.821 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:55.003409+0000", 2024-02-05T04:54:00.821 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 5.7991521190000004, 2024-02-05T04:54:00.821 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.011422261, 2024-02-05T04:54:00.821 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:00.821 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:00.822 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:00.822 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.822 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:00.822 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.003409+0000", 2024-02-05T04:54:00.822 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:00.822 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.822 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.822 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:00.822 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.003409+0000", 2024-02-05T04:54:00.822 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:00.822 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.822 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.822 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:00.823 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.003409+0000", 2024-02-05T04:54:00.823 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5899999999999995e-07 2024-02-05T04:54:00.823 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.823 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.823 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:00.823 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.003761+0000", 2024-02-05T04:54:00.823 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00035215600000000001 2024-02-05T04:54:00.823 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.823 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.823 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:00.823 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.003774+0000", 2024-02-05T04:54:00.823 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.289e-05 2024-02-05T04:54:00.823 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.824 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.824 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:00.824 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.003785+0000", 2024-02-05T04:54:00.825 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0634e-05 2024-02-05T04:54:00.825 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.825 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.825 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:00.825 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.005279+0000", 2024-02-05T04:54:00.825 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001494287 2024-02-05T04:54:00.825 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.825 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.825 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:00.825 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.005289+0000", 2024-02-05T04:54:00.825 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.7839999999999998e-06 2024-02-05T04:54:00.825 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.825 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.825 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:00.825 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.014797+0000", 2024-02-05T04:54:00.826 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0095077860000000007 2024-02-05T04:54:00.826 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.826 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.826 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:00.826 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.014831+0000", 2024-02-05T04:54:00.826 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3964999999999998e-05 2024-02-05T04:54:00.826 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.826 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:00.826 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.826 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.826 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.826 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.0s2 63/62 ECSubWrite(tid=54, reqid=client.4634.0:578, at_version=63'8, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:00.826 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:55.006124+0000", 2024-02-05T04:54:00.826 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 5.7964365600000001, 2024-02-05T04:54:00.827 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0087667140000000001, 2024-02-05T04:54:00.827 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:00.827 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:00.827 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:00.827 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.827 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:00.827 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.006124+0000", 2024-02-05T04:54:00.827 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:00.827 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.827 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.827 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:00.827 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.006124+0000", 2024-02-05T04:54:00.827 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:00.827 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.827 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.828 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:00.828 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.006125+0000", 2024-02-05T04:54:00.828 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8100000000000002e-07 2024-02-05T04:54:00.828 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.828 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.828 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:00.828 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.006211+0000", 2024-02-05T04:54:00.828 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.6119000000000007e-05 2024-02-05T04:54:00.828 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.828 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.828 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:00.828 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.006226+0000", 2024-02-05T04:54:00.828 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4535999999999999e-05 2024-02-05T04:54:00.829 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.829 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.829 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:00.829 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.006235+0000", 2024-02-05T04:54:00.829 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6080000000000002e-06 2024-02-05T04:54:00.829 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.829 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.829 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:00.829 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.006281+0000", 2024-02-05T04:54:00.829 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5664000000000002e-05 2024-02-05T04:54:00.829 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.829 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.829 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:00.829 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.006287+0000", 2024-02-05T04:54:00.830 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9850000000000003e-06 2024-02-05T04:54:00.830 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.830 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.830 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:00.830 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.014862+0000", 2024-02-05T04:54:00.830 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0085751730000000002 2024-02-05T04:54:00.830 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.830 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.830 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:00.830 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.014891+0000", 2024-02-05T04:54:00.830 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8847999999999999e-05 2024-02-05T04:54:00.830 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.830 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:00.830 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.831 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.831 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.831 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.13s3 63/62 ECSubWrite(tid=85, reqid=osd.4.0:83, at_version=0'0, trim_to=0'0, roll_forward_to=63'39))", 2024-02-05T04:54:00.831 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:55.008043+0000", 2024-02-05T04:54:00.831 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 5.7945174310000001, 2024-02-05T04:54:00.831 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0070188220000000001, 2024-02-05T04:54:00.831 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:00.831 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:00.831 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:00.831 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.831 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:00.831 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.008043+0000", 2024-02-05T04:54:00.832 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:00.832 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.832 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.832 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:00.832 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.008043+0000", 2024-02-05T04:54:00.832 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:00.832 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.832 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.832 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:00.832 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.008044+0000", 2024-02-05T04:54:00.832 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8999999999999996e-07 2024-02-05T04:54:00.835 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.835 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.835 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:00.835 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.008055+0000", 2024-02-05T04:54:00.835 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1586000000000001e-05 2024-02-05T04:54:00.836 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.836 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.836 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:00.836 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.008067+0000", 2024-02-05T04:54:00.836 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1936000000000001e-05 2024-02-05T04:54:00.836 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.836 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.836 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:00.836 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.008075+0000", 2024-02-05T04:54:00.836 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9910000000000002e-06 2024-02-05T04:54:00.836 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.836 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.836 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:00.836 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.008139+0000", 2024-02-05T04:54:00.836 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3610999999999998e-05 2024-02-05T04:54:00.837 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.837 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.837 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:00.837 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.008393+0000", 2024-02-05T04:54:00.837 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00025358200000000003 2024-02-05T04:54:00.837 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.837 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.837 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:00.837 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.015002+0000", 2024-02-05T04:54:00.837 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0066095829999999996 2024-02-05T04:54:00.837 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.837 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.837 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:00.837 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.015062+0000", 2024-02-05T04:54:00.837 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9843000000000003e-05 2024-02-05T04:54:00.838 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.838 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:00.838 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.838 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.838 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.838 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.1s4 63/62 ECSubWrite(tid=265, reqid=client.4634.0:594, at_version=63'53, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:00.838 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:55.008496+0000", 2024-02-05T04:54:00.838 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 5.7940648030000004, 2024-02-05T04:54:00.838 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0086120399999999996, 2024-02-05T04:54:00.838 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:00.838 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:00.838 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:00.838 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.838 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:00.838 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.008496+0000", 2024-02-05T04:54:00.839 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:00.839 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.839 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.839 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:00.839 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.008496+0000", 2024-02-05T04:54:00.839 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:00.839 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.839 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.839 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:00.840 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.008497+0000", 2024-02-05T04:54:00.840 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.0599999999999999e-07 2024-02-05T04:54:00.840 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.840 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.840 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:00.840 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.008863+0000", 2024-02-05T04:54:00.840 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000366468 2024-02-05T04:54:00.840 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.840 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.840 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:00.840 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.008875+0000", 2024-02-05T04:54:00.840 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1980999999999999e-05 2024-02-05T04:54:00.840 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.840 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.841 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:00.841 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.008884+0000", 2024-02-05T04:54:00.841 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.9299999999999992e-06 2024-02-05T04:54:00.841 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.841 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.841 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:00.841 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.008965+0000", 2024-02-05T04:54:00.841 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.1299999999999997e-05 2024-02-05T04:54:00.841 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.841 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.841 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:00.841 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.008977+0000", 2024-02-05T04:54:00.841 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1874e-05 2024-02-05T04:54:00.841 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.842 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.842 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:00.842 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.017067+0000", 2024-02-05T04:54:00.842 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0080894299999999999 2024-02-05T04:54:00.842 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.842 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.842 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:00.842 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.017108+0000", 2024-02-05T04:54:00.842 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1151e-05 2024-02-05T04:54:00.842 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.842 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:00.842 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.842 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.842 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.842 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.1s4 63/62 ECSubWrite(tid=267, reqid=client.4634.0:595, at_version=63'54, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:00.843 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:55.010218+0000", 2024-02-05T04:54:00.843 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 5.7923431010000002, 2024-02-05T04:54:00.843 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0069481550000000001, 2024-02-05T04:54:00.843 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:00.843 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:00.843 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:00.843 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.843 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:00.843 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.010218+0000", 2024-02-05T04:54:00.843 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:00.843 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.843 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.843 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:00.843 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.010218+0000", 2024-02-05T04:54:00.843 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:00.843 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.844 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.844 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:00.844 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.010218+0000", 2024-02-05T04:54:00.844 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8300000000000001e-07 2024-02-05T04:54:00.844 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.844 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.844 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:00.844 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.010481+0000", 2024-02-05T04:54:00.844 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00026307700000000002 2024-02-05T04:54:00.844 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.844 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.844 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:00.844 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.010492+0000", 2024-02-05T04:54:00.844 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1484999999999999e-05 2024-02-05T04:54:00.845 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.845 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.845 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:00.845 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.010499+0000", 2024-02-05T04:54:00.845 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1119999999999998e-06 2024-02-05T04:54:00.845 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.845 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.845 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:00.845 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.012334+0000", 2024-02-05T04:54:00.845 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001834114 2024-02-05T04:54:00.845 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.845 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.845 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:00.846 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.012343+0000", 2024-02-05T04:54:00.846 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.3409999999999992e-06 2024-02-05T04:54:00.846 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.846 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.846 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:00.846 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.017148+0000", 2024-02-05T04:54:00.846 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0048050840000000003 2024-02-05T04:54:00.846 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.846 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.846 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:00.846 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.017166+0000", 2024-02-05T04:54:00.846 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7759000000000001e-05 2024-02-05T04:54:00.846 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.846 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:00.846 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.847 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.847 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.847 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.0s2 63/62 ECSubWrite(tid=55, reqid=client.4634.0:579, at_version=63'9, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:00.847 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:55.011796+0000", 2024-02-05T04:54:00.847 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 5.7907642399999997, 2024-02-05T04:54:00.847 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0051310030000000003, 2024-02-05T04:54:00.847 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:00.847 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:00.847 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:00.847 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.847 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:00.847 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.011796+0000", 2024-02-05T04:54:00.847 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:00.847 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.847 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.847 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:00.848 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.011796+0000", 2024-02-05T04:54:00.848 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:00.848 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.849 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.849 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:00.849 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.011797+0000", 2024-02-05T04:54:00.849 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0999999999999998e-07 2024-02-05T04:54:00.849 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.849 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.849 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:00.849 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.011813+0000", 2024-02-05T04:54:00.849 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6175999999999999e-05 2024-02-05T04:54:00.849 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.849 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.850 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:00.850 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.011825+0000", 2024-02-05T04:54:00.850 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1786e-05 2024-02-05T04:54:00.850 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.850 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.850 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:00.850 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.011834+0000", 2024-02-05T04:54:00.850 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4800000000000001e-06 2024-02-05T04:54:00.850 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.850 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.850 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:00.850 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.012408+0000", 2024-02-05T04:54:00.850 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00057394599999999998 2024-02-05T04:54:00.850 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.851 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.851 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:00.851 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.012416+0000", 2024-02-05T04:54:00.851 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4719999999999995e-06 2024-02-05T04:54:00.851 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.851 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.851 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:00.851 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.016876+0000", 2024-02-05T04:54:00.851 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0044597619999999999 2024-02-05T04:54:00.851 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.851 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.851 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:00.851 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.016927+0000", 2024-02-05T04:54:00.851 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1671e-05 2024-02-05T04:54:00.851 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.852 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:00.852 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.852 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.852 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.852 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.1s4 63/62 ECSubWrite(tid=271, reqid=osd.13.0:271, at_version=63'58, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:00.852 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:55.016645+0000", 2024-02-05T04:54:00.852 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 5.7859155439999999, 2024-02-05T04:54:00.852 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0051876810000000004, 2024-02-05T04:54:00.852 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:00.852 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:00.852 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:00.852 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.852 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:00.852 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.016645+0000", 2024-02-05T04:54:00.853 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:00.853 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.853 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.853 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:00.853 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.016645+0000", 2024-02-05T04:54:00.853 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:00.853 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.853 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.853 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:00.853 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.016646+0000", 2024-02-05T04:54:00.853 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8100000000000002e-07 2024-02-05T04:54:00.853 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.853 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.853 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:00.853 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.016661+0000", 2024-02-05T04:54:00.854 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5051e-05 2024-02-05T04:54:00.854 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.854 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.854 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:00.854 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.016701+0000", 2024-02-05T04:54:00.854 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0207999999999999e-05 2024-02-05T04:54:00.854 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.854 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.854 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:00.854 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.016711+0000", 2024-02-05T04:54:00.854 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0107e-05 2024-02-05T04:54:00.854 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.854 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.854 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:00.854 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.016766+0000", 2024-02-05T04:54:00.855 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5167999999999998e-05 2024-02-05T04:54:00.855 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.855 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.855 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:00.855 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.016773+0000", 2024-02-05T04:54:00.855 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3840000000000002e-06 2024-02-05T04:54:00.855 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.855 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.855 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:00.855 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.021798+0000", 2024-02-05T04:54:00.855 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0050257009999999996 2024-02-05T04:54:00.855 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.855 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.855 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:00.855 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.021833+0000", 2024-02-05T04:54:00.856 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4381000000000002e-05 2024-02-05T04:54:00.856 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.856 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:00.856 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.856 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.856 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.856 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.1s4 68/62 ECSubWrite(tid=290, reqid=client.4634.0:621, at_version=68'64, trim_to=0'0, roll_forward_to=66'62))", 2024-02-05T04:54:00.856 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:59.956235+0000", 2024-02-05T04:54:00.856 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.84632533799999998, 2024-02-05T04:54:00.856 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.005917238, 2024-02-05T04:54:00.856 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:00.856 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:00.856 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:00.856 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.856 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:00.857 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.956235+0000", 2024-02-05T04:54:00.857 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:00.857 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.857 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.857 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:00.857 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.956235+0000", 2024-02-05T04:54:00.857 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:00.857 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.857 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.857 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:00.857 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.956236+0000", 2024-02-05T04:54:00.857 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8299999999999998e-07 2024-02-05T04:54:00.857 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.857 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.857 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:00.857 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.956258+0000", 2024-02-05T04:54:00.858 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2123000000000001e-05 2024-02-05T04:54:00.858 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.858 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.858 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:00.858 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.956267+0000", 2024-02-05T04:54:00.858 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8769999999999999e-06 2024-02-05T04:54:00.858 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.858 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.858 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:00.858 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.956275+0000", 2024-02-05T04:54:00.859 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.259e-06 2024-02-05T04:54:00.859 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.859 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.859 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:00.859 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.956340+0000", 2024-02-05T04:54:00.859 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5438000000000006e-05 2024-02-05T04:54:00.859 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.859 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.859 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:00.859 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.956352+0000", 2024-02-05T04:54:00.859 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1793e-05 2024-02-05T04:54:00.859 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.859 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.860 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:00.860 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.962120+0000", 2024-02-05T04:54:00.860 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0057683530000000004 2024-02-05T04:54:00.860 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.860 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.860 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:00.860 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.962153+0000", 2024-02-05T04:54:00.860 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2112e-05 2024-02-05T04:54:00.860 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.861 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:00.861 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.861 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.861 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.861 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.1s4 68/62 ECSubWrite(tid=291, reqid=client.4634.0:622, at_version=68'65, trim_to=0'0, roll_forward_to=66'62))", 2024-02-05T04:54:00.861 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:59.958697+0000", 2024-02-05T04:54:00.861 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.84386378799999995, 2024-02-05T04:54:00.861 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0066247399999999996, 2024-02-05T04:54:00.861 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:00.861 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:00.861 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:00.861 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.861 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:00.861 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.958697+0000", 2024-02-05T04:54:00.861 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:00.862 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.862 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.862 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:00.862 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.958697+0000", 2024-02-05T04:54:00.862 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:00.862 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.862 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.862 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:00.862 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.958697+0000", 2024-02-05T04:54:00.862 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2800000000000003e-07 2024-02-05T04:54:00.862 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.862 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.862 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:00.862 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.958713+0000", 2024-02-05T04:54:00.862 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5682e-05 2024-02-05T04:54:00.862 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.863 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.863 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:00.863 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.958722+0000", 2024-02-05T04:54:00.863 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.9509999999999995e-06 2024-02-05T04:54:00.863 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.863 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.863 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:00.863 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.958731+0000", 2024-02-05T04:54:00.863 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.9029999999999993e-06 2024-02-05T04:54:00.863 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.863 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.863 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:00.863 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.959236+0000", 2024-02-05T04:54:00.863 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00050540800000000005 2024-02-05T04:54:00.864 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.864 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.864 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:00.864 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.959246+0000", 2024-02-05T04:54:00.864 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.004e-05 2024-02-05T04:54:00.864 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.864 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.864 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:00.864 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.965292+0000", 2024-02-05T04:54:00.864 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0060456559999999999 2024-02-05T04:54:00.864 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.864 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.864 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:00.865 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.965322+0000", 2024-02-05T04:54:00.865 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.9771999999999999e-05 2024-02-05T04:54:00.865 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.865 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:00.865 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.865 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.865 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.865 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.13s2 68/67 ECSubWrite(tid=131, reqid=osd.11.0:131, at_version=68'41, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:00.865 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:59.958999+0000", 2024-02-05T04:54:00.865 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.84356126099999995, 2024-02-05T04:54:00.865 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0062521169999999997, 2024-02-05T04:54:00.865 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:00.865 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:00.866 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:00.866 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.866 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:00.866 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.958999+0000", 2024-02-05T04:54:00.866 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:00.866 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.866 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.866 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:00.866 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.958999+0000", 2024-02-05T04:54:00.866 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:00.866 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.866 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.866 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:00.866 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.959000+0000", 2024-02-05T04:54:00.866 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.34e-07 2024-02-05T04:54:00.867 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.867 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.867 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:00.867 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.959023+0000", 2024-02-05T04:54:00.867 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2861999999999999e-05 2024-02-05T04:54:00.867 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.867 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.867 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:00.867 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.959036+0000", 2024-02-05T04:54:00.867 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3281999999999999e-05 2024-02-05T04:54:00.867 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.867 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.867 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:00.867 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.959043+0000", 2024-02-05T04:54:00.867 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4200000000000001e-06 2024-02-05T04:54:00.868 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.868 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.868 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:00.868 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.959112+0000", 2024-02-05T04:54:00.868 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8615999999999995e-05 2024-02-05T04:54:00.868 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.868 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.868 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:00.868 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.959122+0000", 2024-02-05T04:54:00.868 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0195e-05 2024-02-05T04:54:00.868 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.868 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.868 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:00.868 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.965213+0000", 2024-02-05T04:54:00.869 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0060908200000000003 2024-02-05T04:54:00.869 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.869 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.869 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:00.869 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.965252+0000", 2024-02-05T04:54:00.869 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8587999999999997e-05 2024-02-05T04:54:00.869 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.869 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:00.869 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.869 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.869 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.869 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.1s4 68/62 ECSubWrite(tid=292, reqid=client.4634.0:623, at_version=68'66, trim_to=0'0, roll_forward_to=66'62))", 2024-02-05T04:54:00.869 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:59.959533+0000", 2024-02-05T04:54:00.870 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.84302769700000002, 2024-02-05T04:54:00.870 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0058244910000000002, 2024-02-05T04:54:00.870 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:00.870 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:00.870 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:00.870 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.870 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:00.870 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.959533+0000", 2024-02-05T04:54:00.870 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:00.870 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.870 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.870 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:00.870 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.959533+0000", 2024-02-05T04:54:00.870 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:00.871 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.871 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.871 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:00.871 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.959533+0000", 2024-02-05T04:54:00.871 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0299999999999999e-07 2024-02-05T04:54:00.871 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.871 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.871 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:00.871 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.959549+0000", 2024-02-05T04:54:00.871 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5150999999999999e-05 2024-02-05T04:54:00.871 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.871 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.871 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:00.871 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.959558+0000", 2024-02-05T04:54:00.871 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2350000000000005e-06 2024-02-05T04:54:00.871 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.872 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.872 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:00.872 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.959566+0000", 2024-02-05T04:54:00.872 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9449999999999993e-06 2024-02-05T04:54:00.872 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.872 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.872 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:00.872 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.959721+0000", 2024-02-05T04:54:00.872 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00015551599999999999 2024-02-05T04:54:00.872 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.872 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.872 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:00.872 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.959730+0000", 2024-02-05T04:54:00.873 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1249999999999999e-06 2024-02-05T04:54:00.873 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.873 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.873 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:00.873 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.965324+0000", 2024-02-05T04:54:00.873 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0055938280000000003 2024-02-05T04:54:00.873 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.873 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.873 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:00.874 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.965357+0000", 2024-02-05T04:54:00.874 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3188000000000001e-05 2024-02-05T04:54:00.874 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.875 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:00.875 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.875 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.875 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.875 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.12s6 68/67 ECSubWrite(tid=85, reqid=osd.2.0:85, at_version=68'57, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:00.875 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:59.961044+0000", 2024-02-05T04:54:00.875 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.84151652600000004, 2024-02-05T04:54:00.875 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0065076409999999998, 2024-02-05T04:54:00.875 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:00.875 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:00.875 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:00.875 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.875 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:00.875 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.961044+0000", 2024-02-05T04:54:00.876 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:00.876 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.876 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.876 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:00.876 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.961044+0000", 2024-02-05T04:54:00.876 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:00.876 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.876 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.876 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:00.876 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.961045+0000", 2024-02-05T04:54:00.876 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1900000000000002e-07 2024-02-05T04:54:00.876 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.876 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.876 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:00.876 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.961064+0000", 2024-02-05T04:54:00.877 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8802000000000001e-05 2024-02-05T04:54:00.877 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.877 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.877 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:00.877 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.961075+0000", 2024-02-05T04:54:00.877 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1379999999999999e-05 2024-02-05T04:54:00.877 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.877 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.877 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:00.877 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.961083+0000", 2024-02-05T04:54:00.877 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.1699999999999997e-06 2024-02-05T04:54:00.877 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.877 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.877 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:00.877 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.961144+0000", 2024-02-05T04:54:00.878 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0417000000000001e-05 2024-02-05T04:54:00.878 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.878 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.878 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:00.878 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.961154+0000", 2024-02-05T04:54:00.878 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0436e-05 2024-02-05T04:54:00.878 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.878 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.878 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:00.878 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.967516+0000", 2024-02-05T04:54:00.878 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.006362319 2024-02-05T04:54:00.878 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.878 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.879 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:00.879 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.967552+0000", 2024-02-05T04:54:00.879 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5398000000000002e-05 2024-02-05T04:54:00.879 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.879 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:00.879 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.879 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.879 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.879 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.14s2 68/67 ECSubWrite(tid=15, reqid=osd.3.0:15, at_version=68'75, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:00.879 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:59.961971+0000", 2024-02-05T04:54:00.879 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.84058965799999996, 2024-02-05T04:54:00.879 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0056405989999999996, 2024-02-05T04:54:00.879 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:00.879 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:00.879 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:00.880 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.880 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:00.880 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.961971+0000", 2024-02-05T04:54:00.880 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:00.880 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.880 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.880 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:00.880 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.961971+0000", 2024-02-05T04:54:00.880 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:00.880 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.880 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.880 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:00.880 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.961971+0000", 2024-02-05T04:54:00.881 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8799999999999998e-07 2024-02-05T04:54:00.881 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.881 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.881 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:00.881 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.961992+0000", 2024-02-05T04:54:00.881 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0373999999999998e-05 2024-02-05T04:54:00.881 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.881 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.881 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:00.881 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.962001+0000", 2024-02-05T04:54:00.881 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1069999999999994e-06 2024-02-05T04:54:00.881 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.881 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.881 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:00.881 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.962021+0000", 2024-02-05T04:54:00.882 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9904000000000001e-05 2024-02-05T04:54:00.882 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.882 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.882 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:00.882 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.962082+0000", 2024-02-05T04:54:00.882 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1247999999999994e-05 2024-02-05T04:54:00.882 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.882 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.882 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:00.882 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.962091+0000", 2024-02-05T04:54:00.882 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8300000000000002e-06 2024-02-05T04:54:00.882 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.882 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.882 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:00.883 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.967582+0000", 2024-02-05T04:54:00.883 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0054910499999999999 2024-02-05T04:54:00.883 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.883 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.883 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:00.883 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.967612+0000", 2024-02-05T04:54:00.883 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.9697999999999999e-05 2024-02-05T04:54:00.883 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.883 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:00.883 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.883 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.883 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.883 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.14s2 68/67 ECSubWrite(tid=16, reqid=osd.3.0:16, at_version=68'77, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:00.883 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:59.963184+0000", 2024-02-05T04:54:00.883 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.83937651099999999, 2024-02-05T04:54:00.883 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0072996579999999997, 2024-02-05T04:54:00.884 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:00.884 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:00.884 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:00.884 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.884 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:00.884 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.963184+0000", 2024-02-05T04:54:00.884 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:00.884 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.884 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.884 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:00.884 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.963184+0000", 2024-02-05T04:54:00.884 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:00.884 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.885 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.885 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:00.885 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.963185+0000", 2024-02-05T04:54:00.885 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.2699999999999999e-07 2024-02-05T04:54:00.885 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.885 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.885 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:00.885 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.963206+0000", 2024-02-05T04:54:00.885 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0857999999999999e-05 2024-02-05T04:54:00.885 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.885 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.885 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:00.885 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.963216+0000", 2024-02-05T04:54:00.885 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0716999999999999e-05 2024-02-05T04:54:00.885 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.885 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.886 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:00.886 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.963228+0000", 2024-02-05T04:54:00.886 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1233000000000001e-05 2024-02-05T04:54:00.886 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.886 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.886 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:00.886 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.963280+0000", 2024-02-05T04:54:00.886 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2842e-05 2024-02-05T04:54:00.886 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.886 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.886 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:00.886 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.963291+0000", 2024-02-05T04:54:00.886 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0193e-05 2024-02-05T04:54:00.886 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.886 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.886 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:00.887 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.970447+0000", 2024-02-05T04:54:00.887 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0071561350000000001 2024-02-05T04:54:00.887 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.887 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.887 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:00.887 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.970484+0000", 2024-02-05T04:54:00.887 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7052999999999999e-05 2024-02-05T04:54:00.887 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.887 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:00.887 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.887 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.887 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.887 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.13s2 68/67 ECSubWrite(tid=132, reqid=osd.11.0:131, at_version=0'0, trim_to=0'0, roll_forward_to=68'41))", 2024-02-05T04:54:00.887 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:59.968186+0000", 2024-02-05T04:54:00.887 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.83437501800000002, 2024-02-05T04:54:00.888 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0054031460000000002, 2024-02-05T04:54:00.888 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:00.888 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:00.888 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:00.888 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.888 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:00.888 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.968186+0000", 2024-02-05T04:54:00.888 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:00.888 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.888 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.888 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:00.888 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.968186+0000", 2024-02-05T04:54:00.888 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:00.888 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.888 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.889 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:00.889 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.968186+0000", 2024-02-05T04:54:00.889 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9999999999999998e-07 2024-02-05T04:54:00.889 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.889 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.889 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:00.889 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.968196+0000", 2024-02-05T04:54:00.889 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5829999999999996e-06 2024-02-05T04:54:00.889 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.889 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.889 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:00.889 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.968205+0000", 2024-02-05T04:54:00.889 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5340000000000005e-06 2024-02-05T04:54:00.889 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.889 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.890 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:00.890 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.968213+0000", 2024-02-05T04:54:00.890 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0730000000000005e-06 2024-02-05T04:54:00.890 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.890 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.890 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:00.890 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.968295+0000", 2024-02-05T04:54:00.890 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.2050000000000002e-05 2024-02-05T04:54:00.890 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.890 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.890 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:00.890 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.968305+0000", 2024-02-05T04:54:00.890 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2900000000000008e-06 2024-02-05T04:54:00.890 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.890 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.891 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:00.891 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.973063+0000", 2024-02-05T04:54:00.891 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0047588159999999999 2024-02-05T04:54:00.891 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.891 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.891 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:00.891 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.973589+0000", 2024-02-05T04:54:00.891 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00052539999999999998 2024-02-05T04:54:00.891 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.891 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:00.891 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.891 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.891 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.891 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.12s6 68/67 ECSubWrite(tid=86, reqid=osd.2.0:85, at_version=0'0, trim_to=0'0, roll_forward_to=68'57))", 2024-02-05T04:54:00.892 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:59.970644+0000", 2024-02-05T04:54:00.892 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.83191654199999998, 2024-02-05T04:54:00.892 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0052138599999999998, 2024-02-05T04:54:00.892 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:00.892 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:00.892 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:00.892 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.892 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:00.892 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.970644+0000", 2024-02-05T04:54:00.892 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:00.892 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.892 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.892 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:00.893 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.970644+0000", 2024-02-05T04:54:00.893 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:00.893 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.893 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.893 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:00.893 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.970645+0000", 2024-02-05T04:54:00.893 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6300000000000005e-07 2024-02-05T04:54:00.893 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.893 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.893 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:00.893 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.970655+0000", 2024-02-05T04:54:00.893 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0489000000000001e-05 2024-02-05T04:54:00.893 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.893 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.893 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:00.894 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.970666+0000", 2024-02-05T04:54:00.894 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0352e-05 2024-02-05T04:54:00.894 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.894 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.894 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:00.894 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.970674+0000", 2024-02-05T04:54:00.894 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3899999999999993e-06 2024-02-05T04:54:00.894 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.894 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.894 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:00.894 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.970732+0000", 2024-02-05T04:54:00.894 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8034e-05 2024-02-05T04:54:00.895 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.895 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.895 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:00.895 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.970742+0000", 2024-02-05T04:54:00.895 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.815e-06 2024-02-05T04:54:00.895 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.895 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.895 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:00.895 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.975820+0000", 2024-02-05T04:54:00.895 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0050785960000000003 2024-02-05T04:54:00.895 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:00.895 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:00.895 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:00.895 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.975858+0000", 2024-02-05T04:54:00.895 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7521000000000002e-05 2024-02-05T04:54:00.896 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.896 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:00.896 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.896 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:00.896 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:00.896 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T04:54:00.897 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-02-05T04:54:00.898 INFO:tasks.ceph.osd.1.smithi091.stderr:2024-02-05T04:54:00.897+0000 7ff9dc283640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34275) UID: 0 2024-02-05T04:54:00.946 INFO:teuthology.orchestra.run.smithi062.stderr:reweighted osd.3 to 1 (10000) 2024-02-05T04:54:00.953 INFO:tasks.rados.rados.0.smithi196.stdout:249: done (1 left) 2024-02-05T04:54:00.953 INFO:tasks.rados.rados.0.smithi196.stdout:250: copy_from oid 29 from oid 27 current snap is 21 2024-02-05T04:54:00.953 INFO:tasks.rados.rados.0.smithi196.stdout:248: finishing rollback tid 0 to smithi19639362-21 2024-02-05T04:54:00.953 INFO:tasks.rados.rados.0.smithi196.stdout:248: finishing rollback tid 1 to smithi19639362-21 2024-02-05T04:54:00.953 INFO:tasks.rados.rados.0.smithi196.stdout:248: finishing rollback tid 2 to smithi19639362-21 2024-02-05T04:54:00.953 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 21 v 66 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-02-05T04:54:00.953 INFO:tasks.rados.rados.0.smithi196.stdout:248: done (1 left) 2024-02-05T04:54:00.953 INFO:tasks.rados.rados.0.smithi196.stdout:251: copy_from oid 19 from oid 50 current snap is 21 2024-02-05T04:54:00.953 INFO:tasks.rados.rados.0.smithi196.stdout:252: rmattr oid 39 current snap is 21 2024-02-05T04:54:00.953 INFO:tasks.rados.rados.0.smithi196.stdout:252: done (2 left) 2024-02-05T04:54:00.953 INFO:tasks.rados.rados.0.smithi196.stdout:append oid 15 current snap is 21 2024-02-05T04:54:00.953 INFO:tasks.rados.rados.0.smithi196.stdout:253: seq_num 115 ranges {0=770048,770048=704512,1474560=458752} 2024-02-05T04:54:00.956 INFO:tasks.rados.rados.0.smithi196.stdout:253: writing smithi19639362-15 from 0 to 770048 tid 1 2024-02-05T04:54:00.960 INFO:tasks.rados.rados.0.smithi196.stdout:253: writing smithi19639362-15 from 770048 to 1474560 tid 2 2024-02-05T04:54:00.964 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-02-05T04:54:00.965 INFO:tasks.rados.rados.0.smithi196.stdout:253: writing smithi19639362-15 from 1474560 to 1933312 tid 3 2024-02-05T04:54:00.965 INFO:tasks.rados.rados.0.smithi196.stdout:append oid 42 current snap is 21 2024-02-05T04:54:00.965 INFO:tasks.rados.rados.0.smithi196.stdout:254: seq_num 116 ranges {3293184=786432,4079616=458752,4538368=442368,4980736=262144} 2024-02-05T04:54:00.987 INFO:tasks.rados.rados.0.smithi196.stdout:254: writing smithi19639362-42 from 3293184 to 4079616 tid 1 2024-02-05T04:54:00.990 INFO:tasks.rados.rados.0.smithi196.stdout:254: writing smithi19639362-42 from 4079616 to 4538368 tid 2 2024-02-05T04:54:00.992 INFO:tasks.rados.rados.0.smithi196.stdout:254: writing smithi19639362-42 from 4538368 to 4980736 tid 3 2024-02-05T04:54:00.995 INFO:tasks.rados.rados.0.smithi196.stdout:254: writing smithi19639362-42 from 4980736 to 5242880 tid 4 2024-02-05T04:54:00.995 INFO:tasks.rados.rados.0.smithi196.stdout:255: delete oid 11 current snap is 21 2024-02-05T04:54:00.995 INFO:tasks.rados.rados.0.smithi196.stdout:251: finishing copy_from to smithi19639362-19 2024-02-05T04:54:00.995 INFO:tasks.rados.rados.0.smithi196.stdout:251: got expected ENOENT (src dne) 2024-02-05T04:54:00.995 INFO:tasks.rados.rados.0.smithi196.stdout:251: finishing copy_from racing read to smithi19639362-19 2024-02-05T04:54:00.995 INFO:tasks.rados.rados.0.smithi196.stdout:253: finishing write tid 1 to smithi19639362-15 2024-02-05T04:54:00.995 INFO:tasks.rados.rados.0.smithi196.stdout:253: finishing write tid 2 to smithi19639362-15 2024-02-05T04:54:00.996 INFO:tasks.rados.rados.0.smithi196.stdout:253: finishing write tid 3 to smithi19639362-15 2024-02-05T04:54:00.996 INFO:tasks.rados.rados.0.smithi196.stdout:253: finishing write tid 4 to smithi19639362-15 2024-02-05T04:54:00.996 INFO:tasks.rados.rados.0.smithi196.stdout:253: finishing write tid 5 to smithi19639362-15 2024-02-05T04:54:00.996 INFO:tasks.rados.rados.0.smithi196.stdout:253: oid 15 updating version 0 to 38 2024-02-05T04:54:00.996 INFO:tasks.rados.rados.0.smithi196.stdout:253: oid 15 updating version 38 to 41 2024-02-05T04:54:00.996 INFO:tasks.rados.rados.0.smithi196.stdout:253: oid 15 version 41 is already newer than 39 2024-02-05T04:54:00.996 INFO:tasks.rados.rados.0.smithi196.stdout:253: oid 15 version 41 is already newer than 40 2024-02-05T04:54:00.996 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 15 v 41 (ObjNum 115 snap 21 seq_num 115) dirty exists 2024-02-05T04:54:00.996 INFO:tasks.rados.rados.0.smithi196.stdout:253: left oid 15 (ObjNum 115 snap 21 seq_num 115) 2024-02-05T04:54:00.996 INFO:tasks.rados.rados.0.smithi196.stdout:250: finishing copy_from racing read to smithi19639362-29 2024-02-05T04:54:00.996 INFO:tasks.rados.rados.0.smithi196.stdout:250: finishing copy_from to smithi19639362-29 2024-02-05T04:54:00.996 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 29 v 25 (ObjNum 99 snap 11 seq_num 99) dirty exists 2024-02-05T04:54:00.996 INFO:tasks.rados.rados.0.smithi196.stdout:254: finishing write tid 1 to smithi19639362-42 2024-02-05T04:54:00.996 INFO:tasks.rados.rados.0.smithi196.stdout:254: finishing write tid 2 to smithi19639362-42 2024-02-05T04:54:00.997 INFO:tasks.ceph.osd.14.smithi150.stderr:2024-02-05T04:54:00.996+0000 7f5f81083640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 34020) UID: 0 2024-02-05T04:54:00.998 INFO:tasks.rados.rados.0.smithi196.stdout:254: finishing write tid 3 to smithi19639362-42 2024-02-05T04:54:00.998 INFO:tasks.rados.rados.0.smithi196.stdout:250: done (4 left) 2024-02-05T04:54:00.998 INFO:tasks.rados.rados.0.smithi196.stdout:251: done (3 left) 2024-02-05T04:54:00.998 INFO:tasks.rados.rados.0.smithi196.stdout:253: done (2 left) 2024-02-05T04:54:00.998 INFO:tasks.rados.rados.0.smithi196.stdout:255: done (1 left) 2024-02-05T04:54:00.998 INFO:tasks.rados.rados.0.smithi196.stdout:256: copy_from oid 23 from oid 40 current snap is 21 2024-02-05T04:54:00.998 INFO:tasks.rados.rados.0.smithi196.stdout:append oid 26 current snap is 21 2024-02-05T04:54:00.999 INFO:tasks.rados.rados.0.smithi196.stdout:257: seq_num 118 ranges {2572288=442368,3014656=753664,3768320=655360,4423680=147456} 2024-02-05T04:54:01.015 INFO:tasks.rados.rados.0.smithi196.stdout:257: writing smithi19639362-26 from 2572288 to 3014656 tid 1 2024-02-05T04:54:01.019 INFO:tasks.rados.rados.0.smithi196.stdout:257: writing smithi19639362-26 from 3014656 to 3768320 tid 2 2024-02-05T04:54:01.024 INFO:tasks.rados.rados.0.smithi196.stdout:257: writing smithi19639362-26 from 3768320 to 4423680 tid 3 2024-02-05T04:54:01.025 INFO:tasks.rados.rados.0.smithi196.stdout:257: writing smithi19639362-26 from 4423680 to 4571136 tid 4 2024-02-05T04:54:01.025 INFO:tasks.rados.rados.0.smithi196.stdout:258: snap_create 2024-02-05T04:54:01.025 INFO:tasks.rados.rados.0.smithi196.stdout:254: finishing write tid 4 to smithi19639362-42 2024-02-05T04:54:01.025 INFO:tasks.rados.rados.0.smithi196.stdout:254: finishing write tid 5 to smithi19639362-42 2024-02-05T04:54:01.026 INFO:tasks.rados.rados.0.smithi196.stdout:254: finishing write tid 6 to smithi19639362-42 2024-02-05T04:54:01.026 INFO:tasks.rados.rados.0.smithi196.stdout:254: oid 42 updating version 0 to 49 2024-02-05T04:54:01.026 INFO:tasks.rados.rados.0.smithi196.stdout:254: oid 42 updating version 49 to 50 2024-02-05T04:54:01.026 INFO:tasks.rados.rados.0.smithi196.stdout:254: oid 42 version 50 is already newer than 47 2024-02-05T04:54:01.026 INFO:tasks.rados.rados.0.smithi196.stdout:254: oid 42 version 50 is already newer than 48 2024-02-05T04:54:01.026 INFO:tasks.rados.rados.0.smithi196.stdout:254: oid 42 version 50 is already newer than 46 2024-02-05T04:54:01.026 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 42 v 50 (ObjNum 116 snap 21 seq_num 116) dirty exists 2024-02-05T04:54:01.026 INFO:tasks.rados.rados.0.smithi196.stdout:254: left oid 42 (ObjNum 116 snap 21 seq_num 116) 2024-02-05T04:54:01.026 INFO:tasks.rados.rados.0.smithi196.stdout:256: finishing copy_from racing read to smithi19639362-23 2024-02-05T04:54:01.026 INFO:tasks.rados.rados.0.smithi196.stdout:257: finishing write tid 1 to smithi19639362-26 2024-02-05T04:54:01.026 INFO:tasks.rados.rados.0.smithi196.stdout:256: finishing copy_from to smithi19639362-23 2024-02-05T04:54:01.026 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 23 v 29 (ObjNum 73 snap 3 seq_num 73) dirty exists 2024-02-05T04:54:01.027 INFO:tasks.rados.rados.0.smithi196.stdout:257: finishing write tid 2 to smithi19639362-26 2024-02-05T04:54:01.029 INFO:tasks.rados.rados.0.smithi196.stdout:257: finishing write tid 3 to smithi19639362-26 2024-02-05T04:54:01.033 INFO:tasks.rados.rados.0.smithi196.stdout:257: finishing write tid 4 to smithi19639362-26 2024-02-05T04:54:01.033 INFO:tasks.rados.rados.0.smithi196.stdout:257: finishing write tid 5 to smithi19639362-26 2024-02-05T04:54:01.034 INFO:tasks.rados.rados.0.smithi196.stdout:257: finishing write tid 6 to smithi19639362-26 2024-02-05T04:54:01.034 INFO:tasks.rados.rados.0.smithi196.stdout:257: oid 26 updating version 0 to 29 2024-02-05T04:54:01.034 INFO:tasks.rados.rados.0.smithi196.stdout:257: oid 26 updating version 29 to 31 2024-02-05T04:54:01.034 INFO:tasks.rados.rados.0.smithi196.stdout:257: oid 26 updating version 31 to 32 2024-02-05T04:54:01.034 INFO:tasks.rados.rados.0.smithi196.stdout:257: oid 26 version 32 is already newer than 30 2024-02-05T04:54:01.035 INFO:tasks.rados.rados.0.smithi196.stdout:257: oid 26 version 32 is already newer than 28 2024-02-05T04:54:01.035 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 26 v 32 (ObjNum 118 snap 21 seq_num 118) dirty exists 2024-02-05T04:54:01.035 INFO:tasks.rados.rados.0.smithi196.stdout:257: left oid 26 (ObjNum 118 snap 21 seq_num 118) 2024-02-05T04:54:01.035 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-02-05T04:54:01.035 INFO:teuthology.orchestra.run.smithi062.stdout: "ops": [], 2024-02-05T04:54:01.035 INFO:teuthology.orchestra.run.smithi062.stdout: "num_ops": 0 2024-02-05T04:54:01.035 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-02-05T04:54:01.044 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-02-05T04:54:01.099 INFO:tasks.ceph.osd.12.smithi062.stderr:2024-02-05T04:54:01.098+0000 7f5e5847e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 34695) UID: 0 2024-02-05T04:54:01.153 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-02-05T04:54:01.154 INFO:teuthology.orchestra.run.smithi062.stdout: "ops": [], 2024-02-05T04:54:01.154 INFO:teuthology.orchestra.run.smithi062.stdout: "complaint_time": 30, 2024-02-05T04:54:01.154 INFO:teuthology.orchestra.run.smithi062.stdout: "num_blocked_ops": 0 2024-02-05T04:54:01.154 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-02-05T04:54:01.162 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-02-05T04:54:01.204 INFO:tasks.ceph.osd.8.smithi062.stderr:2024-02-05T04:54:01.203+0000 7f9385bf2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 34693) UID: 0 2024-02-05T04:54:01.301 INFO:tasks.ceph.osd.4.smithi062.stderr:2024-02-05T04:54:01.300+0000 7f708c308640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34689) UID: 0 2024-02-05T04:54:01.312 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-02-05T04:54:01.312 INFO:teuthology.orchestra.run.smithi062.stdout: "size": 20, 2024-02-05T04:54:01.312 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 600, 2024-02-05T04:54:01.312 INFO:teuthology.orchestra.run.smithi062.stdout: "ops": [ 2024-02-05T04:54:01.312 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.312 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.bs3 43/31 ECSubWrite(tid=139, reqid=osd.13.0:137, at_version=0'0, trim_to=0'0, roll_forward_to=43'28))", 2024-02-05T04:54:01.312 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:35.180865+0000", 2024-02-05T04:54:01.312 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 26.129505993999999, 2024-02-05T04:54:01.312 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0037505139999999999, 2024-02-05T04:54:01.312 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:01.312 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:54:01.313 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:01.313 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.313 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:01.313 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.180865+0000", 2024-02-05T04:54:01.313 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:01.313 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.313 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.313 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:01.313 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.180865+0000", 2024-02-05T04:54:01.313 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:01.313 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.313 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.313 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:01.313 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.180865+0000", 2024-02-05T04:54:01.313 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.0200000000000001e-07 2024-02-05T04:54:01.313 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.314 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.314 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:01.314 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.180875+0000", 2024-02-05T04:54:01.314 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.8959999999999998e-06 2024-02-05T04:54:01.314 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.314 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.314 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:01.314 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.180892+0000", 2024-02-05T04:54:01.314 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.6531000000000001e-05 2024-02-05T04:54:01.314 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.314 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.314 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:01.314 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.180901+0000", 2024-02-05T04:54:01.314 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.2890000000000003e-06 2024-02-05T04:54:01.314 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.314 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.315 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:01.315 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.180961+0000", 2024-02-05T04:54:01.315 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.0012000000000001e-05 2024-02-05T04:54:01.315 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.315 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.315 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:01.315 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.180971+0000", 2024-02-05T04:54:01.315 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0190999999999999e-05 2024-02-05T04:54:01.315 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.315 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.315 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:01.315 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.184573+0000", 2024-02-05T04:54:01.315 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00360181 2024-02-05T04:54:01.316 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.316 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.316 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:01.316 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.184615+0000", 2024-02-05T04:54:01.316 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.1983000000000001e-05 2024-02-05T04:54:01.316 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:01.316 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:01.316 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:01.316 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.317 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.317 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.bs3 52/31 ECSubWrite(tid=200, reqid=osd.13.0:200, at_version=52'50, trim_to=0'0, roll_forward_to=51'48))", 2024-02-05T04:54:01.317 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:43.953702+0000", 2024-02-05T04:54:01.317 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 17.356668983999999, 2024-02-05T04:54:01.317 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0050156649999999999, 2024-02-05T04:54:01.317 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:01.317 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:54:01.317 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:01.317 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.317 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:01.317 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.953702+0000", 2024-02-05T04:54:01.317 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:01.317 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.317 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.317 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:01.318 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.953702+0000", 2024-02-05T04:54:01.318 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:01.318 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.318 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.318 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:01.318 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.953702+0000", 2024-02-05T04:54:01.318 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.11e-07 2024-02-05T04:54:01.318 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.318 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.318 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:01.318 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.953721+0000", 2024-02-05T04:54:01.318 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.8932000000000001e-05 2024-02-05T04:54:01.318 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.318 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.318 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:01.318 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.953732+0000", 2024-02-05T04:54:01.319 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1564e-05 2024-02-05T04:54:01.319 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.319 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.319 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:01.319 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.953741+0000", 2024-02-05T04:54:01.319 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.2050000000000008e-06 2024-02-05T04:54:01.319 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.319 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.319 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:01.319 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.953801+0000", 2024-02-05T04:54:01.319 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.9344e-05 2024-02-05T04:54:01.319 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.319 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.319 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:01.319 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.953812+0000", 2024-02-05T04:54:01.320 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0845e-05 2024-02-05T04:54:01.320 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.320 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.320 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:01.320 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.958673+0000", 2024-02-05T04:54:01.320 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0048608640000000003 2024-02-05T04:54:01.320 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.320 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.320 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:01.320 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.958717+0000", 2024-02-05T04:54:01.321 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.4700000000000002e-05 2024-02-05T04:54:01.321 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:01.321 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:01.321 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:01.321 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.321 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.321 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.fs4 52/31 ECSubWrite(tid=85, reqid=osd.11.0:85, at_version=52'40, trim_to=0'0, roll_forward_to=51'36))", 2024-02-05T04:54:01.321 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:43.954702+0000", 2024-02-05T04:54:01.321 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 17.355668832999999, 2024-02-05T04:54:01.321 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.005090982, 2024-02-05T04:54:01.321 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:01.321 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:54:01.321 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:01.321 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.322 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:01.322 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.954702+0000", 2024-02-05T04:54:01.322 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:01.322 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.322 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.322 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:01.322 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.954702+0000", 2024-02-05T04:54:01.322 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:01.322 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.322 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.322 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:01.322 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.954702+0000", 2024-02-05T04:54:01.322 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.9999999999999997e-07 2024-02-05T04:54:01.322 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.322 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.322 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:01.323 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.954717+0000", 2024-02-05T04:54:01.323 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.4905e-05 2024-02-05T04:54:01.323 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.323 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.323 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:01.323 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.954737+0000", 2024-02-05T04:54:01.323 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.9578000000000001e-05 2024-02-05T04:54:01.323 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.323 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.323 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:01.323 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.954745+0000", 2024-02-05T04:54:01.323 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.8299999999999996e-06 2024-02-05T04:54:01.323 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.323 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.323 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:01.323 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.954820+0000", 2024-02-05T04:54:01.324 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.4832000000000005e-05 2024-02-05T04:54:01.325 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.325 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.325 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:01.325 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.954827+0000", 2024-02-05T04:54:01.325 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.9349999999999997e-06 2024-02-05T04:54:01.325 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.325 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.325 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:01.325 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.959760+0000", 2024-02-05T04:54:01.325 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0049331669999999996 2024-02-05T04:54:01.325 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.325 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.325 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:01.326 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.959793+0000", 2024-02-05T04:54:01.326 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.3034999999999998e-05 2024-02-05T04:54:01.326 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:01.326 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:01.326 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:01.326 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.327 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.327 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.9s1 52/31 ECSubWrite(tid=44, reqid=osd.14.0:44, at_version=52'19, trim_to=0'0, roll_forward_to=51'17))", 2024-02-05T04:54:01.327 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:43.957858+0000", 2024-02-05T04:54:01.327 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 17.352512340000001, 2024-02-05T04:54:01.327 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0041238409999999996, 2024-02-05T04:54:01.327 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:01.327 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:54:01.327 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:01.327 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.327 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:01.327 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.957858+0000", 2024-02-05T04:54:01.327 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:01.327 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.328 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.328 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:01.328 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.957858+0000", 2024-02-05T04:54:01.328 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:01.328 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.328 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.328 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:01.328 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.957859+0000", 2024-02-05T04:54:01.328 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.1800000000000002e-07 2024-02-05T04:54:01.328 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.328 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.328 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:01.328 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.957886+0000", 2024-02-05T04:54:01.328 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.7733e-05 2024-02-05T04:54:01.329 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.329 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.329 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:01.329 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.957896+0000", 2024-02-05T04:54:01.329 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.5780000000000005e-06 2024-02-05T04:54:01.329 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.329 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.329 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:01.329 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.957901+0000", 2024-02-05T04:54:01.329 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.5949999999999998e-06 2024-02-05T04:54:01.329 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.329 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.329 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:01.329 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.957958+0000", 2024-02-05T04:54:01.329 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.6323999999999998e-05 2024-02-05T04:54:01.330 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.330 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.330 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:01.330 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.957968+0000", 2024-02-05T04:54:01.330 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0093e-05 2024-02-05T04:54:01.330 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.330 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.330 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:01.330 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.961950+0000", 2024-02-05T04:54:01.330 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0039823269999999999 2024-02-05T04:54:01.330 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.330 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.330 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:01.330 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.961982+0000", 2024-02-05T04:54:01.330 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.1872999999999998e-05 2024-02-05T04:54:01.331 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:01.331 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:01.331 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:01.331 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.331 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.331 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.bs3 52/31 ECSubWrite(tid=202, reqid=osd.13.0:200, at_version=0'0, trim_to=0'0, roll_forward_to=52'50))", 2024-02-05T04:54:01.331 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:43.959201+0000", 2024-02-05T04:54:01.331 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 17.351169260999999, 2024-02-05T04:54:01.331 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0055030230000000001, 2024-02-05T04:54:01.331 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:01.331 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:54:01.331 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:01.331 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.331 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:01.331 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.959201+0000", 2024-02-05T04:54:01.332 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:01.332 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.332 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.332 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:01.332 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.959201+0000", 2024-02-05T04:54:01.332 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:01.332 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.332 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.332 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:01.332 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.959202+0000", 2024-02-05T04:54:01.332 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.4899999999999995e-07 2024-02-05T04:54:01.332 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.332 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.332 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:01.332 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.959211+0000", 2024-02-05T04:54:01.333 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.4350000000000003e-06 2024-02-05T04:54:01.333 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.333 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.333 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:01.333 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.959221+0000", 2024-02-05T04:54:01.333 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.8700000000000004e-06 2024-02-05T04:54:01.333 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.333 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.333 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:01.333 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.959227+0000", 2024-02-05T04:54:01.333 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.1349999999999997e-06 2024-02-05T04:54:01.333 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.333 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.333 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:01.333 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.959613+0000", 2024-02-05T04:54:01.333 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00038529800000000001 2024-02-05T04:54:01.334 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.334 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.334 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:01.334 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.959622+0000", 2024-02-05T04:54:01.334 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.4339999999999998e-06 2024-02-05T04:54:01.334 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.334 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.334 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:01.334 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.964670+0000", 2024-02-05T04:54:01.334 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.005047719 2024-02-05T04:54:01.334 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.334 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.334 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:01.334 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.964704+0000", 2024-02-05T04:54:01.334 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.4483000000000002e-05 2024-02-05T04:54:01.334 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:01.335 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:01.335 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:01.335 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.335 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.335 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.fs4 52/31 ECSubWrite(tid=87, reqid=osd.11.0:85, at_version=0'0, trim_to=0'0, roll_forward_to=52'40))", 2024-02-05T04:54:01.335 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:43.961973+0000", 2024-02-05T04:54:01.335 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 17.348397644999999, 2024-02-05T04:54:01.335 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0052598189999999998, 2024-02-05T04:54:01.335 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:01.335 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:54:01.335 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:01.335 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.335 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:01.335 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.961973+0000", 2024-02-05T04:54:01.335 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:01.336 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.336 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.336 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:01.336 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.961973+0000", 2024-02-05T04:54:01.336 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:01.336 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.336 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.336 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:01.336 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.961974+0000", 2024-02-05T04:54:01.336 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.9599999999999999e-07 2024-02-05T04:54:01.337 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.337 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.337 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:01.337 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.961983+0000", 2024-02-05T04:54:01.337 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.7270000000000002e-06 2024-02-05T04:54:01.337 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.337 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.337 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:01.337 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.961994+0000", 2024-02-05T04:54:01.337 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0778000000000001e-05 2024-02-05T04:54:01.337 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.337 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.337 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:01.337 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.962002+0000", 2024-02-05T04:54:01.338 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.5299999999999999e-06 2024-02-05T04:54:01.338 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.338 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.338 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:01.338 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.962052+0000", 2024-02-05T04:54:01.338 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.0080000000000002e-05 2024-02-05T04:54:01.338 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.338 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.338 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:01.338 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.962061+0000", 2024-02-05T04:54:01.338 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.8100000000000004e-06 2024-02-05T04:54:01.338 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.338 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.338 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:01.338 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.967197+0000", 2024-02-05T04:54:01.338 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0051366930000000003 2024-02-05T04:54:01.339 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.339 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.339 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:01.339 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.967233+0000", 2024-02-05T04:54:01.339 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.5505e-05 2024-02-05T04:54:01.339 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:01.339 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:01.339 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:01.339 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.339 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.339 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.9s1 52/31 ECSubWrite(tid=45, reqid=osd.14.0:44, at_version=0'0, trim_to=0'0, roll_forward_to=52'19))", 2024-02-05T04:54:01.339 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:43.962534+0000", 2024-02-05T04:54:01.339 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 17.347836617999999, 2024-02-05T04:54:01.339 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0047221670000000002, 2024-02-05T04:54:01.339 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:01.339 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:54:01.340 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:01.340 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.340 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:01.340 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.962534+0000", 2024-02-05T04:54:01.340 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:01.340 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.340 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.340 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:01.340 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.962534+0000", 2024-02-05T04:54:01.340 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:01.340 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.340 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.340 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:01.340 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.962534+0000", 2024-02-05T04:54:01.341 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.0999999999999999e-07 2024-02-05T04:54:01.341 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.341 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.341 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:01.341 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.962543+0000", 2024-02-05T04:54:01.341 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.4889999999999995e-06 2024-02-05T04:54:01.341 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.341 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.341 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:01.341 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.962554+0000", 2024-02-05T04:54:01.341 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0576e-05 2024-02-05T04:54:01.341 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.341 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.341 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:01.341 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.962562+0000", 2024-02-05T04:54:01.341 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.9150000000000002e-06 2024-02-05T04:54:01.342 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.342 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.342 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:01.342 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.962614+0000", 2024-02-05T04:54:01.342 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.1167e-05 2024-02-05T04:54:01.342 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.342 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.342 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:01.342 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.962623+0000", 2024-02-05T04:54:01.342 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.1670000000000005e-06 2024-02-05T04:54:01.342 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.342 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.342 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:01.342 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.967222+0000", 2024-02-05T04:54:01.342 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.004598787 2024-02-05T04:54:01.342 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.343 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.343 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:01.343 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.967256+0000", 2024-02-05T04:54:01.343 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.4555999999999999e-05 2024-02-05T04:54:01.343 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:01.343 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:01.343 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:01.343 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.343 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.343 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.fs4 54/31 ECSubWrite(tid=88, reqid=client.4634.0:507, at_version=54'42, trim_to=0'0, roll_forward_to=52'40))", 2024-02-05T04:54:01.343 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:45.962031+0000", 2024-02-05T04:54:01.343 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 15.348339073, 2024-02-05T04:54:01.343 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0046418570000000001, 2024-02-05T04:54:01.343 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:01.343 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:54:01.344 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:01.344 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.344 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:01.344 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:45.962031+0000", 2024-02-05T04:54:01.344 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:01.344 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.344 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.344 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:01.344 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:45.962031+0000", 2024-02-05T04:54:01.344 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:01.344 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.344 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.344 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:01.344 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:45.962032+0000", 2024-02-05T04:54:01.344 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.99e-07 2024-02-05T04:54:01.344 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.345 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.345 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:01.345 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:45.962058+0000", 2024-02-05T04:54:01.345 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.6069999999999999e-05 2024-02-05T04:54:01.345 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.345 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.345 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:01.345 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:45.962072+0000", 2024-02-05T04:54:01.345 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.4423000000000001e-05 2024-02-05T04:54:01.345 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.345 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.345 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:01.345 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:45.962084+0000", 2024-02-05T04:54:01.345 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.2150000000000001e-05 2024-02-05T04:54:01.345 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.346 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.346 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:01.346 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:45.962145+0000", 2024-02-05T04:54:01.346 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.0795999999999999e-05 2024-02-05T04:54:01.346 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.346 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.346 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:01.346 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:45.962158+0000", 2024-02-05T04:54:01.346 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.2801e-05 2024-02-05T04:54:01.346 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.346 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.346 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:01.346 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:45.966640+0000", 2024-02-05T04:54:01.346 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0044822799999999999 2024-02-05T04:54:01.346 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.346 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.347 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:01.347 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:45.966673+0000", 2024-02-05T04:54:01.347 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.3138e-05 2024-02-05T04:54:01.347 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:01.347 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:01.347 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:01.347 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.347 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.347 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpRead(3.bs3 58/57 ECSubRead(tid=242, to_read={3:df6528f1:::smithi19639362-33:head=0,532480,0}, subchunks={3:df6528f1:::smithi19639362-33:head=[0,1]}, attrs_to_read=))", 2024-02-05T04:54:01.347 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:49.955701+0000", 2024-02-05T04:54:01.347 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 11.354669177, 2024-02-05T04:54:01.347 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0063665620000000001, 2024-02-05T04:54:01.347 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:01.347 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:54:01.347 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:01.348 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.348 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:01.348 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.955701+0000", 2024-02-05T04:54:01.348 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:01.348 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.348 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.348 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:01.348 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.955701+0000", 2024-02-05T04:54:01.348 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:01.348 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.348 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.348 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:01.348 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.955702+0000", 2024-02-05T04:54:01.348 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.0900000000000001e-07 2024-02-05T04:54:01.348 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.348 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.349 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:01.349 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.955710+0000", 2024-02-05T04:54:01.349 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.2220000000000002e-06 2024-02-05T04:54:01.349 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.349 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.349 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:01.349 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.955732+0000", 2024-02-05T04:54:01.349 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.2018000000000002e-05 2024-02-05T04:54:01.349 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.349 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.349 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:01.349 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.955745+0000", 2024-02-05T04:54:01.349 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.3025e-05 2024-02-05T04:54:01.349 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.349 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.349 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:01.350 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.955862+0000", 2024-02-05T04:54:01.350 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00011653400000000001 2024-02-05T04:54:01.350 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.350 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.350 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:01.350 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.962068+0000", 2024-02-05T04:54:01.350 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0062060539999999999 2024-02-05T04:54:01.350 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:01.350 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:01.350 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:01.350 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.350 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.350 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.4s3 58/57 ECSubWrite(tid=107, reqid=osd.1.0:107, at_version=58'59, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:01.350 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:49.956038+0000", 2024-02-05T04:54:01.350 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 11.354332318999999, 2024-02-05T04:54:01.351 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.005678163, 2024-02-05T04:54:01.351 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:01.351 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:54:01.351 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:01.351 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.351 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:01.351 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956038+0000", 2024-02-05T04:54:01.351 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:01.351 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.351 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.351 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:01.351 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956038+0000", 2024-02-05T04:54:01.351 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:01.351 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.351 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.351 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:01.352 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956039+0000", 2024-02-05T04:54:01.352 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.2200000000000003e-07 2024-02-05T04:54:01.352 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.352 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.352 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:01.352 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956068+0000", 2024-02-05T04:54:01.352 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.9013000000000002e-05 2024-02-05T04:54:01.352 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.352 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.352 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:01.352 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956082+0000", 2024-02-05T04:54:01.352 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.4066999999999999e-05 2024-02-05T04:54:01.352 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.352 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.352 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:01.352 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956091+0000", 2024-02-05T04:54:01.353 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.9539999999999993e-06 2024-02-05T04:54:01.353 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.353 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.353 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:01.353 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956177+0000", 2024-02-05T04:54:01.353 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.5548000000000002e-05 2024-02-05T04:54:01.353 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.353 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.353 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:01.353 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956187+0000", 2024-02-05T04:54:01.353 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0352e-05 2024-02-05T04:54:01.353 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.353 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.353 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:01.353 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.961666+0000", 2024-02-05T04:54:01.354 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0054792979999999996 2024-02-05T04:54:01.354 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.354 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.354 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:01.354 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.961716+0000", 2024-02-05T04:54:01.354 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.0108999999999997e-05 2024-02-05T04:54:01.354 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:01.354 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:01.354 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:01.354 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.354 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.354 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.4s3 58/57 ECSubWrite(tid=108, reqid=osd.1.0:108, at_version=58'61, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:01.354 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:49.956139+0000", 2024-02-05T04:54:01.354 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 11.354231744, 2024-02-05T04:54:01.354 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0079487310000000005, 2024-02-05T04:54:01.354 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:01.355 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:54:01.355 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:01.355 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.355 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:01.355 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956139+0000", 2024-02-05T04:54:01.355 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:01.355 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.355 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.355 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:01.355 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956139+0000", 2024-02-05T04:54:01.355 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:01.355 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.355 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.355 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:01.355 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956139+0000", 2024-02-05T04:54:01.355 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.0999999999999999e-07 2024-02-05T04:54:01.356 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.356 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.356 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:01.356 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956165+0000", 2024-02-05T04:54:01.356 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.5202999999999999e-05 2024-02-05T04:54:01.356 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.356 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.356 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:01.356 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956176+0000", 2024-02-05T04:54:01.356 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1523e-05 2024-02-05T04:54:01.356 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.356 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.356 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:01.356 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956184+0000", 2024-02-05T04:54:01.356 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.0940000000000008e-06 2024-02-05T04:54:01.357 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.357 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.357 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:01.357 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956668+0000", 2024-02-05T04:54:01.357 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000483535 2024-02-05T04:54:01.357 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.357 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.357 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:01.357 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956681+0000", 2024-02-05T04:54:01.357 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.2958e-05 2024-02-05T04:54:01.357 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.357 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.358 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:01.358 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.964043+0000", 2024-02-05T04:54:01.358 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0073623769999999998 2024-02-05T04:54:01.358 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.358 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.358 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:01.358 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.964088+0000", 2024-02-05T04:54:01.358 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.4530999999999998e-05 2024-02-05T04:54:01.358 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:01.358 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:01.358 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:01.358 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.358 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.358 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpRead(3.bs3 58/57 ECSubRead(tid=243, to_read={3:db93bf34:::smithi19639362-24:head=0,98304,0}, subchunks={3:db93bf34:::smithi19639362-24:head=[0,1]}, attrs_to_read=))", 2024-02-05T04:54:01.358 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:49.956639+0000", 2024-02-05T04:54:01.359 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 11.353731079999999, 2024-02-05T04:54:01.359 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0056018429999999996, 2024-02-05T04:54:01.359 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:01.359 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:54:01.359 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:01.359 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.359 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:01.359 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956639+0000", 2024-02-05T04:54:01.359 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:01.359 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.359 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.359 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:01.359 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956639+0000", 2024-02-05T04:54:01.359 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:01.359 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.359 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.360 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:01.360 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956640+0000", 2024-02-05T04:54:01.360 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.6800000000000009e-07 2024-02-05T04:54:01.360 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.360 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.360 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:01.360 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956650+0000", 2024-02-05T04:54:01.360 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.9739999999999997e-06 2024-02-05T04:54:01.360 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.360 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.360 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:01.360 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956681+0000", 2024-02-05T04:54:01.360 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.0655999999999999e-05 2024-02-05T04:54:01.360 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.360 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.361 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:01.361 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956707+0000", 2024-02-05T04:54:01.361 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.5469999999999998e-05 2024-02-05T04:54:01.361 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.361 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.361 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:01.361 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.962076+0000", 2024-02-05T04:54:01.361 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0053699300000000002 2024-02-05T04:54:01.361 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.361 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.361 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:01.361 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.962241+0000", 2024-02-05T04:54:01.361 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000164845 2024-02-05T04:54:01.361 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:01.361 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:01.361 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:01.362 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.362 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.362 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.bs3 58/57 ECSubWrite(tid=251, reqid=osd.13.0:249, at_version=0'0, trim_to=0'0, roll_forward_to=58'52))", 2024-02-05T04:54:01.362 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:49.968190+0000", 2024-02-05T04:54:01.362 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 11.342180625999999, 2024-02-05T04:54:01.362 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0043444590000000002, 2024-02-05T04:54:01.362 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:01.362 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:54:01.362 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:01.362 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.362 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:01.362 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.968190+0000", 2024-02-05T04:54:01.362 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:01.362 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.362 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.363 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:01.363 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.968190+0000", 2024-02-05T04:54:01.363 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:01.363 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.363 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.363 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:01.363 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.968191+0000", 2024-02-05T04:54:01.363 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.4899999999999995e-07 2024-02-05T04:54:01.363 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.363 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.363 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:01.363 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.968199+0000", 2024-02-05T04:54:01.363 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.3480000000000004e-06 2024-02-05T04:54:01.363 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.363 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.363 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:01.364 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.968211+0000", 2024-02-05T04:54:01.364 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1812e-05 2024-02-05T04:54:01.364 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.364 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.364 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:01.364 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.968217+0000", 2024-02-05T04:54:01.364 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.3210000000000001e-06 2024-02-05T04:54:01.364 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.364 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.364 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:01.364 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.968273+0000", 2024-02-05T04:54:01.364 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.5856999999999997e-05 2024-02-05T04:54:01.364 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.364 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.364 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:01.365 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.968284+0000", 2024-02-05T04:54:01.365 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1112e-05 2024-02-05T04:54:01.365 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.365 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.365 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:01.365 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.972492+0000", 2024-02-05T04:54:01.365 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0042084260000000004 2024-02-05T04:54:01.365 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.365 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.365 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:01.365 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.972534+0000", 2024-02-05T04:54:01.365 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.1934000000000002e-05 2024-02-05T04:54:01.365 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:01.365 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:01.365 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:01.365 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.366 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.366 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.bs3 58/57 ECSubWrite(tid=252, reqid=osd.13.0:252, at_version=58'54, trim_to=0'0, roll_forward_to=58'52))", 2024-02-05T04:54:01.366 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:49.968817+0000", 2024-02-05T04:54:01.366 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 11.341554008999999, 2024-02-05T04:54:01.366 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0037491209999999998, 2024-02-05T04:54:01.366 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:01.366 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:54:01.366 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:01.366 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.366 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:01.366 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.968817+0000", 2024-02-05T04:54:01.366 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:01.366 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.366 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.366 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:01.367 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.968817+0000", 2024-02-05T04:54:01.367 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:01.367 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.367 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.367 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:01.367 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.968817+0000", 2024-02-05T04:54:01.367 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.5000000000000002e-07 2024-02-05T04:54:01.367 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.367 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.367 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:01.367 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.968835+0000", 2024-02-05T04:54:01.367 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.7833999999999999e-05 2024-02-05T04:54:01.367 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.367 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.367 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:01.367 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.968846+0000", 2024-02-05T04:54:01.368 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1093e-05 2024-02-05T04:54:01.368 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.368 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.368 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:01.368 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.968859+0000", 2024-02-05T04:54:01.368 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.2690999999999999e-05 2024-02-05T04:54:01.368 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.368 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.368 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:01.368 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.968921+0000", 2024-02-05T04:54:01.368 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.2593000000000002e-05 2024-02-05T04:54:01.368 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.368 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.368 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:01.368 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.968933+0000", 2024-02-05T04:54:01.369 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1212e-05 2024-02-05T04:54:01.369 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.369 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.369 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:01.369 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.972538+0000", 2024-02-05T04:54:01.369 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.003605811 2024-02-05T04:54:01.369 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.369 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.369 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:01.369 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.972566+0000", 2024-02-05T04:54:01.369 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.7237e-05 2024-02-05T04:54:01.369 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:01.369 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:01.369 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:01.369 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.369 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.370 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.fs4 58/31 ECSubWrite(tid=104, reqid=client.4634.0:518, at_version=58'51, trim_to=0'0, roll_forward_to=57'49))", 2024-02-05T04:54:01.370 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:49.973886+0000", 2024-02-05T04:54:01.370 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 11.336484966, 2024-02-05T04:54:01.370 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.004459373, 2024-02-05T04:54:01.370 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:01.370 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:54:01.370 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:01.370 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.370 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:01.370 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.973886+0000", 2024-02-05T04:54:01.370 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:01.370 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.370 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.370 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:01.370 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.973886+0000", 2024-02-05T04:54:01.370 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:01.371 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.371 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.371 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:01.371 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.973886+0000", 2024-02-05T04:54:01.371 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.8100000000000001e-07 2024-02-05T04:54:01.371 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.371 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.371 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:01.371 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.975693+0000", 2024-02-05T04:54:01.371 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0018069480000000001 2024-02-05T04:54:01.371 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.371 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.371 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:01.371 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.975711+0000", 2024-02-05T04:54:01.371 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.7669e-05 2024-02-05T04:54:01.372 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.372 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.372 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:01.372 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.975721+0000", 2024-02-05T04:54:01.372 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.8649999999999996e-06 2024-02-05T04:54:01.372 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.372 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.372 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:01.372 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.975801+0000", 2024-02-05T04:54:01.372 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.0357000000000003e-05 2024-02-05T04:54:01.372 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.372 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.372 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:01.372 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.975813+0000", 2024-02-05T04:54:01.372 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1590000000000001e-05 2024-02-05T04:54:01.372 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.373 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.373 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:01.373 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.978304+0000", 2024-02-05T04:54:01.373 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0024906350000000002 2024-02-05T04:54:01.373 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.373 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.373 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:01.373 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.978345+0000", 2024-02-05T04:54:01.373 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.1427999999999997e-05 2024-02-05T04:54:01.373 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:01.373 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:01.373 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:01.373 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.373 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.373 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.0s3 63/62 ECSubWrite(tid=52, reqid=client.4634.0:576, at_version=63'6, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:01.373 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:54.999782+0000", 2024-02-05T04:54:01.374 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 6.3105887249999997, 2024-02-05T04:54:01.374 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0042417959999999999, 2024-02-05T04:54:01.374 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:01.374 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:54:01.374 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:01.374 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.374 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:01.374 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.999782+0000", 2024-02-05T04:54:01.374 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:01.374 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.374 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.374 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:01.374 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.999782+0000", 2024-02-05T04:54:01.374 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:01.374 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.374 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.375 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:01.375 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.999782+0000", 2024-02-05T04:54:01.375 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.0800000000000004e-07 2024-02-05T04:54:01.375 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.375 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.375 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:01.375 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:55.000297+0000", 2024-02-05T04:54:01.375 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00051424900000000004 2024-02-05T04:54:01.375 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.375 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.375 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:01.375 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:55.000308+0000", 2024-02-05T04:54:01.375 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0964e-05 2024-02-05T04:54:01.375 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.375 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.375 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:01.376 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:55.000316+0000", 2024-02-05T04:54:01.376 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.6179999999999995e-06 2024-02-05T04:54:01.376 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.376 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.376 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:01.376 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:55.000377+0000", 2024-02-05T04:54:01.376 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.0497e-05 2024-02-05T04:54:01.376 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.376 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.376 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:01.376 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:55.000386+0000", 2024-02-05T04:54:01.376 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.1940000000000004e-06 2024-02-05T04:54:01.376 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.376 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.376 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:01.376 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:55.003988+0000", 2024-02-05T04:54:01.377 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0036023549999999998 2024-02-05T04:54:01.377 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.377 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.377 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:01.377 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:55.004024+0000", 2024-02-05T04:54:01.377 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.5311000000000001e-05 2024-02-05T04:54:01.377 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:01.377 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:01.377 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:01.377 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.377 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.377 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.14s3 68/67 ECSubWrite(tid=15, reqid=osd.3.0:15, at_version=68'75, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:01.377 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:59.961574+0000", 2024-02-05T04:54:01.377 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 1.348796345, 2024-02-05T04:54:01.377 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0037465710000000002, 2024-02-05T04:54:01.377 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:01.378 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:54:01.378 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:01.378 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.378 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:01.378 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.961574+0000", 2024-02-05T04:54:01.378 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:01.378 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.378 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.378 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:01.378 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.961574+0000", 2024-02-05T04:54:01.378 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:01.378 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.378 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.378 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:01.378 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.961575+0000", 2024-02-05T04:54:01.379 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.1399999999999996e-07 2024-02-05T04:54:01.379 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.379 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.379 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:01.379 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.961605+0000", 2024-02-05T04:54:01.379 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.0338999999999999e-05 2024-02-05T04:54:01.379 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.379 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.379 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:01.379 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.961621+0000", 2024-02-05T04:54:01.379 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.6118000000000001e-05 2024-02-05T04:54:01.379 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.379 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.379 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:01.379 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.961633+0000", 2024-02-05T04:54:01.380 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1476e-05 2024-02-05T04:54:01.380 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.380 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.380 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:01.380 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.961689+0000", 2024-02-05T04:54:01.380 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.5543e-05 2024-02-05T04:54:01.380 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.380 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.380 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:01.380 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.961700+0000", 2024-02-05T04:54:01.380 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1768e-05 2024-02-05T04:54:01.380 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.380 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.380 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:01.380 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.965280+0000", 2024-02-05T04:54:01.381 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0035801930000000002 2024-02-05T04:54:01.381 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.381 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.381 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:01.381 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.965321+0000", 2024-02-05T04:54:01.381 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.032e-05 2024-02-05T04:54:01.381 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:01.381 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:01.381 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:01.381 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.381 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.381 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.14s3 68/67 ECSubWrite(tid=16, reqid=osd.3.0:16, at_version=68'77, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:01.381 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:59.962945+0000", 2024-02-05T04:54:01.381 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 1.3474257439999999, 2024-02-05T04:54:01.381 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0053159289999999996, 2024-02-05T04:54:01.381 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:01.382 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:54:01.382 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:01.382 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.382 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:01.382 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.962945+0000", 2024-02-05T04:54:01.382 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:01.382 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.382 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.382 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:01.382 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.962945+0000", 2024-02-05T04:54:01.382 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:01.382 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.382 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.382 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:01.382 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.962946+0000", 2024-02-05T04:54:01.382 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.7599999999999996e-07 2024-02-05T04:54:01.383 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.383 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.383 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:01.383 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.962973+0000", 2024-02-05T04:54:01.383 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.6835e-05 2024-02-05T04:54:01.383 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.383 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.383 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:01.383 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.962987+0000", 2024-02-05T04:54:01.383 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.4788e-05 2024-02-05T04:54:01.383 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.383 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.384 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:01.384 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.962997+0000", 2024-02-05T04:54:01.384 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0169999999999999e-05 2024-02-05T04:54:01.384 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.384 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.384 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:01.384 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.963083+0000", 2024-02-05T04:54:01.384 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.5257e-05 2024-02-05T04:54:01.384 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.384 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.384 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:01.384 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.963097+0000", 2024-02-05T04:54:01.384 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.4477999999999999e-05 2024-02-05T04:54:01.384 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.384 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.385 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:01.385 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.968227+0000", 2024-02-05T04:54:01.385 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0051300190000000004 2024-02-05T04:54:01.385 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.385 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.385 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:01.385 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.968261+0000", 2024-02-05T04:54:01.385 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.3506000000000001e-05 2024-02-05T04:54:01.385 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:01.385 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:01.385 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:01.385 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.385 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.385 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.14s3 68/67 ECSubWrite(tid=17, reqid=osd.3.0:16, at_version=0'0, trim_to=0'0, roll_forward_to=68'77))", 2024-02-05T04:54:01.385 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:59.974045+0000", 2024-02-05T04:54:01.385 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 1.336325239, 2024-02-05T04:54:01.386 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0081087980000000004, 2024-02-05T04:54:01.386 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:01.386 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:54:01.386 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:01.386 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.386 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:01.386 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.974045+0000", 2024-02-05T04:54:01.386 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:01.386 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.386 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.386 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:01.386 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.974045+0000", 2024-02-05T04:54:01.386 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:01.386 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.386 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.386 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:01.387 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.974046+0000", 2024-02-05T04:54:01.387 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.5400000000000001e-07 2024-02-05T04:54:01.387 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.387 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.387 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:01.387 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.974056+0000", 2024-02-05T04:54:01.387 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.9960000000000005e-06 2024-02-05T04:54:01.387 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.387 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.387 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:01.387 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.974068+0000", 2024-02-05T04:54:01.387 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.2502e-05 2024-02-05T04:54:01.387 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.387 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.387 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:01.387 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.974078+0000", 2024-02-05T04:54:01.388 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.1889999999999996e-06 2024-02-05T04:54:01.388 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.388 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.388 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:01.388 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.974140+0000", 2024-02-05T04:54:01.388 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.2535999999999999e-05 2024-02-05T04:54:01.388 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.388 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.388 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:01.388 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.974151+0000", 2024-02-05T04:54:01.388 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1263e-05 2024-02-05T04:54:01.388 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.388 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.388 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:01.388 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.982119+0000", 2024-02-05T04:54:01.389 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0079679380000000008 2024-02-05T04:54:01.389 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.389 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.389 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:01.389 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.982154+0000", 2024-02-05T04:54:01.389 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.472e-05 2024-02-05T04:54:01.389 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:01.389 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:01.389 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:01.389 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.389 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.389 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.8s3 69/62 ECSubWrite(tid=88, reqid=osd.4.0:88, at_version=69'31, trim_to=0'0, roll_forward_to=66'29))", 2024-02-05T04:54:01.389 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:54:00.965409+0000", 2024-02-05T04:54:01.389 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 0.34496200500000002, 2024-02-05T04:54:01.389 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.003871172, 2024-02-05T04:54:01.389 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:01.390 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:54:01.390 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:01.390 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.390 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:01.390 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.965409+0000", 2024-02-05T04:54:01.390 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:01.390 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.390 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.390 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:01.390 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.965409+0000", 2024-02-05T04:54:01.390 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:01.390 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.390 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.390 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:01.390 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.965409+0000", 2024-02-05T04:54:01.390 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.1300000000000001e-07 2024-02-05T04:54:01.391 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.391 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.391 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:01.391 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.965437+0000", 2024-02-05T04:54:01.391 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.8464000000000001e-05 2024-02-05T04:54:01.391 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.391 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.391 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:01.391 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.965494+0000", 2024-02-05T04:54:01.391 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.7024999999999997e-05 2024-02-05T04:54:01.391 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.391 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.391 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:01.391 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.965517+0000", 2024-02-05T04:54:01.391 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.2348999999999999e-05 2024-02-05T04:54:01.392 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.392 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.392 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:01.392 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.965593+0000", 2024-02-05T04:54:01.392 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.6556000000000004e-05 2024-02-05T04:54:01.392 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.392 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.392 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:01.392 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.965606+0000", 2024-02-05T04:54:01.392 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.2434e-05 2024-02-05T04:54:01.392 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.392 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.392 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:01.392 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.969221+0000", 2024-02-05T04:54:01.392 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0036152900000000002 2024-02-05T04:54:01.393 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:01.393 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:01.393 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:01.393 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.969280+0000", 2024-02-05T04:54:01.393 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.8740999999999999e-05 2024-02-05T04:54:01.393 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:01.393 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:01.393 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:01.393 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:01.393 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:01.393 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-02-05T04:54:01.394 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-02-05T04:54:01.402 INFO:tasks.ceph.osd.14.smithi150.stderr:2024-02-05T04:54:01.401+0000 7f5f81083640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 34020) UID: 0 2024-02-05T04:54:01.503 INFO:tasks.ceph.osd.0.smithi062.stderr:2024-02-05T04:54:01.502+0000 7fece8e03640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34688) UID: 0 2024-02-05T04:54:01.517 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T04:54:01.543 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-02-05T04:54:01.543 INFO:teuthology.orchestra.run.smithi062.stdout: "ops": [], 2024-02-05T04:54:01.543 INFO:teuthology.orchestra.run.smithi062.stdout: "num_ops": 0 2024-02-05T04:54:01.543 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-02-05T04:54:01.552 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-02-05T04:54:01.604 INFO:tasks.ceph.osd.3.smithi196.stderr:2024-02-05T04:54:01.603+0000 7fc8817c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34605) UID: 0 2024-02-05T04:54:01.643 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0: {} 2024-02-05T04:54:01.643 INFO:teuthology.orchestra.run.smithi062.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T04:54:01.662 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1: {} 2024-02-05T04:54:01.662 INFO:teuthology.orchestra.run.smithi062.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T04:54:01.680 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2: {} 2024-02-05T04:54:01.680 INFO:teuthology.orchestra.run.smithi062.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T04:54:01.698 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3: {} 2024-02-05T04:54:01.698 INFO:teuthology.orchestra.run.smithi062.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T04:54:01.702 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-02-05T04:54:01.702 INFO:teuthology.orchestra.run.smithi062.stdout: "ops": [], 2024-02-05T04:54:01.702 INFO:teuthology.orchestra.run.smithi062.stdout: "complaint_time": 30, 2024-02-05T04:54:01.702 INFO:teuthology.orchestra.run.smithi062.stdout: "num_blocked_ops": 0 2024-02-05T04:54:01.702 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-02-05T04:54:01.704 INFO:tasks.ceph.osd.12.smithi062.stderr:2024-02-05T04:54:01.703+0000 7f5e5847e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 34695) UID: 0 2024-02-05T04:54:01.711 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-02-05T04:54:01.715 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4: {} 2024-02-05T04:54:01.716 INFO:teuthology.orchestra.run.smithi062.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T04:54:01.734 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5: {} 2024-02-05T04:54:01.734 INFO:teuthology.orchestra.run.smithi062.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T04:54:01.752 INFO:teuthology.orchestra.run.smithi062.stdout:osd.6: {} 2024-02-05T04:54:01.753 INFO:teuthology.orchestra.run.smithi062.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T04:54:01.770 INFO:teuthology.orchestra.run.smithi062.stdout:osd.7: {} 2024-02-05T04:54:01.770 INFO:teuthology.orchestra.run.smithi062.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T04:54:01.788 INFO:teuthology.orchestra.run.smithi062.stdout:osd.8: {} 2024-02-05T04:54:01.788 INFO:teuthology.orchestra.run.smithi062.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-05T04:54:01.806 INFO:tasks.ceph.osd.10.smithi150.stderr:2024-02-05T04:54:01.805+0000 7f3075a4d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 34018) UID: 0 2024-02-05T04:54:01.806 INFO:teuthology.orchestra.run.smithi062.stdout:osd.9: {} 2024-02-05T04:54:01.806 INFO:teuthology.orchestra.run.smithi062.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-05T04:54:01.824 INFO:teuthology.orchestra.run.smithi062.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:01.824 INFO:teuthology.orchestra.run.smithi062.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:01.825 INFO:teuthology.orchestra.run.smithi062.stdout:osd.10: {} 2024-02-05T04:54:01.825 INFO:teuthology.orchestra.run.smithi062.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-05T04:54:01.834 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-02-05T04:54:01.842 INFO:teuthology.orchestra.run.smithi062.stdout:osd.11: {} 2024-02-05T04:54:01.843 INFO:teuthology.orchestra.run.smithi062.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-05T04:54:01.864 INFO:teuthology.orchestra.run.smithi062.stdout:osd.12: {} 2024-02-05T04:54:01.864 INFO:teuthology.orchestra.run.smithi062.stderr:osd.12: osd_enable_op_tracker = '' 2024-02-05T04:54:01.878 INFO:teuthology.orchestra.run.smithi062.stdout:osd.13: {} 2024-02-05T04:54:01.879 INFO:teuthology.orchestra.run.smithi062.stderr:osd.13: osd_enable_op_tracker = '' 2024-02-05T04:54:01.897 INFO:teuthology.orchestra.run.smithi062.stdout:osd.14: {} 2024-02-05T04:54:01.898 INFO:teuthology.orchestra.run.smithi062.stderr:osd.14: osd_enable_op_tracker = '' 2024-02-05T04:54:01.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-02-05T04:54:01.897+0000 7f8d6647e640 0 -- 172.21.15.62:0/4086762070 >> [v2:172.21.15.196:6826/3187639683,v1:172.21.15.196:6827/3187639683] conn(0x7f8d60221d70 msgr2=0x7f8d60224160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-02-05T04:54:01.906 INFO:tasks.ceph.osd.8.smithi062.stderr:2024-02-05T04:54:01.905+0000 7f9385bf2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 34693) UID: 0 2024-02-05T04:54:01.950 INFO:teuthology.orchestra.run.smithi062.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:01.950 INFO:teuthology.orchestra.run.smithi062.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:01.959 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-02-05T04:54:01.969 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-02-05T04:54:02.007 INFO:tasks.ceph.osd.1.smithi091.stderr:2024-02-05T04:54:02.007+0000 7ff9dc283640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34275) UID: 0 2024-02-05T04:54:02.013 INFO:teuthology.orchestra.run.smithi062.stderr:reweighted osd.4 to 1 (10000) 2024-02-05T04:54:02.015 INFO:tasks.rados.rados.0.smithi196.stdout:254: done (3 left) 2024-02-05T04:54:02.015 INFO:tasks.rados.rados.0.smithi196.stdout:256: done (2 left) 2024-02-05T04:54:02.015 INFO:tasks.rados.rados.0.smithi196.stdout:257: done (1 left) 2024-02-05T04:54:02.015 INFO:tasks.rados.rados.0.smithi196.stdout:258: done (0 left) 2024-02-05T04:54:02.015 INFO:tasks.rados.rados.0.smithi196.stdout:259: delete oid 37 current snap is 22 2024-02-05T04:54:02.020 INFO:tasks.rados.rados.0.smithi196.stdout:259: done (0 left) 2024-02-05T04:54:02.020 INFO:tasks.rados.rados.0.smithi196.stdout:260: snap_remove snap 15 2024-02-05T04:54:02.028 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-02-05T04:54:02.111 INFO:teuthology.orchestra.run.smithi062.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:02.112 INFO:teuthology.orchestra.run.smithi062.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:02.112 INFO:tasks.ceph.osd.8.smithi062.stderr:2024-02-05T04:54:02.111+0000 7f9385bf2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 34693) UID: 0 2024-02-05T04:54:02.121 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-02-05T04:54:02.134 INFO:teuthology.orchestra.run.smithi062.stdout:osd.15: {} 2024-02-05T04:54:02.134 INFO:teuthology.orchestra.run.smithi062.stderr:osd.15: osd_enable_op_tracker = '' 2024-02-05T04:54:02.212 INFO:tasks.ceph.osd.7.smithi196.stderr:2024-02-05T04:54:02.211+0000 7fb281419640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34606) UID: 0 2024-02-05T04:54:02.244 INFO:teuthology.orchestra.run.smithi062.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:02.244 INFO:teuthology.orchestra.run.smithi062.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:02.253 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-02-05T04:54:02.314 INFO:tasks.ceph.osd.11.smithi196.stderr:2024-02-05T04:54:02.312+0000 7f85a8d86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 34609) UID: 0 2024-02-05T04:54:02.360 INFO:teuthology.orchestra.run.smithi062.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:02.361 INFO:teuthology.orchestra.run.smithi062.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:02.369 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-02-05T04:54:02.414 INFO:tasks.ceph.osd.8.smithi062.stderr:2024-02-05T04:54:02.413+0000 7f9385bf2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 34693) UID: 0 2024-02-05T04:54:02.477 INFO:teuthology.orchestra.run.smithi062.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:02.478 INFO:teuthology.orchestra.run.smithi062.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:02.486 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-02-05T04:54:02.515 INFO:tasks.ceph.osd.2.smithi150.stderr:2024-02-05T04:54:02.514+0000 7f7337bf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 34014) UID: 0 2024-02-05T04:54:02.616 INFO:tasks.ceph.osd.0.smithi062.stderr:2024-02-05T04:54:02.615+0000 7fece8e03640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34688) UID: 0 2024-02-05T04:54:02.632 INFO:teuthology.orchestra.run.smithi062.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:02.633 INFO:teuthology.orchestra.run.smithi062.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:02.641 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-02-05T04:54:02.717 INFO:tasks.ceph.osd.0.smithi062.stderr:2024-02-05T04:54:02.716+0000 7fece8e03640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34688) UID: 0 2024-02-05T04:54:02.759 INFO:teuthology.orchestra.run.smithi091.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:02.759 INFO:teuthology.orchestra.run.smithi091.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:02.768 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-02-05T04:54:02.818 INFO:tasks.ceph.osd.1.smithi091.stderr:2024-02-05T04:54:02.817+0000 7ff9dc283640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34275) UID: 0 2024-02-05T04:54:02.875 INFO:teuthology.orchestra.run.smithi091.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:02.875 INFO:teuthology.orchestra.run.smithi091.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:02.884 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-02-05T04:54:02.919 INFO:tasks.ceph.osd.11.smithi196.stderr:2024-02-05T04:54:02.919+0000 7f85a8d86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 34609) UID: 0 2024-02-05T04:54:03.016 INFO:teuthology.orchestra.run.smithi062.stderr:noscrub is unset 2024-02-05T04:54:03.017 INFO:teuthology.orchestra.run.smithi062.stderr:reweighted osd.5 to 1 (10000) 2024-02-05T04:54:03.020 INFO:tasks.ceph.osd.4.smithi062.stderr:2024-02-05T04:54:03.019+0000 7f708c308640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34689) UID: 0 2024-02-05T04:54:03.025 INFO:tasks.rados.rados.0.smithi196.stdout:260: done (0 left) 2024-02-05T04:54:03.025 INFO:tasks.rados.rados.0.smithi196.stdout:append oid 23 current snap is 22 2024-02-05T04:54:03.026 INFO:tasks.rados.rados.0.smithi196.stdout:261: seq_num 120 ranges {2670592=475136,3145728=655360,3801088=622592,4423680=278528} 2024-02-05T04:54:03.037 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-02-05T04:54:03.040 INFO:teuthology.orchestra.run.smithi091.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:03.040 INFO:teuthology.orchestra.run.smithi091.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:03.043 INFO:tasks.rados.rados.0.smithi196.stdout:261: writing smithi19639362-23 from 2670592 to 3145728 tid 1 2024-02-05T04:54:03.047 INFO:tasks.rados.rados.0.smithi196.stdout:261: writing smithi19639362-23 from 3145728 to 3801088 tid 2 2024-02-05T04:54:03.050 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-02-05T04:54:03.051 INFO:tasks.rados.rados.0.smithi196.stdout:261: writing smithi19639362-23 from 3801088 to 4423680 tid 3 2024-02-05T04:54:03.053 INFO:tasks.rados.rados.0.smithi196.stdout:261: writing smithi19639362-23 from 4423680 to 4702208 tid 4 2024-02-05T04:54:03.053 INFO:tasks.rados.rados.0.smithi196.stdout:262: rollback oid 33 current snap is 22 2024-02-05T04:54:03.053 INFO:tasks.rados.rados.0.smithi196.stdout:rollback oid 33 to 20 2024-02-05T04:54:03.053 INFO:tasks.rados.rados.0.smithi196.stdout:263: snap_remove snap 13 2024-02-05T04:54:03.122 INFO:tasks.ceph.osd.3.smithi196.stderr:2024-02-05T04:54:03.121+0000 7fc8817c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34605) UID: 0 2024-02-05T04:54:03.155 INFO:teuthology.orchestra.run.smithi091.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:03.156 INFO:teuthology.orchestra.run.smithi091.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:03.164 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-02-05T04:54:03.222 INFO:tasks.ceph.osd.1.smithi091.stderr:2024-02-05T04:54:03.222+0000 7ff9dc283640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34275) UID: 0 2024-02-05T04:54:03.270 INFO:teuthology.orchestra.run.smithi091.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:03.270 INFO:teuthology.orchestra.run.smithi091.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:03.278 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-02-05T04:54:03.323 INFO:tasks.ceph.osd.12.smithi062.stderr:2024-02-05T04:54:03.322+0000 7f5e5847e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 34695) UID: 0 2024-02-05T04:54:03.424 INFO:tasks.ceph.osd.11.smithi196.stderr:2024-02-05T04:54:03.423+0000 7f85a8d86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 34609) UID: 0 2024-02-05T04:54:03.425 INFO:teuthology.orchestra.run.smithi091.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:03.425 INFO:teuthology.orchestra.run.smithi091.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:03.433 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-02-05T04:54:03.525 INFO:tasks.ceph.osd.9.smithi091.stderr:2024-02-05T04:54:03.524+0000 7efc18827640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 34279) UID: 0 2024-02-05T04:54:03.538 INFO:teuthology.orchestra.run.smithi091.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:03.538 INFO:teuthology.orchestra.run.smithi091.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:03.547 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-02-05T04:54:03.626 INFO:tasks.ceph.osd.8.smithi062.stderr:2024-02-05T04:54:03.625+0000 7f9385bf2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 34693) UID: 0 2024-02-05T04:54:03.695 INFO:teuthology.orchestra.run.smithi091.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:03.695 INFO:teuthology.orchestra.run.smithi091.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:03.705 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-02-05T04:54:03.727 INFO:tasks.ceph.osd.8.smithi062.stderr:2024-02-05T04:54:03.727+0000 7f9385bf2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 34693) UID: 0 2024-02-05T04:54:03.810 INFO:teuthology.orchestra.run.smithi091.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:03.810 INFO:teuthology.orchestra.run.smithi091.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:03.818 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-02-05T04:54:03.828 INFO:tasks.ceph.osd.12.smithi062.stderr:2024-02-05T04:54:03.827+0000 7f5e5847e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 34695) UID: 0 2024-02-05T04:54:03.924 INFO:teuthology.orchestra.run.smithi091.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:03.925 INFO:teuthology.orchestra.run.smithi091.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:03.929 INFO:tasks.ceph.osd.3.smithi196.stderr:2024-02-05T04:54:03.929+0000 7fc8817c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34605) UID: 0 2024-02-05T04:54:03.933 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-02-05T04:54:04.024 INFO:teuthology.orchestra.run.smithi062.stderr:reweighted osd.6 to 1 (10000) 2024-02-05T04:54:04.029 INFO:tasks.rados.rados.0.smithi196.stdout:263: done (2 left) 2024-02-05T04:54:04.029 INFO:tasks.rados.rados.0.smithi196.stdout:264: rollback oid 39 current snap is 22 2024-02-05T04:54:04.029 INFO:tasks.rados.rados.0.smithi196.stdout:rollback oid 39 to 20 2024-02-05T04:54:04.029 INFO:tasks.rados.rados.0.smithi196.stdout:261: finishing write tid 1 to smithi19639362-23 2024-02-05T04:54:04.030 INFO:tasks.rados.rados.0.smithi196.stdout:261: finishing write tid 2 to smithi19639362-23 2024-02-05T04:54:04.030 INFO:tasks.rados.rados.0.smithi196.stdout:261: finishing write tid 3 to smithi19639362-23 2024-02-05T04:54:04.030 INFO:tasks.rados.rados.0.smithi196.stdout:262: finishing rollback tid 0 to smithi19639362-33 2024-02-05T04:54:04.030 INFO:tasks.rados.rados.0.smithi196.stdout:262: finishing rollback tid 1 to smithi19639362-33 2024-02-05T04:54:04.030 INFO:tasks.rados.rados.0.smithi196.stdout:261: finishing write tid 4 to smithi19639362-23 2024-02-05T04:54:04.030 INFO:tasks.rados.rados.0.smithi196.stdout:261: finishing write tid 5 to smithi19639362-23 2024-02-05T04:54:04.030 INFO:tasks.rados.rados.0.smithi196.stdout:262: finishing rollback tid 2 to smithi19639362-33 2024-02-05T04:54:04.030 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 33 v 74 (ObjNum 94 snap 11 seq_num 94) dirty exists 2024-02-05T04:54:04.030 INFO:tasks.rados.rados.0.smithi196.stdout:261: finishing write tid 6 to smithi19639362-23 2024-02-05T04:54:04.030 INFO:tasks.rados.rados.0.smithi196.stdout:261: oid 23 updating version 0 to 59 2024-02-05T04:54:04.030 INFO:tasks.rados.rados.0.smithi196.stdout:261: oid 23 updating version 59 to 63 2024-02-05T04:54:04.030 INFO:tasks.rados.rados.0.smithi196.stdout:261: oid 23 version 63 is already newer than 62 2024-02-05T04:54:04.030 INFO:tasks.rados.rados.0.smithi196.stdout:261: oid 23 version 63 is already newer than 60 2024-02-05T04:54:04.030 INFO:tasks.rados.rados.0.smithi196.stdout:261: oid 23 version 63 is already newer than 61 2024-02-05T04:54:04.030 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 23 v 63 (ObjNum 120 snap 22 seq_num 120) dirty exists 2024-02-05T04:54:04.031 INFO:tasks.rados.rados.0.smithi196.stdout:261: left oid 23 (ObjNum 120 snap 22 seq_num 120) 2024-02-05T04:54:04.031 INFO:tasks.rados.rados.0.smithi196.stdout:261: done (2 left) 2024-02-05T04:54:04.031 INFO:tasks.rados.rados.0.smithi196.stdout:262: done (1 left) 2024-02-05T04:54:04.031 INFO:tasks.rados.rados.0.smithi196.stdout:265: delete oid 16 current snap is 22 2024-02-05T04:54:04.032 INFO:tasks.ceph.osd.9.smithi091.stderr:2024-02-05T04:54:04.031+0000 7efc18827640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 34279) UID: 0 2024-02-05T04:54:04.036 INFO:tasks.rados.rados.0.smithi196.stdout:265: done (1 left) 2024-02-05T04:54:04.037 INFO:tasks.rados.rados.0.smithi196.stdout:266: copy_from oid 13 from oid 28 current snap is 22 2024-02-05T04:54:04.038 INFO:tasks.rados.rados.0.smithi196.stdout:267: read oid 11 snap -1 2024-02-05T04:54:04.038 INFO:tasks.rados.rados.0.smithi196.stdout:267: expect deleted 2024-02-05T04:54:04.039 INFO:tasks.rados.rados.0.smithi196.stdout:268: rollback oid 10 current snap is 22 2024-02-05T04:54:04.039 INFO:tasks.rados.rados.0.smithi196.stdout:rollback oid 10 to 19 2024-02-05T04:54:04.039 INFO:tasks.rados.rados.0.smithi196.stdout:append oid 43 current snap is 22 2024-02-05T04:54:04.039 INFO:tasks.rados.rados.0.smithi196.stdout:269: seq_num 122 ranges {0=507904,507904=425984,933888=442368,1376256=540672,1916928=147456} 2024-02-05T04:54:04.043 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-02-05T04:54:04.044 INFO:tasks.rados.rados.0.smithi196.stdout:269: writing smithi19639362-43 from 0 to 507904 tid 1 2024-02-05T04:54:04.045 INFO:tasks.rados.rados.0.smithi196.stdout:269: writing smithi19639362-43 from 507904 to 933888 tid 2 2024-02-05T04:54:04.049 INFO:tasks.rados.rados.0.smithi196.stdout:269: writing smithi19639362-43 from 933888 to 1376256 tid 3 2024-02-05T04:54:04.052 INFO:teuthology.orchestra.run.smithi091.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:04.053 INFO:teuthology.orchestra.run.smithi091.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:04.053 INFO:tasks.rados.rados.0.smithi196.stdout:269: writing smithi19639362-43 from 1376256 to 1916928 tid 4 2024-02-05T04:54:04.054 INFO:tasks.rados.rados.0.smithi196.stdout:269: writing smithi19639362-43 from 1916928 to 2064384 tid 5 2024-02-05T04:54:04.054 INFO:tasks.rados.rados.0.smithi196.stdout:270: snap_create 2024-02-05T04:54:04.055 INFO:tasks.rados.rados.0.smithi196.stdout:264: finishing rollback tid 0 to smithi19639362-39 2024-02-05T04:54:04.055 INFO:tasks.rados.rados.0.smithi196.stdout:264: finishing rollback tid 1 to smithi19639362-39 2024-02-05T04:54:04.055 INFO:tasks.rados.rados.0.smithi196.stdout:264: finishing rollback tid 2 to smithi19639362-39 2024-02-05T04:54:04.055 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 39 v 45 (ObjNum 79 snap 4 seq_num 79) dirty exists 2024-02-05T04:54:04.055 INFO:tasks.rados.rados.0.smithi196.stdout:266: finishing copy_from racing read to smithi19639362-13 2024-02-05T04:54:04.055 INFO:tasks.rados.rados.0.smithi196.stdout:268: finishing rollback tid 0 to smithi19639362-10 2024-02-05T04:54:04.055 INFO:tasks.rados.rados.0.smithi196.stdout:269: finishing write tid 1 to smithi19639362-43 2024-02-05T04:54:04.056 INFO:tasks.rados.rados.0.smithi196.stdout:268: finishing rollback tid 1 to smithi19639362-10 2024-02-05T04:54:04.056 INFO:tasks.rados.rados.0.smithi196.stdout:269: finishing write tid 2 to smithi19639362-43 2024-02-05T04:54:04.056 INFO:tasks.rados.rados.0.smithi196.stdout:268: finishing rollback tid 2 to smithi19639362-10 2024-02-05T04:54:04.056 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 10 v 25 (ObjNum 53 snap 0 seq_num 53) dirty exists 2024-02-05T04:54:04.057 INFO:tasks.rados.rados.0.smithi196.stdout:266: finishing copy_from to smithi19639362-13 2024-02-05T04:54:04.057 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 13 v 20 (ObjNum 60 snap 2 seq_num 60) dirty exists 2024-02-05T04:54:04.060 INFO:tasks.rados.rados.0.smithi196.stdout:269: finishing write tid 3 to smithi19639362-43 2024-02-05T04:54:04.060 INFO:tasks.rados.rados.0.smithi196.stdout:269: finishing write tid 4 to smithi19639362-43 2024-02-05T04:54:04.061 INFO:tasks.rados.rados.0.smithi196.stdout:269: finishing write tid 5 to smithi19639362-43 2024-02-05T04:54:04.061 INFO:tasks.rados.rados.0.smithi196.stdout:269: finishing write tid 6 to smithi19639362-43 2024-02-05T04:54:04.062 INFO:tasks.rados.rados.0.smithi196.stdout:269: finishing write tid 7 to smithi19639362-43 2024-02-05T04:54:04.063 INFO:tasks.rados.rados.0.smithi196.stdout:269: oid 43 updating version 0 to 83 2024-02-05T04:54:04.063 INFO:tasks.rados.rados.0.smithi196.stdout:269: oid 43 version 83 is already newer than 79 2024-02-05T04:54:04.063 INFO:tasks.rados.rados.0.smithi196.stdout:269: oid 43 version 83 is already newer than 80 2024-02-05T04:54:04.063 INFO:tasks.rados.rados.0.smithi196.stdout:269: oid 43 updating version 83 to 84 2024-02-05T04:54:04.063 INFO:tasks.rados.rados.0.smithi196.stdout:269: oid 43 updating version 84 to 85 2024-02-05T04:54:04.063 INFO:tasks.rados.rados.0.smithi196.stdout:269: oid 43 updating version 85 to 86 2024-02-05T04:54:04.063 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 43 v 86 (ObjNum 122 snap 22 seq_num 122) dirty exists 2024-02-05T04:54:04.063 INFO:tasks.rados.rados.0.smithi196.stdout:269: left oid 43 (ObjNum 122 snap 22 seq_num 122) 2024-02-05T04:54:04.066 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-02-05T04:54:04.133 INFO:tasks.ceph.osd.10.smithi150.stderr:2024-02-05T04:54:04.132+0000 7f3075a4d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 34018) UID: 0 2024-02-05T04:54:04.146 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T04:54:04.211 INFO:teuthology.orchestra.run.smithi091.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:04.211 INFO:teuthology.orchestra.run.smithi091.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:04.220 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-02-05T04:54:04.266 INFO:tasks.ceph.osd.10.smithi150.stderr:2024-02-05T04:54:04.265+0000 7f3075a4d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 34018) UID: 0 2024-02-05T04:54:04.273 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0: {} 2024-02-05T04:54:04.274 INFO:teuthology.orchestra.run.smithi062.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T04:54:04.292 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1: {} 2024-02-05T04:54:04.293 INFO:teuthology.orchestra.run.smithi062.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T04:54:04.310 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2: {} 2024-02-05T04:54:04.310 INFO:teuthology.orchestra.run.smithi062.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T04:54:04.329 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3: {} 2024-02-05T04:54:04.329 INFO:teuthology.orchestra.run.smithi062.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T04:54:04.334 INFO:tasks.ceph.osd.15.smithi196.stderr:2024-02-05T04:54:04.333+0000 7f1facc45640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 34611) UID: 0 2024-02-05T04:54:04.339 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-02-05T04:54:04.339 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [], 2024-02-05T04:54:04.339 INFO:teuthology.orchestra.run.smithi150.stdout: "num_ops": 0 2024-02-05T04:54:04.339 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-02-05T04:54:04.346 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4: {} 2024-02-05T04:54:04.346 INFO:teuthology.orchestra.run.smithi062.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T04:54:04.348 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-02-05T04:54:04.363 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5: {} 2024-02-05T04:54:04.364 INFO:teuthology.orchestra.run.smithi062.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T04:54:04.381 INFO:teuthology.orchestra.run.smithi062.stdout:osd.6: {} 2024-02-05T04:54:04.382 INFO:teuthology.orchestra.run.smithi062.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T04:54:04.399 INFO:teuthology.orchestra.run.smithi062.stdout:osd.7: {} 2024-02-05T04:54:04.400 INFO:teuthology.orchestra.run.smithi062.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T04:54:04.417 INFO:teuthology.orchestra.run.smithi062.stdout:osd.8: {} 2024-02-05T04:54:04.417 INFO:teuthology.orchestra.run.smithi062.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-05T04:54:04.435 INFO:tasks.ceph.osd.4.smithi062.stderr:2024-02-05T04:54:04.434+0000 7f708c308640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34689) UID: 0 2024-02-05T04:54:04.435 INFO:teuthology.orchestra.run.smithi062.stdout:osd.9: {} 2024-02-05T04:54:04.435 INFO:teuthology.orchestra.run.smithi062.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-05T04:54:04.453 INFO:teuthology.orchestra.run.smithi062.stdout:osd.10: {} 2024-02-05T04:54:04.453 INFO:teuthology.orchestra.run.smithi062.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-05T04:54:04.471 INFO:teuthology.orchestra.run.smithi062.stdout:osd.11: {} 2024-02-05T04:54:04.471 INFO:teuthology.orchestra.run.smithi062.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-05T04:54:04.487 INFO:teuthology.orchestra.run.smithi062.stdout:osd.12: {} 2024-02-05T04:54:04.488 INFO:teuthology.orchestra.run.smithi062.stderr:osd.12: osd_enable_op_tracker = '' 2024-02-05T04:54:04.495 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-02-05T04:54:04.496 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [], 2024-02-05T04:54:04.496 INFO:teuthology.orchestra.run.smithi150.stdout: "complaint_time": 30, 2024-02-05T04:54:04.496 INFO:teuthology.orchestra.run.smithi150.stdout: "num_blocked_ops": 0 2024-02-05T04:54:04.496 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-02-05T04:54:04.504 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-02-05T04:54:04.505 INFO:teuthology.orchestra.run.smithi062.stdout:osd.13: {} 2024-02-05T04:54:04.505 INFO:teuthology.orchestra.run.smithi062.stderr:osd.13: osd_enable_op_tracker = '' 2024-02-05T04:54:04.523 INFO:teuthology.orchestra.run.smithi062.stdout:osd.14: {} 2024-02-05T04:54:04.523 INFO:teuthology.orchestra.run.smithi062.stderr:osd.14: osd_enable_op_tracker = '' 2024-02-05T04:54:04.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-02-05T04:54:04.523+0000 7fd60ad9d640 0 -- 172.21.15.62:0/2160261671 >> [v2:172.21.15.196:6826/3187639683,v1:172.21.15.196:6827/3187639683] conn(0x7fd60c2220a0 msgr2=0x7fd60c224490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-02-05T04:54:04.536 INFO:tasks.ceph.osd.9.smithi091.stderr:2024-02-05T04:54:04.535+0000 7efc18827640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 34279) UID: 0 2024-02-05T04:54:04.611 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-02-05T04:54:04.612 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 20, 2024-02-05T04:54:04.612 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 600, 2024-02-05T04:54:04.612 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [ 2024-02-05T04:54:04.612 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.612 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.4209.0:85 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'55, mlcod=24'50)", 2024-02-05T04:54:04.612 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:09.697543+0000", 2024-02-05T04:54:04.612 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 54.912056018999998, 2024-02-05T04:54:04.612 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0081615030000000005, 2024-02-05T04:54:04.612 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:04.612 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:04.612 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:04.612 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.612 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:04.612 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.697543+0000", 2024-02-05T04:54:04.613 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:04.613 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.613 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.613 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:04.613 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.697543+0000", 2024-02-05T04:54:04.613 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:04.613 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.613 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.613 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:04.613 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.697544+0000", 2024-02-05T04:54:04.613 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.7899999999999999e-07 2024-02-05T04:54:04.613 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.613 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.613 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:04.613 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.697546+0000", 2024-02-05T04:54:04.614 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.7970000000000001e-06 2024-02-05T04:54:04.614 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.614 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.614 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:04.614 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.697553+0000", 2024-02-05T04:54:04.614 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.2929999999999997e-06 2024-02-05T04:54:04.614 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.614 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.614 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:04.614 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.697556+0000", 2024-02-05T04:54:04.614 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.4309999999999998e-06 2024-02-05T04:54:04.614 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.614 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.614 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:04.614 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.697596+0000", 2024-02-05T04:54:04.615 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.977e-05 2024-02-05T04:54:04.615 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.615 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.615 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:54:04.615 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.697607+0000", 2024-02-05T04:54:04.615 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1459e-05 2024-02-05T04:54:04.615 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.615 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.615 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-02-05T04:54:04.615 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.705669+0000", 2024-02-05T04:54:04.615 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0080620009999999992 2024-02-05T04:54:04.615 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.615 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.615 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:04.615 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.705705+0000", 2024-02-05T04:54:04.616 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.5373000000000002e-05 2024-02-05T04:54:04.616 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:04.616 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:04.616 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:04.616 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.616 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.616 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.4209.0:86 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'56, mlcod=24'52)", 2024-02-05T04:54:04.616 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:09.698014+0000", 2024-02-05T04:54:04.616 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 54.911584918999999, 2024-02-05T04:54:04.616 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0077071750000000001, 2024-02-05T04:54:04.616 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:04.616 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:04.616 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:04.616 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.616 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:04.617 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698014+0000", 2024-02-05T04:54:04.617 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:04.617 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.617 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.617 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:04.617 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698014+0000", 2024-02-05T04:54:04.617 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:04.617 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.617 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.617 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:04.617 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698015+0000", 2024-02-05T04:54:04.617 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.9900000000000002e-07 2024-02-05T04:54:04.617 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.617 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.617 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:04.617 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698018+0000", 2024-02-05T04:54:04.618 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.0019999999999998e-06 2024-02-05T04:54:04.618 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.618 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.618 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:04.618 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698024+0000", 2024-02-05T04:54:04.618 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.251e-06 2024-02-05T04:54:04.618 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.618 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.618 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:04.618 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698028+0000", 2024-02-05T04:54:04.618 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.76e-06 2024-02-05T04:54:04.618 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.618 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.618 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:04.618 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698066+0000", 2024-02-05T04:54:04.619 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.8556999999999997e-05 2024-02-05T04:54:04.619 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.619 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.619 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:54:04.619 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698078+0000", 2024-02-05T04:54:04.619 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1620999999999999e-05 2024-02-05T04:54:04.619 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.619 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.619 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-02-05T04:54:04.619 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.705706+0000", 2024-02-05T04:54:04.619 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0076284400000000002 2024-02-05T04:54:04.619 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.619 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.619 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:04.619 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.705721+0000", 2024-02-05T04:54:04.620 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.5245000000000001e-05 2024-02-05T04:54:04.620 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:04.620 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:04.620 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:04.620 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.620 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.620 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.4209.0:117 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'79, mlcod=24'69)", 2024-02-05T04:54:04.620 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:09.857803+0000", 2024-02-05T04:54:04.620 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 54.751795778999998, 2024-02-05T04:54:04.620 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0076301729999999996, 2024-02-05T04:54:04.620 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:04.620 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:04.620 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:04.620 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.621 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:04.621 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.857803+0000", 2024-02-05T04:54:04.621 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:04.621 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.621 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.621 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:04.621 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.857803+0000", 2024-02-05T04:54:04.621 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:04.621 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.621 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.621 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:04.621 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.857804+0000", 2024-02-05T04:54:04.621 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.1999999999999999e-07 2024-02-05T04:54:04.621 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.621 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.621 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:04.621 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.857810+0000", 2024-02-05T04:54:04.622 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.8690000000000001e-06 2024-02-05T04:54:04.622 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.622 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.622 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:04.622 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.857821+0000", 2024-02-05T04:54:04.622 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1059e-05 2024-02-05T04:54:04.622 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.622 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.622 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:04.622 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.857827+0000", 2024-02-05T04:54:04.622 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.1430000000000003e-06 2024-02-05T04:54:04.622 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.622 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.622 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:04.622 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.860138+0000", 2024-02-05T04:54:04.623 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0023111160000000002 2024-02-05T04:54:04.623 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.623 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.623 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:54:04.623 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.860152+0000", 2024-02-05T04:54:04.623 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.3896999999999999e-05 2024-02-05T04:54:04.623 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.623 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.623 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-02-05T04:54:04.623 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.865400+0000", 2024-02-05T04:54:04.623 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0052475040000000001 2024-02-05T04:54:04.623 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.623 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.623 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:04.623 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.865434+0000", 2024-02-05T04:54:04.623 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.3865000000000002e-05 2024-02-05T04:54:04.623 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:04.624 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:04.624 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:04.624 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.624 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.624 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.4634.0:246 3.es0 3:796382fe:::smithi19639362-29:head [append 0~0] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:54:04.624 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:29.186925+0000", 2024-02-05T04:54:04.624 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 35.422674459, 2024-02-05T04:54:04.624 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0088703129999999995, 2024-02-05T04:54:04.624 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:04.624 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:04.624 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-02-05T04:54:04.624 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.4634", 2024-02-05T04:54:04.624 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:04.624 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 246 2024-02-05T04:54:04.624 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.625 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:04.625 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.625 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:04.625 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186925+0000", 2024-02-05T04:54:04.625 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:04.625 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.625 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.625 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:04.625 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186925+0000", 2024-02-05T04:54:04.625 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:04.625 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.625 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.625 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:04.625 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186925+0000", 2024-02-05T04:54:04.625 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.53e-07 2024-02-05T04:54:04.625 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.626 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.626 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:04.626 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186928+0000", 2024-02-05T04:54:04.626 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.6900000000000001e-06 2024-02-05T04:54:04.626 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.626 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.626 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:04.626 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186933+0000", 2024-02-05T04:54:04.626 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.502e-06 2024-02-05T04:54:04.626 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.626 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.626 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:04.626 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186937+0000", 2024-02-05T04:54:04.626 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.6200000000000001e-06 2024-02-05T04:54:04.626 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.627 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.627 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:04.627 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.190223+0000", 2024-02-05T04:54:04.627 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0032863050000000002 2024-02-05T04:54:04.627 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.627 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.627 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:54:04.627 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.190312+0000", 2024-02-05T04:54:04.627 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.8821999999999998e-05 2024-02-05T04:54:04.627 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.627 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.627 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:04.627 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.190713+0000", 2024-02-05T04:54:04.627 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00040077400000000001 2024-02-05T04:54:04.627 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.627 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.627 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:04.628 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.193148+0000", 2024-02-05T04:54:04.628 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.002435544 2024-02-05T04:54:04.628 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.628 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.628 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-02-05T04:54:04.628 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.195751+0000", 2024-02-05T04:54:04.628 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0026025000000000002 2024-02-05T04:54:04.628 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.628 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.628 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:04.628 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.195795+0000", 2024-02-05T04:54:04.628 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.4202999999999997e-05 2024-02-05T04:54:04.628 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:04.628 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:04.629 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:04.629 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.629 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.629 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.4634.0:249 3.es0 3:70a5b9dd:::smithi19639362-46:head [delete] snapc 3=[3] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:54:04.629 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:29.186943+0000", 2024-02-05T04:54:04.629 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 35.422656498999999, 2024-02-05T04:54:04.629 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.011398484, 2024-02-05T04:54:04.629 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:04.629 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:04.629 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-02-05T04:54:04.629 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.4634", 2024-02-05T04:54:04.629 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:04.629 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 249 2024-02-05T04:54:04.629 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.629 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:04.629 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.630 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:04.630 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186943+0000", 2024-02-05T04:54:04.630 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:04.630 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.630 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.630 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:04.630 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186943+0000", 2024-02-05T04:54:04.630 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:04.630 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.630 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.630 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:04.630 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186943+0000", 2024-02-05T04:54:04.630 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.1899999999999999e-07 2024-02-05T04:54:04.630 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.630 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.630 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:04.631 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186944+0000", 2024-02-05T04:54:04.631 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.207e-06 2024-02-05T04:54:04.631 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.631 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.631 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:04.631 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186948+0000", 2024-02-05T04:54:04.631 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.7129999999999999e-06 2024-02-05T04:54:04.631 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.631 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.631 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:04.631 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186951+0000", 2024-02-05T04:54:04.631 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.9979999999999999e-06 2024-02-05T04:54:04.631 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.631 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.631 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:04.632 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.191572+0000", 2024-02-05T04:54:04.632 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0046207669999999996 2024-02-05T04:54:04.632 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.632 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.632 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:54:04.632 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.191630+0000", 2024-02-05T04:54:04.632 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.8774999999999999e-05 2024-02-05T04:54:04.632 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.632 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.632 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:04.632 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.192049+0000", 2024-02-05T04:54:04.632 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00041888 2024-02-05T04:54:04.632 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.632 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.632 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:04.633 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.194188+0000", 2024-02-05T04:54:04.633 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0021391359999999998 2024-02-05T04:54:04.633 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.633 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.633 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-02-05T04:54:04.633 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.198284+0000", 2024-02-05T04:54:04.633 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0040960509999999999 2024-02-05T04:54:04.633 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.633 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.633 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:04.633 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.198341+0000", 2024-02-05T04:54:04.633 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.6737999999999998e-05 2024-02-05T04:54:04.633 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:04.633 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:04.633 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:04.633 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.634 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.634 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.4634.0:382 3.es0 3:7f0ef6bd:::smithi19639362-19:head [append 0~655360] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-02-05T04:54:04.634 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:34.167434+0000", 2024-02-05T04:54:04.634 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 30.442165209999999, 2024-02-05T04:54:04.634 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0077709989999999998, 2024-02-05T04:54:04.634 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:04.634 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:04.634 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-02-05T04:54:04.634 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.4634", 2024-02-05T04:54:04.634 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:04.634 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 382 2024-02-05T04:54:04.634 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.634 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:04.634 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.634 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:04.634 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.167434+0000", 2024-02-05T04:54:04.635 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:04.635 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.635 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.635 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:04.635 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.167434+0000", 2024-02-05T04:54:04.635 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:04.635 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.635 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.635 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:04.635 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.167436+0000", 2024-02-05T04:54:04.635 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.885e-06 2024-02-05T04:54:04.635 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.635 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.635 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:04.635 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.168595+0000", 2024-02-05T04:54:04.635 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0011590350000000001 2024-02-05T04:54:04.636 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.636 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.636 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:04.636 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.168610+0000", 2024-02-05T04:54:04.636 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.4778e-05 2024-02-05T04:54:04.636 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.636 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.636 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:04.636 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.168700+0000", 2024-02-05T04:54:04.636 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.0462000000000002e-05 2024-02-05T04:54:04.636 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.636 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.636 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:04.636 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.168791+0000", 2024-02-05T04:54:04.636 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.1015999999999997e-05 2024-02-05T04:54:04.637 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.637 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.637 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:54:04.637 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.168911+0000", 2024-02-05T04:54:04.637 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000119864 2024-02-05T04:54:04.637 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.637 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.637 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:04.637 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.170527+0000", 2024-02-05T04:54:04.637 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0016163589999999999 2024-02-05T04:54:04.637 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.637 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.638 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:04.638 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.174753+0000", 2024-02-05T04:54:04.638 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0042254429999999997 2024-02-05T04:54:04.638 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.638 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.638 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-02-05T04:54:04.638 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.175115+0000", 2024-02-05T04:54:04.638 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00036262499999999999 2024-02-05T04:54:04.638 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.638 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.638 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:04.638 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.175205+0000", 2024-02-05T04:54:04.638 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.9531999999999996e-05 2024-02-05T04:54:04.638 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:04.638 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:04.639 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:04.639 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.639 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.639 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.3s4 57/31 ECSubWrite(tid=76, reqid=osd.4.0:76, at_version=57'35, trim_to=0'0, roll_forward_to=47'33))", 2024-02-05T04:54:04.639 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:48.952725+0000", 2024-02-05T04:54:04.639 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 15.656874427, 2024-02-05T04:54:04.639 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.007485312, 2024-02-05T04:54:04.639 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:04.639 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:04.639 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:04.639 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.639 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:04.639 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.952725+0000", 2024-02-05T04:54:04.639 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:04.639 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.639 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.640 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:04.640 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.952725+0000", 2024-02-05T04:54:04.640 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:04.640 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.640 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.640 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:04.640 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.952726+0000", 2024-02-05T04:54:04.640 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.7100000000000011e-07 2024-02-05T04:54:04.640 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.640 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.640 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:04.640 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.952751+0000", 2024-02-05T04:54:04.640 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.5477e-05 2024-02-05T04:54:04.640 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.641 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.641 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:04.641 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.952765+0000", 2024-02-05T04:54:04.641 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.3502e-05 2024-02-05T04:54:04.641 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.641 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.641 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:04.641 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.952773+0000", 2024-02-05T04:54:04.641 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.1499999999999999e-06 2024-02-05T04:54:04.641 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.641 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.642 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:04.642 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.952832+0000", 2024-02-05T04:54:04.642 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.9095000000000003e-05 2024-02-05T04:54:04.642 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.642 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.642 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:04.642 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.952839+0000", 2024-02-05T04:54:04.642 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.764e-06 2024-02-05T04:54:04.642 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.642 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.642 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:04.642 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.960160+0000", 2024-02-05T04:54:04.642 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0073216820000000004 2024-02-05T04:54:04.642 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.642 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.643 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:04.643 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.960210+0000", 2024-02-05T04:54:04.643 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.9670999999999999e-05 2024-02-05T04:54:04.643 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:04.643 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:04.643 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:04.643 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.643 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.643 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.fs1 58/31 ECSubWrite(tid=104, reqid=client.4634.0:518, at_version=58'51, trim_to=0'0, roll_forward_to=57'49))", 2024-02-05T04:54:04.643 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:49.974036+0000", 2024-02-05T04:54:04.643 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 14.635563267, 2024-02-05T04:54:04.643 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0089249829999999992, 2024-02-05T04:54:04.643 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:04.643 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:04.643 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:04.643 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.644 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:04.644 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.974036+0000", 2024-02-05T04:54:04.644 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:04.644 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.644 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.644 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:04.644 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.974036+0000", 2024-02-05T04:54:04.644 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:04.644 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.644 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.644 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:04.644 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.974037+0000", 2024-02-05T04:54:04.644 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.0999999999999997e-07 2024-02-05T04:54:04.644 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.644 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.645 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:04.645 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.976765+0000", 2024-02-05T04:54:04.645 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0027280059999999998 2024-02-05T04:54:04.645 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.645 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.645 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:04.645 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.976780+0000", 2024-02-05T04:54:04.645 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.5238999999999999e-05 2024-02-05T04:54:04.645 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.645 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.645 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:04.645 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.976789+0000", 2024-02-05T04:54:04.645 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.8759999999999994e-06 2024-02-05T04:54:04.645 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.645 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.645 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:04.646 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.976857+0000", 2024-02-05T04:54:04.646 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.7984999999999994e-05 2024-02-05T04:54:04.646 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.646 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.646 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:04.646 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.976869+0000", 2024-02-05T04:54:04.646 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1579999999999999e-05 2024-02-05T04:54:04.646 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.646 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.646 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:04.646 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.982917+0000", 2024-02-05T04:54:04.646 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0060485069999999998 2024-02-05T04:54:04.646 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.646 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.646 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:04.647 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.982961+0000", 2024-02-05T04:54:04.647 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.388e-05 2024-02-05T04:54:04.647 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:04.647 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:04.647 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:04.647 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.647 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.647 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.fs1 63/31 ECSubWrite(tid=122, reqid=osd.11.0:122, at_version=63'56, trim_to=0'0, roll_forward_to=58'51))", 2024-02-05T04:54:04.647 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:54.990151+0000", 2024-02-05T04:54:04.647 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 9.6194477220000003, 2024-02-05T04:54:04.647 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.011036684, 2024-02-05T04:54:04.647 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:04.647 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:04.647 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:04.647 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.648 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:04.648 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.990151+0000", 2024-02-05T04:54:04.648 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:04.648 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.648 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.648 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:04.648 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.990151+0000", 2024-02-05T04:54:04.648 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:04.648 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.648 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.648 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:04.648 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.990152+0000", 2024-02-05T04:54:04.648 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.9999999999999997e-07 2024-02-05T04:54:04.648 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.648 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.649 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:04.649 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.990172+0000", 2024-02-05T04:54:04.649 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.0000000000000002e-05 2024-02-05T04:54:04.649 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.649 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.649 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:04.649 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.990183+0000", 2024-02-05T04:54:04.649 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0845999999999999e-05 2024-02-05T04:54:04.649 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.649 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.649 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:04.649 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.990196+0000", 2024-02-05T04:54:04.649 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.258e-05 2024-02-05T04:54:04.649 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.649 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.649 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:04.650 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.990255+0000", 2024-02-05T04:54:04.650 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.9175000000000002e-05 2024-02-05T04:54:04.650 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.650 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.650 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:04.650 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.990384+0000", 2024-02-05T04:54:04.650 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000128942 2024-02-05T04:54:04.650 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.650 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.650 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:04.650 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.001103+0000", 2024-02-05T04:54:04.650 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.010718811999999999 2024-02-05T04:54:04.650 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.650 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.650 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:04.651 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.001188+0000", 2024-02-05T04:54:04.651 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.5629000000000003e-05 2024-02-05T04:54:04.651 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:04.651 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:04.651 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:04.651 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.651 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.651 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.13s4 63/62 ECSubWrite(tid=83, reqid=osd.4.0:83, at_version=63'39, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:04.651 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:54.996293+0000", 2024-02-05T04:54:04.651 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 9.6133062349999996, 2024-02-05T04:54:04.651 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.010849175000000001, 2024-02-05T04:54:04.651 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:04.651 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:04.651 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:04.651 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.652 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:04.652 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.996293+0000", 2024-02-05T04:54:04.652 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:04.652 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.652 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.652 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:04.652 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.996293+0000", 2024-02-05T04:54:04.652 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:04.652 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.652 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.652 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:04.652 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.996293+0000", 2024-02-05T04:54:04.652 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.2799999999999996e-07 2024-02-05T04:54:04.652 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.652 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.652 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:04.653 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.996317+0000", 2024-02-05T04:54:04.653 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.3402000000000001e-05 2024-02-05T04:54:04.653 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.653 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.653 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:04.653 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.996333+0000", 2024-02-05T04:54:04.653 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.6438000000000001e-05 2024-02-05T04:54:04.653 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.653 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.653 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:04.653 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.996342+0000", 2024-02-05T04:54:04.653 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.6899999999999998e-06 2024-02-05T04:54:04.653 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.653 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.653 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:04.653 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.996393+0000", 2024-02-05T04:54:04.654 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.0819999999999998e-05 2024-02-05T04:54:04.654 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.654 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.654 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:04.654 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.996402+0000", 2024-02-05T04:54:04.654 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.3000000000000007e-06 2024-02-05T04:54:04.654 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.654 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.654 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:04.654 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.007100+0000", 2024-02-05T04:54:04.654 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.010698193 2024-02-05T04:54:04.654 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.654 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.654 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:04.655 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.007142+0000", 2024-02-05T04:54:04.655 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.1804000000000002e-05 2024-02-05T04:54:04.655 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:04.655 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:04.655 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:04.655 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.655 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.655 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.12s1 63/62 ECSubWrite(tid=73, reqid=osd.12.0:73, at_version=63'55, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:04.655 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:54.998972+0000", 2024-02-05T04:54:04.655 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 9.6106274290000009, 2024-02-05T04:54:04.655 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.011914983000000001, 2024-02-05T04:54:04.655 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:04.655 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:04.655 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:04.655 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.655 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:04.656 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.998972+0000", 2024-02-05T04:54:04.656 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:04.656 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.656 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.656 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:04.656 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.998972+0000", 2024-02-05T04:54:04.656 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:04.656 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.656 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.656 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:04.656 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.998972+0000", 2024-02-05T04:54:04.656 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.5199999999999996e-07 2024-02-05T04:54:04.656 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.656 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.657 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:04.657 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.998992+0000", 2024-02-05T04:54:04.657 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.9911999999999998e-05 2024-02-05T04:54:04.657 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.657 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.657 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:04.657 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.999013+0000", 2024-02-05T04:54:04.657 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.107e-05 2024-02-05T04:54:04.657 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.657 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.657 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:04.657 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.999022+0000", 2024-02-05T04:54:04.657 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.1310000000000006e-06 2024-02-05T04:54:04.657 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.657 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.657 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:04.658 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.999126+0000", 2024-02-05T04:54:04.658 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000104842 2024-02-05T04:54:04.658 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.658 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.658 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:04.658 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.999136+0000", 2024-02-05T04:54:04.658 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.8810000000000008e-06 2024-02-05T04:54:04.658 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.658 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.658 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:04.658 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.010847+0000", 2024-02-05T04:54:04.658 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.011711222 2024-02-05T04:54:04.658 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.658 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.659 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:04.659 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.010887+0000", 2024-02-05T04:54:04.659 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.9273000000000002e-05 2024-02-05T04:54:04.659 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:04.659 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:04.659 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:04.659 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.659 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.659 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.0s1 63/62 ECSubWrite(tid=52, reqid=client.4634.0:576, at_version=63'6, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:04.659 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:54.999558+0000", 2024-02-05T04:54:04.659 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 9.6100409340000006, 2024-02-05T04:54:04.659 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0077513749999999996, 2024-02-05T04:54:04.659 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:04.659 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:04.659 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:04.660 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.660 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:04.660 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.999558+0000", 2024-02-05T04:54:04.660 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:04.660 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.660 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.660 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:04.660 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.999558+0000", 2024-02-05T04:54:04.660 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:04.660 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.660 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.660 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:04.660 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.999559+0000", 2024-02-05T04:54:04.660 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.6400000000000001e-07 2024-02-05T04:54:04.660 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.661 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.661 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:04.661 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.000299+0000", 2024-02-05T04:54:04.661 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00074006600000000005 2024-02-05T04:54:04.661 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.661 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.661 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:04.661 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.000312+0000", 2024-02-05T04:54:04.661 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2996000000000001e-05 2024-02-05T04:54:04.661 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.661 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.661 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:04.661 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.000321+0000", 2024-02-05T04:54:04.661 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.9029999999999993e-06 2024-02-05T04:54:04.661 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.662 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.662 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:04.662 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.000393+0000", 2024-02-05T04:54:04.662 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.1783e-05 2024-02-05T04:54:04.662 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.662 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.662 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:04.662 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.000403+0000", 2024-02-05T04:54:04.662 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.9690000000000006e-06 2024-02-05T04:54:04.662 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.662 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.662 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:04.662 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.007278+0000", 2024-02-05T04:54:04.662 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0068756479999999998 2024-02-05T04:54:04.662 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.663 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.663 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:04.663 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.007310+0000", 2024-02-05T04:54:04.663 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.1245999999999999e-05 2024-02-05T04:54:04.663 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:04.663 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:04.663 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:04.663 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.663 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.663 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.fs1 63/31 ECSubWrite(tid=124, reqid=osd.11.0:122, at_version=0'0, trim_to=0'0, roll_forward_to=63'56))", 2024-02-05T04:54:04.663 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:55.001791+0000", 2024-02-05T04:54:04.663 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 9.6078079729999999, 2024-02-05T04:54:04.663 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.009426182, 2024-02-05T04:54:04.663 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:04.663 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:04.664 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:04.664 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.664 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:04.664 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.001791+0000", 2024-02-05T04:54:04.664 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:04.664 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.664 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.664 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:04.664 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.001791+0000", 2024-02-05T04:54:04.664 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:04.664 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.664 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.664 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:04.664 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.001792+0000", 2024-02-05T04:54:04.664 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.6899999999999996e-07 2024-02-05T04:54:04.665 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.665 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.665 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:04.665 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.001807+0000", 2024-02-05T04:54:04.665 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.4671999999999999e-05 2024-02-05T04:54:04.665 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.665 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.665 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:04.665 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.001822+0000", 2024-02-05T04:54:04.665 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.5071e-05 2024-02-05T04:54:04.665 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.665 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.665 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:04.665 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.001831+0000", 2024-02-05T04:54:04.665 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.1419999999999999e-06 2024-02-05T04:54:04.666 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.666 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.666 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:04.666 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.001987+0000", 2024-02-05T04:54:04.666 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00015554300000000001 2024-02-05T04:54:04.666 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.666 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.666 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:04.666 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.002123+0000", 2024-02-05T04:54:04.666 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000136231 2024-02-05T04:54:04.666 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.666 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.666 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:04.666 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.011175+0000", 2024-02-05T04:54:04.666 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0090525250000000005 2024-02-05T04:54:04.666 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.667 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.667 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:04.667 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.011217+0000", 2024-02-05T04:54:04.667 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.2029e-05 2024-02-05T04:54:04.667 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:04.667 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:04.667 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:04.667 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.667 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.667 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.4s1 63/62 ECSubWrite(tid=122, reqid=osd.1.0:122, at_version=63'73, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:04.667 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:55.002586+0000", 2024-02-05T04:54:04.667 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 9.6070136319999992, 2024-02-05T04:54:04.667 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.011429455999999999, 2024-02-05T04:54:04.667 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:04.668 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:04.668 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:04.668 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.668 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:04.668 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.002586+0000", 2024-02-05T04:54:04.668 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:04.668 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.668 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.668 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:04.668 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.002586+0000", 2024-02-05T04:54:04.668 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:04.668 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.668 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.668 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:04.668 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.002586+0000", 2024-02-05T04:54:04.669 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.8599999999999998e-07 2024-02-05T04:54:04.669 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.669 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.669 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:04.669 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.002607+0000", 2024-02-05T04:54:04.669 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.0480999999999999e-05 2024-02-05T04:54:04.669 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.669 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.669 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:04.669 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.002619+0000", 2024-02-05T04:54:04.669 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2099999999999999e-05 2024-02-05T04:54:04.669 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.669 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.669 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:04.669 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.002627+0000", 2024-02-05T04:54:04.669 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.9719999999999992e-06 2024-02-05T04:54:04.670 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.670 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.670 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:04.670 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.002691+0000", 2024-02-05T04:54:04.670 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.4784999999999997e-05 2024-02-05T04:54:04.670 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.670 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.670 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:04.670 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.002702+0000", 2024-02-05T04:54:04.670 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0138e-05 2024-02-05T04:54:04.670 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.670 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.670 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:04.670 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.013981+0000", 2024-02-05T04:54:04.670 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.011279632 2024-02-05T04:54:04.670 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.671 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.671 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:04.671 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.014015+0000", 2024-02-05T04:54:04.671 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.3862000000000002e-05 2024-02-05T04:54:04.671 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:04.671 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:04.671 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:04.671 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.671 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.671 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.8s2 63/62 ECSubWrite(tid=84, reqid=client.4634.0:593, at_version=0'0, trim_to=0'0, roll_forward_to=63'27))", 2024-02-05T04:54:04.671 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:55.002745+0000", 2024-02-05T04:54:04.671 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 9.6068544019999997, 2024-02-05T04:54:04.671 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0076694420000000003, 2024-02-05T04:54:04.671 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:04.671 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:04.672 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:04.672 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.672 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:04.672 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.002745+0000", 2024-02-05T04:54:04.672 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:04.672 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.672 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.672 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:04.672 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.002745+0000", 2024-02-05T04:54:04.672 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:04.672 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.672 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.672 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:04.672 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.002746+0000", 2024-02-05T04:54:04.672 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.2099999999999995e-07 2024-02-05T04:54:04.672 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.673 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.673 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:04.673 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.002756+0000", 2024-02-05T04:54:04.673 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0509999999999999e-05 2024-02-05T04:54:04.673 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.673 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.673 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:04.673 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.002773+0000", 2024-02-05T04:54:04.673 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.6682000000000001e-05 2024-02-05T04:54:04.673 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.673 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.673 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:04.673 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.002782+0000", 2024-02-05T04:54:04.673 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.6910000000000003e-06 2024-02-05T04:54:04.673 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.674 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.674 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:04.674 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.002858+0000", 2024-02-05T04:54:04.674 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.6508999999999995e-05 2024-02-05T04:54:04.674 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.674 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.674 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:04.674 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.002869+0000", 2024-02-05T04:54:04.674 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0796e-05 2024-02-05T04:54:04.674 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.674 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.674 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:04.674 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.010372+0000", 2024-02-05T04:54:04.674 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0075036180000000001 2024-02-05T04:54:04.674 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.675 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.675 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:04.675 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.010414+0000", 2024-02-05T04:54:04.675 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.1715000000000003e-05 2024-02-05T04:54:04.675 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:04.675 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:04.675 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:04.675 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.675 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.675 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.0s1 63/62 ECSubWrite(tid=53, reqid=client.4634.0:577, at_version=63'7, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:04.675 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:55.002908+0000", 2024-02-05T04:54:04.675 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 9.6066909109999994, 2024-02-05T04:54:04.675 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0075376250000000001, 2024-02-05T04:54:04.675 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:04.675 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:04.675 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:04.676 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.676 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:04.676 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.002908+0000", 2024-02-05T04:54:04.676 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:04.676 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.676 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.676 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:04.676 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.002908+0000", 2024-02-05T04:54:04.676 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:04.676 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.676 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.676 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:04.676 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.002909+0000", 2024-02-05T04:54:04.676 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0499999999999999e-06 2024-02-05T04:54:04.677 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.677 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.677 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:04.677 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.002993+0000", 2024-02-05T04:54:04.677 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.4120000000000001e-05 2024-02-05T04:54:04.677 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.677 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.677 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:04.677 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.003017+0000", 2024-02-05T04:54:04.677 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.3777999999999998e-05 2024-02-05T04:54:04.677 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.677 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.677 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:04.677 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.003027+0000", 2024-02-05T04:54:04.677 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0208e-05 2024-02-05T04:54:04.677 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.678 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.678 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:04.678 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.003094+0000", 2024-02-05T04:54:04.678 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.6623000000000002e-05 2024-02-05T04:54:04.678 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.678 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.678 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:04.678 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.003107+0000", 2024-02-05T04:54:04.678 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.3329999999999999e-05 2024-02-05T04:54:04.678 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.678 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.678 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:04.678 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.010417+0000", 2024-02-05T04:54:04.678 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.007309945 2024-02-05T04:54:04.678 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.679 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.679 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:04.679 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.010446+0000", 2024-02-05T04:54:04.679 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.8571000000000001e-05 2024-02-05T04:54:04.679 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:04.679 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:04.679 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:04.679 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.679 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.679 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.1s5 63/62 ECSubWrite(tid=265, reqid=client.4634.0:594, at_version=63'53, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:04.679 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:55.004101+0000", 2024-02-05T04:54:04.679 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 9.6054982009999996, 2024-02-05T04:54:04.679 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0098795819999999996, 2024-02-05T04:54:04.679 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:04.679 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:04.679 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:04.680 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.680 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:04.680 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.004101+0000", 2024-02-05T04:54:04.680 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:04.680 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.680 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.680 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:04.680 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.004101+0000", 2024-02-05T04:54:04.680 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:04.680 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.680 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.680 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:04.680 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.004102+0000", 2024-02-05T04:54:04.680 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.5700000000000001e-07 2024-02-05T04:54:04.680 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.681 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.681 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:04.681 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.005989+0000", 2024-02-05T04:54:04.681 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.001887644 2024-02-05T04:54:04.681 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.681 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.681 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:04.681 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.006010+0000", 2024-02-05T04:54:04.681 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.0429e-05 2024-02-05T04:54:04.681 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.681 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.681 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:04.681 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.006021+0000", 2024-02-05T04:54:04.681 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0658e-05 2024-02-05T04:54:04.681 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.682 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.682 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:04.682 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.006094+0000", 2024-02-05T04:54:04.682 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.3268000000000004e-05 2024-02-05T04:54:04.682 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.682 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.682 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:04.682 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.006104+0000", 2024-02-05T04:54:04.682 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0331e-05 2024-02-05T04:54:04.682 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.682 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.682 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:04.682 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.013933+0000", 2024-02-05T04:54:04.682 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0078288880000000009 2024-02-05T04:54:04.682 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.682 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.683 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:04.683 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.013981+0000", 2024-02-05T04:54:04.683 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.7506999999999998e-05 2024-02-05T04:54:04.683 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:04.683 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:04.683 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:04.683 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.683 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.683 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.0s1 63/62 ECSubWrite(tid=54, reqid=client.4634.0:578, at_version=63'8, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:04.683 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:55.005643+0000", 2024-02-05T04:54:04.683 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 9.6039557080000009, 2024-02-05T04:54:04.683 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0084155380000000002, 2024-02-05T04:54:04.683 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:04.683 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:04.683 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:04.683 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.684 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:04.684 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.005643+0000", 2024-02-05T04:54:04.684 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:04.684 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.684 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.684 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:04.684 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.005643+0000", 2024-02-05T04:54:04.684 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:04.684 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.684 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.684 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:04.684 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.005644+0000", 2024-02-05T04:54:04.684 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.6300000000000004e-07 2024-02-05T04:54:04.684 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.684 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.685 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:04.685 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.005730+0000", 2024-02-05T04:54:04.685 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.6080000000000003e-05 2024-02-05T04:54:04.685 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.685 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.685 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:04.685 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.005748+0000", 2024-02-05T04:54:04.685 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.7277000000000001e-05 2024-02-05T04:54:04.685 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.685 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.685 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:04.685 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.005758+0000", 2024-02-05T04:54:04.685 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0030000000000001e-05 2024-02-05T04:54:04.685 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.685 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.686 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:04.686 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.005849+0000", 2024-02-05T04:54:04.686 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.1558000000000007e-05 2024-02-05T04:54:04.686 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.686 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.686 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:04.686 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.005860+0000", 2024-02-05T04:54:04.686 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0671e-05 2024-02-05T04:54:04.686 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.686 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.686 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:04.686 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.014029+0000", 2024-02-05T04:54:04.686 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0081686699999999994 2024-02-05T04:54:04.686 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.686 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.686 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:04.687 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.014059+0000", 2024-02-05T04:54:04.687 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.0389e-05 2024-02-05T04:54:04.687 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:04.687 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:04.687 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:04.687 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.687 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.687 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.4634.0:625 3.es0 3:796382fe:::smithi19639362-29:head [copy-from ver 25] snapc 16=[16,15,12,11,f,b,8,7] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-02-05T04:54:04.687 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:54:00.952558+0000", 2024-02-05T04:54:04.687 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 3.657041505, 2024-02-05T04:54:04.687 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.032663225999999997, 2024-02-05T04:54:04.687 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:04.687 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:04.687 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-02-05T04:54:04.687 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.4634", 2024-02-05T04:54:04.687 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:04.688 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 625 2024-02-05T04:54:04.688 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.688 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:04.688 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.688 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:04.688 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.952558+0000", 2024-02-05T04:54:04.688 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:04.688 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.688 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.688 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:04.688 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.952558+0000", 2024-02-05T04:54:04.688 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:04.688 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.688 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.689 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:04.689 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.952560+0000", 2024-02-05T04:54:04.689 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.0329999999999998e-06 2024-02-05T04:54:04.689 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.689 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.689 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:04.689 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.952568+0000", 2024-02-05T04:54:04.689 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.5629999999999992e-06 2024-02-05T04:54:04.689 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.689 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.689 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:04.689 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.952582+0000", 2024-02-05T04:54:04.689 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.3784999999999999e-05 2024-02-05T04:54:04.689 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.690 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.690 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:04.690 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.952592+0000", 2024-02-05T04:54:04.690 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0132999999999999e-05 2024-02-05T04:54:04.690 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.690 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.690 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:04.690 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.952723+0000", 2024-02-05T04:54:04.690 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000130968 2024-02-05T04:54:04.690 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.690 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.690 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:54:04.690 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.952851+0000", 2024-02-05T04:54:04.690 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000127849 2024-02-05T04:54:04.690 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.691 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.691 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:04.691 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.976458+0000", 2024-02-05T04:54:04.691 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.023607337999999999 2024-02-05T04:54:04.691 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.691 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.691 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:04.691 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.981678+0000", 2024-02-05T04:54:04.691 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.005219942 2024-02-05T04:54:04.691 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.691 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.691 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-02-05T04:54:04.691 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.985162+0000", 2024-02-05T04:54:04.691 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0034837090000000002 2024-02-05T04:54:04.691 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.692 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.692 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:04.692 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.985221+0000", 2024-02-05T04:54:04.692 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.8906000000000001e-05 2024-02-05T04:54:04.692 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:04.692 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:04.692 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:04.692 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.692 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.692 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.4634.0:626 3.es0 3:796382fe:::smithi19639362-29:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e69)", 2024-02-05T04:54:04.692 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:54:00.952658+0000", 2024-02-05T04:54:04.692 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 3.6569415119999999, 2024-02-05T04:54:04.692 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.027128118, 2024-02-05T04:54:04.692 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:04.693 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "started", 2024-02-05T04:54:04.693 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-02-05T04:54:04.693 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.4634", 2024-02-05T04:54:04.693 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:04.693 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 626 2024-02-05T04:54:04.693 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.693 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:04.693 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.693 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:04.693 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.952658+0000", 2024-02-05T04:54:04.693 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:04.693 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.693 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.693 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:04.694 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.952658+0000", 2024-02-05T04:54:04.694 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:04.694 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.694 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.694 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:04.694 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.952659+0000", 2024-02-05T04:54:04.694 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.8000000000000004e-07 2024-02-05T04:54:04.694 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.694 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.694 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:04.694 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.952663+0000", 2024-02-05T04:54:04.694 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.7219999999999999e-06 2024-02-05T04:54:04.694 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.694 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.694 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:04.694 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.952674+0000", 2024-02-05T04:54:04.695 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0886e-05 2024-02-05T04:54:04.695 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.695 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.695 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:04.695 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.952681+0000", 2024-02-05T04:54:04.695 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.2169999999999997e-06 2024-02-05T04:54:04.695 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.695 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.695 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:04.695 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.953055+0000", 2024-02-05T04:54:04.695 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000373254 2024-02-05T04:54:04.695 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.695 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.695 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for blocked object", 2024-02-05T04:54:04.695 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.953099+0000", 2024-02-05T04:54:04.695 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.4583999999999999e-05 2024-02-05T04:54:04.696 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.696 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.696 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:04.696 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.979623+0000", 2024-02-05T04:54:04.696 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.026523794999999999 2024-02-05T04:54:04.696 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.696 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.696 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:54:04.696 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.979709+0000", 2024-02-05T04:54:04.696 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.6291000000000004e-05 2024-02-05T04:54:04.696 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:04.696 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:04.696 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:04.696 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.979786+0000", 2024-02-05T04:54:04.696 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.6489000000000006e-05 2024-02-05T04:54:04.697 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:04.697 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:04.697 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:04.697 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:04.697 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:04.697 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-02-05T04:54:04.698 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-02-05T04:54:04.699 INFO:tasks.ceph.osd.13.smithi091.stderr:2024-02-05T04:54:04.698+0000 7f526fd4d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 34281) UID: 0 2024-02-05T04:54:04.743 INFO:teuthology.orchestra.run.smithi062.stdout:osd.15: {} 2024-02-05T04:54:04.743 INFO:teuthology.orchestra.run.smithi062.stderr:osd.15: osd_enable_op_tracker = '' 2024-02-05T04:54:04.799 INFO:tasks.ceph.osd.8.smithi062.stderr:2024-02-05T04:54:04.798+0000 7f9385bf2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 34693) UID: 0 2024-02-05T04:54:04.805 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-02-05T04:54:04.805 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [], 2024-02-05T04:54:04.805 INFO:teuthology.orchestra.run.smithi150.stdout: "num_ops": 0 2024-02-05T04:54:04.805 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-02-05T04:54:04.814 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-02-05T04:54:04.900 INFO:tasks.ceph.osd.9.smithi091.stderr:2024-02-05T04:54:04.899+0000 7efc18827640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 34279) UID: 0 2024-02-05T04:54:04.920 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-02-05T04:54:04.920 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [], 2024-02-05T04:54:04.920 INFO:teuthology.orchestra.run.smithi150.stdout: "complaint_time": 30, 2024-02-05T04:54:04.920 INFO:teuthology.orchestra.run.smithi150.stdout: "num_blocked_ops": 0 2024-02-05T04:54:04.920 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-02-05T04:54:04.929 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-02-05T04:54:05.001 INFO:tasks.ceph.osd.0.smithi062.stderr:2024-02-05T04:54:05.001+0000 7fece8e03640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34688) UID: 0 2024-02-05T04:54:05.028 INFO:teuthology.orchestra.run.smithi062.stderr:reweighted osd.7 to 1 (10000) 2024-02-05T04:54:05.031 INFO:tasks.rados.rados.0.smithi196.stdout:264: done (5 left) 2024-02-05T04:54:05.031 INFO:tasks.rados.rados.0.smithi196.stdout:266: done (4 left) 2024-02-05T04:54:05.031 INFO:tasks.rados.rados.0.smithi196.stdout:267: done (3 left) 2024-02-05T04:54:05.031 INFO:tasks.rados.rados.0.smithi196.stdout:268: done (2 left) 2024-02-05T04:54:05.031 INFO:tasks.rados.rados.0.smithi196.stdout:269: done (1 left) 2024-02-05T04:54:05.031 INFO:tasks.rados.rados.0.smithi196.stdout:270: done (0 left) 2024-02-05T04:54:05.031 INFO:tasks.rados.rados.0.smithi196.stdout:271: rollback oid 50 current snap is 23 2024-02-05T04:54:05.031 INFO:tasks.rados.rados.0.smithi196.stdout:rollback oid 50 to 6 2024-02-05T04:54:05.031 INFO:tasks.rados.rados.0.smithi196.stdout:272: read oid 43 snap -1 2024-02-05T04:54:05.031 INFO:tasks.rados.rados.0.smithi196.stdout:272: expect (ObjNum 122 snap 22 seq_num 122) 2024-02-05T04:54:05.031 INFO:tasks.rados.rados.0.smithi196.stdout:273: snap_remove snap 9 2024-02-05T04:54:05.037 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-02-05T04:54:05.038 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-02-05T04:54:05.038 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 20, 2024-02-05T04:54:05.038 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 600, 2024-02-05T04:54:05.038 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [ 2024-02-05T04:54:05.039 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.039 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.2s3 37/31 ECSubWrite(tid=22, reqid=client.4634.0:254, at_version=37'15, trim_to=0'0, roll_forward_to=37'10))", 2024-02-05T04:54:05.039 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:29.208543+0000", 2024-02-05T04:54:05.039 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 35.827679355999997, 2024-02-05T04:54:05.039 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0047425410000000003, 2024-02-05T04:54:05.039 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:05.039 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:05.039 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:05.039 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.039 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:05.039 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.208543+0000", 2024-02-05T04:54:05.039 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:05.039 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.040 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.040 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:05.040 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.208543+0000", 2024-02-05T04:54:05.040 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:05.040 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.040 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.040 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:05.040 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.208544+0000", 2024-02-05T04:54:05.040 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.6700000000000003e-07 2024-02-05T04:54:05.040 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.040 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.040 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:05.040 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.208803+0000", 2024-02-05T04:54:05.040 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00025897499999999999 2024-02-05T04:54:05.040 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.041 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.041 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:05.041 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.208815+0000", 2024-02-05T04:54:05.041 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1630999999999999e-05 2024-02-05T04:54:05.041 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.041 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.041 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:05.041 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.208821+0000", 2024-02-05T04:54:05.041 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.331e-06 2024-02-05T04:54:05.041 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.041 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.041 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:05.041 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.208875+0000", 2024-02-05T04:54:05.041 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.3909999999999997e-05 2024-02-05T04:54:05.042 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.042 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.042 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:05.042 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.208884+0000", 2024-02-05T04:54:05.042 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.4169999999999998e-06 2024-02-05T04:54:05.042 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.042 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.042 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:05.042 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.213250+0000", 2024-02-05T04:54:05.042 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0043660360000000002 2024-02-05T04:54:05.042 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.042 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.042 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:05.042 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.213286+0000", 2024-02-05T04:54:05.043 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.5673999999999998e-05 2024-02-05T04:54:05.043 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.043 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:05.043 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.043 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.043 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.043 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.7s1 57/31 ECSubWrite(tid=238, reqid=osd.13.0:238, at_version=57'27, trim_to=0'0, roll_forward_to=52'25))", 2024-02-05T04:54:05.043 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:48.949954+0000", 2024-02-05T04:54:05.043 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 16.086268523000001, 2024-02-05T04:54:05.043 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0068144089999999996, 2024-02-05T04:54:05.043 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:05.043 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:05.043 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:05.043 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.044 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:05.044 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.949954+0000", 2024-02-05T04:54:05.044 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:05.044 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.044 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.044 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:05.044 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.949954+0000", 2024-02-05T04:54:05.044 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:05.044 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.044 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.044 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:05.044 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.949955+0000", 2024-02-05T04:54:05.044 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.1300000000000001e-07 2024-02-05T04:54:05.044 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.045 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.045 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:05.045 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.949980+0000", 2024-02-05T04:54:05.045 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.5418999999999998e-05 2024-02-05T04:54:05.045 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.045 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.045 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:05.045 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.949992+0000", 2024-02-05T04:54:05.045 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1776e-05 2024-02-05T04:54:05.045 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.045 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.045 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:05.045 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.950008+0000", 2024-02-05T04:54:05.045 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.5930000000000002e-05 2024-02-05T04:54:05.045 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.045 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.046 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:05.046 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.950393+0000", 2024-02-05T04:54:05.046 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00038503799999999998 2024-02-05T04:54:05.046 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.046 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.046 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:05.046 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.950403+0000", 2024-02-05T04:54:05.046 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0564000000000001e-05 2024-02-05T04:54:05.046 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.046 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.046 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:05.046 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.956728+0000", 2024-02-05T04:54:05.046 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0063242669999999997 2024-02-05T04:54:05.046 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.046 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.047 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:05.047 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.956769+0000", 2024-02-05T04:54:05.047 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.1001999999999998e-05 2024-02-05T04:54:05.047 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.047 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:05.047 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.047 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.047 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.047 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.7s1 57/31 ECSubWrite(tid=239, reqid=osd.13.0:239, at_version=57'29, trim_to=0'0, roll_forward_to=52'25))", 2024-02-05T04:54:05.047 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:48.950347+0000", 2024-02-05T04:54:05.047 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 16.085875411, 2024-02-05T04:54:05.047 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0071722510000000001, 2024-02-05T04:54:05.047 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:05.047 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:05.047 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:05.048 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.048 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:05.048 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.950347+0000", 2024-02-05T04:54:05.048 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:05.048 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.048 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.048 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:05.048 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.950347+0000", 2024-02-05T04:54:05.048 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:05.048 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.048 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.048 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:05.048 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.950348+0000", 2024-02-05T04:54:05.048 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.5000000000000003e-07 2024-02-05T04:54:05.049 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.049 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.049 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:05.049 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.950384+0000", 2024-02-05T04:54:05.049 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.5756999999999997e-05 2024-02-05T04:54:05.049 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.049 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.049 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:05.049 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.950396+0000", 2024-02-05T04:54:05.049 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2816999999999999e-05 2024-02-05T04:54:05.049 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.050 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.050 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:05.050 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.950405+0000", 2024-02-05T04:54:05.050 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.8759999999999994e-06 2024-02-05T04:54:05.050 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.050 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.050 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:05.050 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.950989+0000", 2024-02-05T04:54:05.050 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00058388199999999998 2024-02-05T04:54:05.050 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.050 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.050 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:05.050 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.951008+0000", 2024-02-05T04:54:05.050 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.9276999999999999e-05 2024-02-05T04:54:05.050 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.051 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.051 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:05.051 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.957477+0000", 2024-02-05T04:54:05.051 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0064689830000000002 2024-02-05T04:54:05.051 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.051 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.051 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:05.051 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.957520+0000", 2024-02-05T04:54:05.051 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.2109e-05 2024-02-05T04:54:05.051 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.051 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:05.051 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.051 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.051 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.051 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.cs4 57/31 ECSubWrite(tid=90, reqid=osd.11.0:90, at_version=57'13, trim_to=0'0, roll_forward_to=52'11))", 2024-02-05T04:54:05.052 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:48.950812+0000", 2024-02-05T04:54:05.052 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 16.085411125, 2024-02-05T04:54:05.052 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.006633066, 2024-02-05T04:54:05.052 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:05.052 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:05.052 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:05.052 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.052 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:05.052 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.950812+0000", 2024-02-05T04:54:05.052 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:05.052 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.052 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.052 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:05.053 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.950812+0000", 2024-02-05T04:54:05.053 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:05.053 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.053 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.053 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:05.053 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.950812+0000", 2024-02-05T04:54:05.053 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.9799999999999999e-07 2024-02-05T04:54:05.053 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.053 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.053 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:05.053 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.950838+0000", 2024-02-05T04:54:05.053 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.5690999999999998e-05 2024-02-05T04:54:05.053 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.053 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.054 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:05.054 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.951042+0000", 2024-02-05T04:54:05.054 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00020407499999999999 2024-02-05T04:54:05.054 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.054 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.054 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:05.054 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.951051+0000", 2024-02-05T04:54:05.054 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.5189999999999998e-06 2024-02-05T04:54:05.054 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.054 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.054 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:05.054 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.951124+0000", 2024-02-05T04:54:05.054 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.2348999999999995e-05 2024-02-05T04:54:05.054 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.054 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.055 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:05.055 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.951133+0000", 2024-02-05T04:54:05.055 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.4749999999999999e-06 2024-02-05T04:54:05.055 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.055 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.055 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:05.055 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.957397+0000", 2024-02-05T04:54:05.055 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0062641709999999998 2024-02-05T04:54:05.055 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.055 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.055 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:05.055 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.957445+0000", 2024-02-05T04:54:05.055 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.7487999999999997e-05 2024-02-05T04:54:05.055 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.056 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:05.056 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.056 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.056 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.056 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.5s6 58/57 ECSubWrite(tid=60, reqid=osd.15.0:60, at_version=58'40, trim_to=0'0, roll_forward_to=58'38))", 2024-02-05T04:54:05.056 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:49.969350+0000", 2024-02-05T04:54:05.056 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 15.066873048, 2024-02-05T04:54:05.056 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0079587159999999994, 2024-02-05T04:54:05.056 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:05.056 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:05.056 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:05.056 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.056 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:05.056 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.969350+0000", 2024-02-05T04:54:05.057 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:05.057 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.057 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.057 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:05.057 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.969350+0000", 2024-02-05T04:54:05.057 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:05.057 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.057 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.057 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:05.057 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.969350+0000", 2024-02-05T04:54:05.057 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.6400000000000002e-07 2024-02-05T04:54:05.057 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.057 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.057 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:05.057 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.969365+0000", 2024-02-05T04:54:05.058 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.4589e-05 2024-02-05T04:54:05.058 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.058 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.058 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:05.058 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.969375+0000", 2024-02-05T04:54:05.058 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.7550000000000006e-06 2024-02-05T04:54:05.058 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.058 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.058 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:05.058 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.969380+0000", 2024-02-05T04:54:05.058 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.9529999999999998e-06 2024-02-05T04:54:05.058 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.058 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.058 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:05.059 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.969430+0000", 2024-02-05T04:54:05.059 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.0139999999999998e-05 2024-02-05T04:54:05.059 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.059 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.059 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:05.059 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.969439+0000", 2024-02-05T04:54:05.059 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.918e-06 2024-02-05T04:54:05.059 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.059 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.059 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:05.059 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.977256+0000", 2024-02-05T04:54:05.059 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0078177799999999999 2024-02-05T04:54:05.059 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.059 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.059 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:05.060 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.977308+0000", 2024-02-05T04:54:05.060 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.2017e-05 2024-02-05T04:54:05.060 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.060 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:05.060 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.060 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.060 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.060 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.7s1 58/31 ECSubWrite(tid=254, reqid=client.4634.0:520, at_version=0'0, trim_to=0'0, roll_forward_to=58'30))", 2024-02-05T04:54:05.060 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:49.972620+0000", 2024-02-05T04:54:05.060 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 15.063603059, 2024-02-05T04:54:05.060 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0073166899999999998, 2024-02-05T04:54:05.060 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:05.060 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:05.060 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:05.060 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.061 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:05.061 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.972620+0000", 2024-02-05T04:54:05.061 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:05.061 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.061 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.061 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:05.061 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.972620+0000", 2024-02-05T04:54:05.061 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:05.061 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.061 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.061 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:05.061 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.972621+0000", 2024-02-05T04:54:05.061 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.3799999999999996e-07 2024-02-05T04:54:05.061 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.061 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.061 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:05.062 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.972632+0000", 2024-02-05T04:54:05.062 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1358999999999999e-05 2024-02-05T04:54:05.062 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.062 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.062 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:05.062 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.972650+0000", 2024-02-05T04:54:05.062 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.7679000000000001e-05 2024-02-05T04:54:05.062 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.062 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.062 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:05.062 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.972658+0000", 2024-02-05T04:54:05.062 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.9099999999999994e-06 2024-02-05T04:54:05.062 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.063 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.063 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:05.063 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.972728+0000", 2024-02-05T04:54:05.063 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.9334999999999999e-05 2024-02-05T04:54:05.063 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.063 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.063 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:05.063 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.972739+0000", 2024-02-05T04:54:05.063 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0839000000000001e-05 2024-02-05T04:54:05.063 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.063 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.063 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:05.063 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.979838+0000", 2024-02-05T04:54:05.063 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0070992340000000003 2024-02-05T04:54:05.063 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.064 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.064 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:05.064 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.979936+0000", 2024-02-05T04:54:05.064 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.8496e-05 2024-02-05T04:54:05.064 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.064 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:05.064 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.064 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.064 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.064 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.es5 63/57 ECSubWrite(tid=77, reqid=osd.2.0:77, at_version=63'58, trim_to=0'0, roll_forward_to=61'56))", 2024-02-05T04:54:05.064 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:54.981521+0000", 2024-02-05T04:54:05.064 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 10.054701783000001, 2024-02-05T04:54:05.064 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0079934140000000008, 2024-02-05T04:54:05.064 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:05.064 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:05.065 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:05.065 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.065 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:05.065 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.981521+0000", 2024-02-05T04:54:05.065 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:05.065 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.065 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.065 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:05.065 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.981521+0000", 2024-02-05T04:54:05.065 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:05.065 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.065 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.065 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:05.065 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.981521+0000", 2024-02-05T04:54:05.065 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.7799999999999997e-07 2024-02-05T04:54:05.066 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.066 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.066 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:05.066 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.981546+0000", 2024-02-05T04:54:05.066 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.4807000000000002e-05 2024-02-05T04:54:05.066 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.066 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.066 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:05.066 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.981562+0000", 2024-02-05T04:54:05.066 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.5770999999999999e-05 2024-02-05T04:54:05.066 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.066 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.066 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:05.066 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.981573+0000", 2024-02-05T04:54:05.066 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1292e-05 2024-02-05T04:54:05.067 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.067 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.067 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:05.067 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.981638+0000", 2024-02-05T04:54:05.067 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.4795999999999994e-05 2024-02-05T04:54:05.067 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.067 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.067 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:05.067 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.981650+0000", 2024-02-05T04:54:05.067 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1732e-05 2024-02-05T04:54:05.067 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.067 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.067 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:05.067 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.989474+0000", 2024-02-05T04:54:05.067 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0078242590000000001 2024-02-05T04:54:05.068 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.068 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.068 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:05.068 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.989514+0000", 2024-02-05T04:54:05.068 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.0479000000000003e-05 2024-02-05T04:54:05.068 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.068 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:05.068 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.068 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.068 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.068 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.bs4 63/57 ECSubWrite(tid=263, reqid=osd.13.0:263, at_version=63'58, trim_to=0'0, roll_forward_to=61'56))", 2024-02-05T04:54:05.068 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:54.983110+0000", 2024-02-05T04:54:05.068 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 10.053113213, 2024-02-05T04:54:05.068 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.01205758, 2024-02-05T04:54:05.068 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:05.069 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:05.069 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:05.069 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.069 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:05.069 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.983110+0000", 2024-02-05T04:54:05.069 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:05.069 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.069 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.069 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:05.069 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.983110+0000", 2024-02-05T04:54:05.069 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:05.069 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.069 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.069 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:05.069 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.983110+0000", 2024-02-05T04:54:05.069 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.16e-07 2024-02-05T04:54:05.070 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.070 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.070 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:05.070 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.983133+0000", 2024-02-05T04:54:05.070 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.3490999999999999e-05 2024-02-05T04:54:05.070 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.070 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.070 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:05.070 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.983143+0000", 2024-02-05T04:54:05.070 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0101e-05 2024-02-05T04:54:05.070 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.070 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.070 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:05.070 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.983154+0000", 2024-02-05T04:54:05.070 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0210999999999999e-05 2024-02-05T04:54:05.071 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.071 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.071 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:05.071 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.983231+0000", 2024-02-05T04:54:05.071 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.7187000000000005e-05 2024-02-05T04:54:05.071 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.071 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.071 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:05.071 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.983241+0000", 2024-02-05T04:54:05.071 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.8649999999999996e-06 2024-02-05T04:54:05.071 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.071 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.071 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:05.071 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.995135+0000", 2024-02-05T04:54:05.071 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.011894604 2024-02-05T04:54:05.072 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.072 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.072 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:05.072 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.995167+0000", 2024-02-05T04:54:05.072 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.1905e-05 2024-02-05T04:54:05.072 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.072 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:05.072 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.072 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.072 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.072 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.cs4 63/31 ECSubWrite(tid=120, reqid=osd.11.0:120, at_version=63'25, trim_to=0'0, roll_forward_to=62'23))", 2024-02-05T04:54:05.072 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:54.986409+0000", 2024-02-05T04:54:05.072 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 10.049813657, 2024-02-05T04:54:05.072 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.012297479, 2024-02-05T04:54:05.072 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:05.073 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:05.073 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:05.073 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.073 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:05.073 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.986409+0000", 2024-02-05T04:54:05.073 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:05.073 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.073 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.073 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:05.073 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.986409+0000", 2024-02-05T04:54:05.073 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:05.073 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.073 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.073 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:05.073 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.986409+0000", 2024-02-05T04:54:05.073 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.3799999999999999e-07 2024-02-05T04:54:05.074 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.074 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.074 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:05.074 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.986427+0000", 2024-02-05T04:54:05.074 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.7631000000000001e-05 2024-02-05T04:54:05.074 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.074 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.074 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:05.074 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.986437+0000", 2024-02-05T04:54:05.074 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0434999999999999e-05 2024-02-05T04:54:05.074 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.074 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.074 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:05.074 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.986446+0000", 2024-02-05T04:54:05.075 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.1060000000000006e-06 2024-02-05T04:54:05.075 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.075 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.075 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:05.075 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.986514+0000", 2024-02-05T04:54:05.075 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.7785000000000002e-05 2024-02-05T04:54:05.075 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.075 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.075 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:05.075 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.986525+0000", 2024-02-05T04:54:05.075 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0322e-05 2024-02-05T04:54:05.075 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.075 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.075 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:05.075 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.998666+0000", 2024-02-05T04:54:05.076 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.012141175000000001 2024-02-05T04:54:05.076 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.076 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.076 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:05.076 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.998707+0000", 2024-02-05T04:54:05.076 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.0886999999999997e-05 2024-02-05T04:54:05.076 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.076 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:05.076 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.076 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.076 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.076 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.8s6 63/62 ECSubWrite(tid=80, reqid=client.4634.0:591, at_version=63'25, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:05.076 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:54.987410+0000", 2024-02-05T04:54:05.076 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 10.048812291000001, 2024-02-05T04:54:05.076 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.011470258000000001, 2024-02-05T04:54:05.077 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:05.077 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:05.077 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:05.077 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.077 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:05.077 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.987410+0000", 2024-02-05T04:54:05.077 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:05.077 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.077 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.077 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:05.077 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.987410+0000", 2024-02-05T04:54:05.077 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:05.077 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.077 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.077 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:05.078 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.987411+0000", 2024-02-05T04:54:05.078 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.85e-07 2024-02-05T04:54:05.078 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.078 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.078 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:05.078 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.987434+0000", 2024-02-05T04:54:05.078 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.234e-05 2024-02-05T04:54:05.078 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.078 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.078 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:05.078 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.987451+0000", 2024-02-05T04:54:05.078 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.7529000000000001e-05 2024-02-05T04:54:05.078 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.078 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.078 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:05.079 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.987461+0000", 2024-02-05T04:54:05.079 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.6129999999999993e-06 2024-02-05T04:54:05.079 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.079 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.079 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:05.079 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.987548+0000", 2024-02-05T04:54:05.079 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.6891000000000005e-05 2024-02-05T04:54:05.079 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.079 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.079 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:05.079 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.987558+0000", 2024-02-05T04:54:05.079 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.9369999999999999e-06 2024-02-05T04:54:05.079 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.079 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.079 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:05.080 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.998845+0000", 2024-02-05T04:54:05.080 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.011287108000000001 2024-02-05T04:54:05.080 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.080 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.080 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:05.080 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.998881+0000", 2024-02-05T04:54:05.080 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.5954999999999997e-05 2024-02-05T04:54:05.080 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.080 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:05.080 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.080 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.080 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.080 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.8s6 63/62 ECSubWrite(tid=81, reqid=client.4634.0:592, at_version=63'26, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:05.080 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:54.989269+0000", 2024-02-05T04:54:05.080 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 10.046953464, 2024-02-05T04:54:05.081 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.01244234, 2024-02-05T04:54:05.081 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:05.081 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:05.081 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:05.081 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.081 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:05.081 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.989269+0000", 2024-02-05T04:54:05.081 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:05.081 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.081 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.081 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:05.081 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.989269+0000", 2024-02-05T04:54:05.081 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:05.081 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.081 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.082 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:05.082 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.989270+0000", 2024-02-05T04:54:05.082 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.0200000000000001e-07 2024-02-05T04:54:05.082 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.082 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.082 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:05.082 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.989290+0000", 2024-02-05T04:54:05.082 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.9978000000000001e-05 2024-02-05T04:54:05.082 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.082 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.082 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:05.082 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.989304+0000", 2024-02-05T04:54:05.082 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.3606e-05 2024-02-05T04:54:05.082 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.082 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.083 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:05.083 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.989311+0000", 2024-02-05T04:54:05.083 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.6650000000000003e-06 2024-02-05T04:54:05.083 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.083 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.083 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:05.083 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.990422+0000", 2024-02-05T04:54:05.083 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0011108380000000001 2024-02-05T04:54:05.083 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.083 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.083 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:05.083 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.990432+0000", 2024-02-05T04:54:05.083 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.4569999999999994e-06 2024-02-05T04:54:05.083 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.083 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.084 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:05.084 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.001676+0000", 2024-02-05T04:54:05.084 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.011244413999999999 2024-02-05T04:54:05.084 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.084 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.084 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:05.084 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.001712+0000", 2024-02-05T04:54:05.084 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.5580000000000002e-05 2024-02-05T04:54:05.084 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.084 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:05.084 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.084 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.084 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.084 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.8s6 63/62 ECSubWrite(tid=82, reqid=client.4634.0:593, at_version=63'27, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:05.084 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:54.990293+0000", 2024-02-05T04:54:05.085 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 10.045929513000001, 2024-02-05T04:54:05.085 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.011444138, 2024-02-05T04:54:05.085 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:05.085 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:05.085 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:05.085 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.085 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:05.085 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.990293+0000", 2024-02-05T04:54:05.085 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:05.085 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.085 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.085 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:05.085 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.990293+0000", 2024-02-05T04:54:05.085 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:05.085 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.086 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.086 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:05.086 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.990294+0000", 2024-02-05T04:54:05.086 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.0599999999999999e-07 2024-02-05T04:54:05.086 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.086 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.086 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:05.086 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.990315+0000", 2024-02-05T04:54:05.086 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.0828000000000001e-05 2024-02-05T04:54:05.086 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.086 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.086 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:05.086 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.990326+0000", 2024-02-05T04:54:05.086 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.137e-05 2024-02-05T04:54:05.086 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.086 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.087 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:05.087 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.990336+0000", 2024-02-05T04:54:05.087 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.1900000000000001e-06 2024-02-05T04:54:05.087 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.087 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.087 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:05.087 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.995306+0000", 2024-02-05T04:54:05.087 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0049702549999999998 2024-02-05T04:54:05.087 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.087 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.087 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:05.087 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.995318+0000", 2024-02-05T04:54:05.087 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2309e-05 2024-02-05T04:54:05.087 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.087 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.088 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:05.088 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.001716+0000", 2024-02-05T04:54:05.088 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0063976809999999997 2024-02-05T04:54:05.088 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.088 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.088 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:05.088 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.001737+0000", 2024-02-05T04:54:05.088 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.1599000000000001e-05 2024-02-05T04:54:05.088 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.088 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:05.088 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.088 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.088 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.088 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.es5 63/57 ECSubWrite(tid=78, reqid=osd.2.0:77, at_version=0'0, trim_to=0'0, roll_forward_to=63'58))", 2024-02-05T04:54:05.088 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:54.991809+0000", 2024-02-05T04:54:05.089 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 10.044413492, 2024-02-05T04:54:05.089 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0069556749999999997, 2024-02-05T04:54:05.089 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:05.089 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:05.089 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:05.089 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.089 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:05.089 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.991809+0000", 2024-02-05T04:54:05.089 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:05.089 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.089 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.089 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:05.089 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.991809+0000", 2024-02-05T04:54:05.089 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:05.089 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.089 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.090 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:05.090 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.991810+0000", 2024-02-05T04:54:05.090 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.1699999999999997e-07 2024-02-05T04:54:05.090 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.090 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.090 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:05.090 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.991819+0000", 2024-02-05T04:54:05.090 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.2909999999999996e-06 2024-02-05T04:54:05.090 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.090 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.090 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:05.090 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.991835+0000", 2024-02-05T04:54:05.090 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.525e-05 2024-02-05T04:54:05.090 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.090 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.091 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:05.091 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.991843+0000", 2024-02-05T04:54:05.091 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.2379999999999997e-06 2024-02-05T04:54:05.091 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.091 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.091 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:05.091 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.991907+0000", 2024-02-05T04:54:05.091 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.4315999999999999e-05 2024-02-05T04:54:05.091 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.091 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.091 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:05.091 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.991917+0000", 2024-02-05T04:54:05.091 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0322999999999999e-05 2024-02-05T04:54:05.091 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.091 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.091 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:05.092 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.998728+0000", 2024-02-05T04:54:05.092 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.006810567 2024-02-05T04:54:05.092 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.092 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.092 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:05.092 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.998765+0000", 2024-02-05T04:54:05.092 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.6872999999999997e-05 2024-02-05T04:54:05.092 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.092 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:05.092 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.092 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.092 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.092 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.5s6 63/62 ECSubWrite(tid=62, reqid=osd.15.0:62, at_version=63'42, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:05.092 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:54.998156+0000", 2024-02-05T04:54:05.092 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 10.038066583999999, 2024-02-05T04:54:05.093 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.005925006, 2024-02-05T04:54:05.093 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:05.093 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:05.093 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:05.093 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.093 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:05.093 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.998156+0000", 2024-02-05T04:54:05.093 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:05.093 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.093 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.093 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:05.093 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.998156+0000", 2024-02-05T04:54:05.093 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:05.093 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.093 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.094 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:05.094 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.998157+0000", 2024-02-05T04:54:05.094 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.39e-07 2024-02-05T04:54:05.094 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.094 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.094 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:05.094 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.998181+0000", 2024-02-05T04:54:05.094 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.3986e-05 2024-02-05T04:54:05.094 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.094 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.094 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:05.094 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.998190+0000", 2024-02-05T04:54:05.094 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.6840000000000008e-06 2024-02-05T04:54:05.094 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.094 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.095 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:05.095 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.998199+0000", 2024-02-05T04:54:05.095 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.9479999999999997e-06 2024-02-05T04:54:05.095 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.095 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.095 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:05.095 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.998328+0000", 2024-02-05T04:54:05.095 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00012877 2024-02-05T04:54:05.095 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.095 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.095 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:05.095 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.998339+0000", 2024-02-05T04:54:05.095 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0824e-05 2024-02-05T04:54:05.095 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.095 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.096 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:05.096 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.004026+0000", 2024-02-05T04:54:05.096 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0056868789999999997 2024-02-05T04:54:05.096 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.096 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.096 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:05.096 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.004081+0000", 2024-02-05T04:54:05.096 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.5476000000000002e-05 2024-02-05T04:54:05.096 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.096 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:05.096 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.096 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.096 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.096 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.12s3 63/62 ECSubWrite(tid=73, reqid=osd.12.0:73, at_version=63'55, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:05.096 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:54.999034+0000", 2024-02-05T04:54:05.097 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 10.037188902, 2024-02-05T04:54:05.097 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0064765760000000004, 2024-02-05T04:54:05.097 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:05.097 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:05.097 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:05.097 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.097 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:05.097 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.999034+0000", 2024-02-05T04:54:05.097 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:05.097 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.097 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.097 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:05.097 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.999034+0000", 2024-02-05T04:54:05.097 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:05.097 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.098 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.098 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:05.098 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.999035+0000", 2024-02-05T04:54:05.098 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.4799999999999997e-07 2024-02-05T04:54:05.098 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.098 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.098 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:05.098 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.999056+0000", 2024-02-05T04:54:05.098 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.0948999999999999e-05 2024-02-05T04:54:05.098 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.098 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.098 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:05.098 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.999068+0000", 2024-02-05T04:54:05.098 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2653999999999999e-05 2024-02-05T04:54:05.098 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.098 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.099 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:05.099 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.999077+0000", 2024-02-05T04:54:05.099 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.6049999999999998e-06 2024-02-05T04:54:05.099 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.099 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.099 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:05.099 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.999212+0000", 2024-02-05T04:54:05.099 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00013492199999999999 2024-02-05T04:54:05.099 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.099 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.099 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:05.099 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.999233+0000", 2024-02-05T04:54:05.099 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.1662999999999999e-05 2024-02-05T04:54:05.099 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.099 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.099 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:05.100 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.005429+0000", 2024-02-05T04:54:05.100 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0061952750000000001 2024-02-05T04:54:05.100 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.100 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.100 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:05.100 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.005510+0000", 2024-02-05T04:54:05.100 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.1760000000000003e-05 2024-02-05T04:54:05.100 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.100 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:05.100 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.100 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.100 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.100 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.0s5 63/62 ECSubWrite(tid=53, reqid=client.4634.0:577, at_version=63'7, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:05.100 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:55.002912+0000", 2024-02-05T04:54:05.101 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 10.033311060999999, 2024-02-05T04:54:05.101 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0071061939999999997, 2024-02-05T04:54:05.101 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:05.101 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:05.101 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:05.101 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.101 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:05.101 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.002912+0000", 2024-02-05T04:54:05.101 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:05.101 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.101 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.101 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:05.101 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.002912+0000", 2024-02-05T04:54:05.101 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:05.101 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.101 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.102 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:05.102 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.002913+0000", 2024-02-05T04:54:05.102 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.5000000000000001e-07 2024-02-05T04:54:05.102 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.102 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.102 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:05.102 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.003096+0000", 2024-02-05T04:54:05.102 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00018306900000000001 2024-02-05T04:54:05.102 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.102 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.102 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:05.102 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.003109+0000", 2024-02-05T04:54:05.102 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.3458000000000001e-05 2024-02-05T04:54:05.102 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.102 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.103 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:05.103 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.003118+0000", 2024-02-05T04:54:05.103 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.473e-06 2024-02-05T04:54:05.103 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.103 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.103 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:05.103 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.003197+0000", 2024-02-05T04:54:05.103 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.9375999999999994e-05 2024-02-05T04:54:05.103 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.103 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.103 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:05.103 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.003204+0000", 2024-02-05T04:54:05.103 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.4490000000000002e-06 2024-02-05T04:54:05.103 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.103 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.104 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:05.104 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.009967+0000", 2024-02-05T04:54:05.104 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0067627069999999997 2024-02-05T04:54:05.104 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.104 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.104 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:05.104 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.010018+0000", 2024-02-05T04:54:05.104 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.0812000000000001e-05 2024-02-05T04:54:05.104 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.104 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:05.104 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.104 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.104 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.104 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.0s5 63/62 ECSubWrite(tid=54, reqid=client.4634.0:578, at_version=63'8, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:05.105 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:55.005715+0000", 2024-02-05T04:54:05.105 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 10.030507632000001, 2024-02-05T04:54:05.105 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0051116139999999996, 2024-02-05T04:54:05.105 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:05.105 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:05.105 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:05.105 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.105 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:05.105 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.005715+0000", 2024-02-05T04:54:05.105 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:05.105 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.105 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.105 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:05.105 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.005715+0000", 2024-02-05T04:54:05.106 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:05.106 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.106 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.106 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:05.106 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.005716+0000", 2024-02-05T04:54:05.106 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.2200000000000003e-07 2024-02-05T04:54:05.106 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.106 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.106 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:05.106 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.005766+0000", 2024-02-05T04:54:05.106 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.0424999999999999e-05 2024-02-05T04:54:05.106 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.106 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.106 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:05.106 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.005781+0000", 2024-02-05T04:54:05.107 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.4894e-05 2024-02-05T04:54:05.107 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.107 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.107 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:05.107 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.005791+0000", 2024-02-05T04:54:05.107 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0154e-05 2024-02-05T04:54:05.107 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.107 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.107 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:05.107 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.005847+0000", 2024-02-05T04:54:05.107 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.5223999999999998e-05 2024-02-05T04:54:05.107 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.107 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.107 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:05.107 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.005858+0000", 2024-02-05T04:54:05.107 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.119e-05 2024-02-05T04:54:05.108 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.108 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.108 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:05.108 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.010779+0000", 2024-02-05T04:54:05.108 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0049212830000000003 2024-02-05T04:54:05.108 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.108 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.108 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:05.108 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.010827+0000", 2024-02-05T04:54:05.108 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.7621999999999999e-05 2024-02-05T04:54:05.108 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.108 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:05.108 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.108 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.109 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.109 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.12s3 68/67 ECSubWrite(tid=85, reqid=osd.2.0:85, at_version=68'57, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:05.109 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:59.960865+0000", 2024-02-05T04:54:05.109 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 5.0753576689999997, 2024-02-05T04:54:05.109 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0058713070000000001, 2024-02-05T04:54:05.109 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:05.109 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:05.109 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:05.109 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.109 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:05.109 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:59.960865+0000", 2024-02-05T04:54:05.109 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:05.109 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.109 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.109 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:05.110 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:59.960865+0000", 2024-02-05T04:54:05.110 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:05.110 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.110 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.110 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:05.110 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:59.960866+0000", 2024-02-05T04:54:05.110 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.8199999999999999e-07 2024-02-05T04:54:05.110 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.110 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.110 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:05.110 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:59.960889+0000", 2024-02-05T04:54:05.110 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.3527999999999999e-05 2024-02-05T04:54:05.110 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.110 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.110 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:05.111 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:59.960909+0000", 2024-02-05T04:54:05.111 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.9154e-05 2024-02-05T04:54:05.111 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.111 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.111 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:05.111 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:59.960917+0000", 2024-02-05T04:54:05.111 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.5930000000000006e-06 2024-02-05T04:54:05.111 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.111 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.111 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:05.111 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:59.960974+0000", 2024-02-05T04:54:05.111 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.6956000000000002e-05 2024-02-05T04:54:05.111 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.111 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.111 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:05.111 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:59.960985+0000", 2024-02-05T04:54:05.112 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1100999999999999e-05 2024-02-05T04:54:05.112 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.112 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.112 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:05.112 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:59.966699+0000", 2024-02-05T04:54:05.112 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0057133590000000003 2024-02-05T04:54:05.112 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.112 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.112 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:05.112 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:59.966736+0000", 2024-02-05T04:54:05.112 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.7833999999999997e-05 2024-02-05T04:54:05.112 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.112 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:05.112 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.112 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.113 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.113 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.es5 69/57 ECSubWrite(tid=87, reqid=client.4634.0:625, at_version=69'62, trim_to=0'0, roll_forward_to=66'60))", 2024-02-05T04:54:05.113 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:54:00.976648+0000", 2024-02-05T04:54:05.113 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 4.0595746630000002, 2024-02-05T04:54:05.113 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0056872399999999997, 2024-02-05T04:54:05.113 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:05.113 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:05.113 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:05.113 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.113 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:05.113 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.976648+0000", 2024-02-05T04:54:05.113 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:05.113 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.113 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.113 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:05.114 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.976648+0000", 2024-02-05T04:54:05.114 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:05.114 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.114 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.114 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:05.114 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.976648+0000", 2024-02-05T04:54:05.114 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.3699999999999999e-07 2024-02-05T04:54:05.114 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.114 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.114 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:05.114 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.977284+0000", 2024-02-05T04:54:05.114 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00063593999999999999 2024-02-05T04:54:05.114 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.114 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.115 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:05.115 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.977302+0000", 2024-02-05T04:54:05.115 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.7927e-05 2024-02-05T04:54:05.115 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.115 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.115 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:05.115 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.977311+0000", 2024-02-05T04:54:05.115 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.9609999999999994e-06 2024-02-05T04:54:05.115 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.115 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.115 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:05.115 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.977422+0000", 2024-02-05T04:54:05.115 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000110857 2024-02-05T04:54:05.115 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.116 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.116 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:05.116 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.977446+0000", 2024-02-05T04:54:05.116 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.4363e-05 2024-02-05T04:54:05.116 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.116 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.116 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:05.116 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.982303+0000", 2024-02-05T04:54:05.116 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0048566720000000002 2024-02-05T04:54:05.116 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.116 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.116 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:05.116 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.982335+0000", 2024-02-05T04:54:05.116 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.2283000000000002e-05 2024-02-05T04:54:05.117 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.117 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:05.117 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.117 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.117 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.117 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.2s3 69/62 ECSubWrite(tid=75, reqid=client.4634.0:641, at_version=69'55, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:05.117 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:54:01.014345+0000", 2024-02-05T04:54:05.117 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 4.0218777899999996, 2024-02-05T04:54:05.117 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0064386449999999998, 2024-02-05T04:54:05.117 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:05.117 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:05.117 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:05.117 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.117 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:05.117 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:01.014345+0000", 2024-02-05T04:54:05.118 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:05.118 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.118 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.118 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:05.118 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:01.014345+0000", 2024-02-05T04:54:05.118 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:05.118 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.118 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.118 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:05.118 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:01.014345+0000", 2024-02-05T04:54:05.118 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.1800000000000002e-07 2024-02-05T04:54:05.118 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.118 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.118 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:05.118 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:01.017228+0000", 2024-02-05T04:54:05.119 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.002882332 2024-02-05T04:54:05.119 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.119 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.119 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:05.119 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:01.017246+0000", 2024-02-05T04:54:05.119 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.8712e-05 2024-02-05T04:54:05.119 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.119 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.119 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:05.119 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:01.017257+0000", 2024-02-05T04:54:05.119 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0868e-05 2024-02-05T04:54:05.119 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.119 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.119 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:05.119 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:01.017346+0000", 2024-02-05T04:54:05.119 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.8517999999999995e-05 2024-02-05T04:54:05.120 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.120 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.120 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:05.120 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:01.017358+0000", 2024-02-05T04:54:05.120 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2571e-05 2024-02-05T04:54:05.120 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.120 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.120 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:05.120 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:01.020745+0000", 2024-02-05T04:54:05.120 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0033866479999999999 2024-02-05T04:54:05.120 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.120 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.120 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:05.120 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:01.020784+0000", 2024-02-05T04:54:05.121 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.8677999999999998e-05 2024-02-05T04:54:05.121 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.121 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:05.121 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.121 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.121 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:05.121 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-02-05T04:54:05.123 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 9 1 2024-02-05T04:54:05.124 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-02-05T04:54:05.124 INFO:tasks.ceph.osd.14.smithi150.stderr:2024-02-05T04:54:05.121+0000 7f5f81083640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 34020) UID: 0 2024-02-05T04:54:05.223 INFO:tasks.ceph.osd.10.smithi150.stderr:2024-02-05T04:54:05.222+0000 7f3075a4d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 34018) UID: 0 2024-02-05T04:54:05.270 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-02-05T04:54:05.270 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [], 2024-02-05T04:54:05.270 INFO:teuthology.orchestra.run.smithi150.stdout: "num_ops": 0 2024-02-05T04:54:05.270 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-02-05T04:54:05.279 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-02-05T04:54:05.324 INFO:tasks.ceph.osd.0.smithi062.stderr:2024-02-05T04:54:05.323+0000 7fece8e03640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34688) UID: 0 2024-02-05T04:54:05.424 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-02-05T04:54:05.425 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [], 2024-02-05T04:54:05.425 INFO:teuthology.orchestra.run.smithi150.stdout: "complaint_time": 30, 2024-02-05T04:54:05.425 INFO:teuthology.orchestra.run.smithi150.stdout: "num_blocked_ops": 0 2024-02-05T04:54:05.425 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-02-05T04:54:05.425 INFO:tasks.ceph.osd.10.smithi150.stderr:2024-02-05T04:54:05.423+0000 7f3075a4d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 34018) UID: 0 2024-02-05T04:54:05.433 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-02-05T04:54:05.526 INFO:tasks.ceph.osd.8.smithi062.stderr:2024-02-05T04:54:05.525+0000 7f9385bf2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 34693) UID: 0 2024-02-05T04:54:05.581 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-02-05T04:54:05.581 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 20, 2024-02-05T04:54:05.581 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 600, 2024-02-05T04:54:05.581 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [ 2024-02-05T04:54:05.581 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.581 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.9s6 57/31 ECSubWrite(tid=47, reqid=osd.14.0:47, at_version=57'21, trim_to=0'0, roll_forward_to=52'19))", 2024-02-05T04:54:05.581 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:48.946360+0000", 2024-02-05T04:54:05.581 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 16.632599224, 2024-02-05T04:54:05.581 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0061455640000000001, 2024-02-05T04:54:05.581 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:05.582 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:05.582 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:05.582 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.582 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:05.582 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.946360+0000", 2024-02-05T04:54:05.582 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:05.582 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.582 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.582 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:05.582 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.946360+0000", 2024-02-05T04:54:05.582 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:05.582 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.582 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.582 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:05.583 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.946361+0000", 2024-02-05T04:54:05.583 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.8099999999999999e-07 2024-02-05T04:54:05.583 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.583 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.583 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:05.583 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.946393+0000", 2024-02-05T04:54:05.583 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.2657000000000003e-05 2024-02-05T04:54:05.583 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.583 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.583 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:05.583 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.946408+0000", 2024-02-05T04:54:05.583 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.5163e-05 2024-02-05T04:54:05.583 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.583 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.583 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:05.584 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.946422+0000", 2024-02-05T04:54:05.584 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.3868e-05 2024-02-05T04:54:05.584 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.584 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.584 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:05.584 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.946545+0000", 2024-02-05T04:54:05.584 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000122216 2024-02-05T04:54:05.584 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.584 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.584 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:05.584 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.946557+0000", 2024-02-05T04:54:05.585 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.27e-05 2024-02-05T04:54:05.585 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.585 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.585 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:05.585 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.951709+0000", 2024-02-05T04:54:05.585 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0051519699999999996 2024-02-05T04:54:05.585 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.585 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.585 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:05.585 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.952506+0000", 2024-02-05T04:54:05.585 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00079680899999999999 2024-02-05T04:54:05.585 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.585 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:05.585 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.586 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.586 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.586 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.ds3 57/31 ECSubWrite(tid=52, reqid=osd.7.0:52, at_version=57'32, trim_to=0'0, roll_forward_to=56'30))", 2024-02-05T04:54:05.586 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:48.948953+0000", 2024-02-05T04:54:05.586 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 16.630006344999998, 2024-02-05T04:54:05.586 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0083893240000000001, 2024-02-05T04:54:05.586 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:05.586 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:05.586 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:05.586 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.586 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:05.586 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.948953+0000", 2024-02-05T04:54:05.586 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:05.586 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.586 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.587 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:05.587 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.948953+0000", 2024-02-05T04:54:05.587 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:05.587 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.587 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.587 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:05.587 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.948954+0000", 2024-02-05T04:54:05.587 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.4200000000000002e-07 2024-02-05T04:54:05.587 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.587 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.587 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:05.587 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.948986+0000", 2024-02-05T04:54:05.587 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.2277999999999998e-05 2024-02-05T04:54:05.587 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.587 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.587 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:05.588 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.949006+0000", 2024-02-05T04:54:05.588 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.067e-05 2024-02-05T04:54:05.588 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.588 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.588 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:05.588 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.949017+0000", 2024-02-05T04:54:05.588 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0471e-05 2024-02-05T04:54:05.588 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.588 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.588 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:05.589 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.951240+0000", 2024-02-05T04:54:05.589 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0022229509999999999 2024-02-05T04:54:05.589 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.589 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.589 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:05.589 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.951248+0000", 2024-02-05T04:54:05.589 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.1589999999999993e-06 2024-02-05T04:54:05.589 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.589 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.589 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:05.589 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.957292+0000", 2024-02-05T04:54:05.589 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0060442589999999997 2024-02-05T04:54:05.589 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.589 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.590 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:05.590 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.957343+0000", 2024-02-05T04:54:05.590 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.0294000000000003e-05 2024-02-05T04:54:05.590 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.590 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:05.590 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.590 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.590 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.591 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "pg_update_log_missing(3.fs5 epoch 57/31 rep_tid 91 entries 57'43 (0'0) error 3:fdb87099:::smithi19639362-41:head by client.4634.0:517 0.000000 -2 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 52'40)", 2024-02-05T04:54:05.591 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:48.951289+0000", 2024-02-05T04:54:05.591 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 16.627670985000002, 2024-02-05T04:54:05.591 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0059310200000000004, 2024-02-05T04:54:05.591 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:05.591 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:05.591 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:05.591 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.591 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:05.592 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.951289+0000", 2024-02-05T04:54:05.592 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:05.592 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.592 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.592 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:05.592 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.951289+0000", 2024-02-05T04:54:05.592 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:05.592 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.592 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.593 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:05.593 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.951290+0000", 2024-02-05T04:54:05.593 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.8100000000000001e-07 2024-02-05T04:54:05.593 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.593 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.593 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:05.593 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.951300+0000", 2024-02-05T04:54:05.593 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0059e-05 2024-02-05T04:54:05.593 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.594 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.594 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:05.594 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.951316+0000", 2024-02-05T04:54:05.594 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.6489000000000001e-05 2024-02-05T04:54:05.594 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.594 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.594 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:05.594 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.951331+0000", 2024-02-05T04:54:05.594 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.4769999999999999e-05 2024-02-05T04:54:05.595 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.595 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.595 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:05.595 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.951431+0000", 2024-02-05T04:54:05.595 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00010024800000000001 2024-02-05T04:54:05.595 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.595 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.595 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:05.596 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.957220+0000", 2024-02-05T04:54:05.597 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.005788573 2024-02-05T04:54:05.597 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.597 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:05.599 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.599 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.599 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.600 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.fs5 57/31 ECSubWrite(tid=92, reqid=osd.11.0:92, at_version=57'45, trim_to=0'0, roll_forward_to=54'42))", 2024-02-05T04:54:05.600 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:48.951368+0000", 2024-02-05T04:54:05.600 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 16.627591638999998, 2024-02-05T04:54:05.600 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00588275, 2024-02-05T04:54:05.600 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:05.600 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:05.600 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:05.600 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.600 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:05.601 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.951368+0000", 2024-02-05T04:54:05.601 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:05.601 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.601 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.601 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:05.601 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.951368+0000", 2024-02-05T04:54:05.601 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:05.601 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.602 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.602 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:05.602 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.951368+0000", 2024-02-05T04:54:05.602 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.6699999999999999e-07 2024-02-05T04:54:05.602 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.602 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.602 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:05.602 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.951393+0000", 2024-02-05T04:54:05.602 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.4813000000000001e-05 2024-02-05T04:54:05.603 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.603 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.603 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:05.603 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.951408+0000", 2024-02-05T04:54:05.603 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.5231e-05 2024-02-05T04:54:05.603 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.603 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.603 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:05.604 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.951425+0000", 2024-02-05T04:54:05.604 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.6915999999999999e-05 2024-02-05T04:54:05.604 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.604 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.604 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:05.604 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.951755+0000", 2024-02-05T04:54:05.604 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00032987899999999999 2024-02-05T04:54:05.604 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.604 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.604 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:05.605 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.951765+0000", 2024-02-05T04:54:05.605 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.9240000000000002e-06 2024-02-05T04:54:05.605 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.605 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.605 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:05.605 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.957224+0000", 2024-02-05T04:54:05.605 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0054592690000000001 2024-02-05T04:54:05.605 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.605 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.605 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:05.605 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.957251+0000", 2024-02-05T04:54:05.605 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.6350999999999999e-05 2024-02-05T04:54:05.605 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.605 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:05.606 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.606 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.606 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.606 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.fs5 57/31 ECSubWrite(tid=93, reqid=osd.11.0:93, at_version=57'47, trim_to=0'0, roll_forward_to=54'42))", 2024-02-05T04:54:05.606 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:48.952554+0000", 2024-02-05T04:54:05.606 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 16.626405950999999, 2024-02-05T04:54:05.606 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0075743909999999998, 2024-02-05T04:54:05.606 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:05.606 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:05.606 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:05.606 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.606 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:05.606 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.952554+0000", 2024-02-05T04:54:05.606 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:05.606 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.607 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.607 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:05.607 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.952554+0000", 2024-02-05T04:54:05.607 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:05.607 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.607 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.607 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:05.607 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.952554+0000", 2024-02-05T04:54:05.607 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.7000000000000004e-07 2024-02-05T04:54:05.607 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.607 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.607 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:05.607 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.952576+0000", 2024-02-05T04:54:05.607 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.1228000000000001e-05 2024-02-05T04:54:05.607 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.608 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.608 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:05.608 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.952588+0000", 2024-02-05T04:54:05.608 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2021999999999999e-05 2024-02-05T04:54:05.608 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.608 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.608 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:05.608 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.952597+0000", 2024-02-05T04:54:05.608 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.7319999999999993e-06 2024-02-05T04:54:05.609 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.609 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.609 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:05.609 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.952660+0000", 2024-02-05T04:54:05.609 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.2285000000000004e-05 2024-02-05T04:54:05.609 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.609 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.609 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:05.609 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.952669+0000", 2024-02-05T04:54:05.609 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.3440000000000007e-06 2024-02-05T04:54:05.609 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.610 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.610 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:05.610 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.960084+0000", 2024-02-05T04:54:05.610 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.007415222 2024-02-05T04:54:05.610 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.610 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.610 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:05.610 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.960128+0000", 2024-02-05T04:54:05.610 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.3788000000000002e-05 2024-02-05T04:54:05.610 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.610 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:05.611 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.611 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.611 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.611 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.ds3 57/31 ECSubWrite(tid=53, reqid=osd.7.0:52, at_version=0'0, trim_to=0'0, roll_forward_to=57'32))", 2024-02-05T04:54:05.611 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:48.958059+0000", 2024-02-05T04:54:05.611 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 16.620900930000001, 2024-02-05T04:54:05.611 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0069118499999999998, 2024-02-05T04:54:05.611 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:05.611 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:05.611 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:05.611 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.611 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:05.611 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.958059+0000", 2024-02-05T04:54:05.611 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:05.611 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.611 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.612 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:05.612 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.958059+0000", 2024-02-05T04:54:05.612 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:05.612 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.612 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.612 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:05.612 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.958060+0000", 2024-02-05T04:54:05.612 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.4300000000000002e-07 2024-02-05T04:54:05.612 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.612 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.612 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:05.612 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.958072+0000", 2024-02-05T04:54:05.613 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2687999999999999e-05 2024-02-05T04:54:05.613 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.613 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.613 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:05.613 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.958088+0000", 2024-02-05T04:54:05.613 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.6161999999999999e-05 2024-02-05T04:54:05.613 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.613 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.613 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:05.613 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.958100+0000", 2024-02-05T04:54:05.613 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.206e-05 2024-02-05T04:54:05.613 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.614 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.614 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:05.614 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.958160+0000", 2024-02-05T04:54:05.614 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.9407999999999998e-05 2024-02-05T04:54:05.614 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.614 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.614 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:05.614 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.958170+0000", 2024-02-05T04:54:05.614 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.7899999999999994e-06 2024-02-05T04:54:05.614 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.614 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.614 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:05.614 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.964927+0000", 2024-02-05T04:54:05.614 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0067569220000000003 2024-02-05T04:54:05.614 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.615 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.615 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:05.615 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.964971+0000", 2024-02-05T04:54:05.615 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.3977000000000003e-05 2024-02-05T04:54:05.615 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.615 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:05.615 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.615 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.615 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.615 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.cs3 57/31 ECSubWrite(tid=94, reqid=osd.11.0:90, at_version=0'0, trim_to=0'0, roll_forward_to=57'13))", 2024-02-05T04:54:05.615 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:48.958302+0000", 2024-02-05T04:54:05.615 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 16.620657653999999, 2024-02-05T04:54:05.615 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.006675505, 2024-02-05T04:54:05.615 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:05.615 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:05.616 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:05.616 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.616 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:05.616 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.958302+0000", 2024-02-05T04:54:05.616 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:05.616 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.616 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.616 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:05.616 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.958302+0000", 2024-02-05T04:54:05.616 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:05.616 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.616 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.616 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:05.616 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.958303+0000", 2024-02-05T04:54:05.617 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.6499999999999998e-07 2024-02-05T04:54:05.617 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.617 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.617 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:05.617 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.958314+0000", 2024-02-05T04:54:05.617 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1314999999999999e-05 2024-02-05T04:54:05.617 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.617 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.617 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:05.617 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.958329+0000", 2024-02-05T04:54:05.617 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.526e-05 2024-02-05T04:54:05.617 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.617 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.617 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:05.618 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.958340+0000", 2024-02-05T04:54:05.618 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0210999999999999e-05 2024-02-05T04:54:05.618 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.618 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.618 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:05.618 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.958406+0000", 2024-02-05T04:54:05.618 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.6867999999999997e-05 2024-02-05T04:54:05.618 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.618 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.618 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:05.618 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.958416+0000", 2024-02-05T04:54:05.619 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.5529999999999998e-06 2024-02-05T04:54:05.619 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.619 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.619 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:05.619 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.964942+0000", 2024-02-05T04:54:05.619 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0065262940000000002 2024-02-05T04:54:05.619 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.619 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.619 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:05.619 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.964978+0000", 2024-02-05T04:54:05.619 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.5238999999999999e-05 2024-02-05T04:54:05.619 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.619 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:05.620 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.620 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.620 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.620 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.ds3 57/31 ECSubWrite(tid=54, reqid=client.4634.0:525, at_version=57'34, trim_to=0'0, roll_forward_to=57'32))", 2024-02-05T04:54:05.620 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:48.959297+0000", 2024-02-05T04:54:05.620 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 16.619663020000001, 2024-02-05T04:54:05.620 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0062495950000000002, 2024-02-05T04:54:05.620 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:05.620 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:05.620 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:05.620 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.620 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:05.621 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.959297+0000", 2024-02-05T04:54:05.621 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:05.621 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.621 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.621 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:05.621 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.959297+0000", 2024-02-05T04:54:05.621 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:05.621 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.621 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.621 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:05.621 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.959297+0000", 2024-02-05T04:54:05.621 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.4000000000000001e-07 2024-02-05T04:54:05.621 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.621 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.621 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:05.622 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.959319+0000", 2024-02-05T04:54:05.622 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.143e-05 2024-02-05T04:54:05.622 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.622 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.622 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:05.622 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.959331+0000", 2024-02-05T04:54:05.622 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1924e-05 2024-02-05T04:54:05.622 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.622 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.622 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:05.622 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.959339+0000", 2024-02-05T04:54:05.623 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.3720000000000005e-06 2024-02-05T04:54:05.623 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.623 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.623 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:05.623 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.959399+0000", 2024-02-05T04:54:05.623 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.0108000000000001e-05 2024-02-05T04:54:05.623 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.623 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.623 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:05.623 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.959409+0000", 2024-02-05T04:54:05.623 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0276e-05 2024-02-05T04:54:05.623 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.624 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.624 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:05.624 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.965510+0000", 2024-02-05T04:54:05.624 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0061009610000000002 2024-02-05T04:54:05.624 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.624 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.624 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:05.624 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.965546+0000", 2024-02-05T04:54:05.624 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.5883999999999997e-05 2024-02-05T04:54:05.624 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.624 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:05.624 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.624 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.625 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.625 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.ds3 58/31 ECSubWrite(tid=56, reqid=client.4634.0:526, at_version=58'35, trim_to=0'0, roll_forward_to=57'34))", 2024-02-05T04:54:05.625 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:49.970260+0000", 2024-02-05T04:54:05.625 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 15.60869993, 2024-02-05T04:54:05.625 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.01268597, 2024-02-05T04:54:05.625 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:05.625 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:05.625 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:05.625 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.625 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:05.625 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.970260+0000", 2024-02-05T04:54:05.625 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:05.625 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.625 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.626 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:05.626 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.970260+0000", 2024-02-05T04:54:05.626 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:05.626 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.626 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.626 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:05.626 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.970260+0000", 2024-02-05T04:54:05.626 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.7700000000000001e-07 2024-02-05T04:54:05.626 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.626 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.626 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:05.626 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.972666+0000", 2024-02-05T04:54:05.626 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0024063370000000001 2024-02-05T04:54:05.626 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.626 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.626 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:05.627 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.972679+0000", 2024-02-05T04:54:05.627 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2510000000000001e-05 2024-02-05T04:54:05.627 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.627 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.627 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:05.627 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.972689+0000", 2024-02-05T04:54:05.627 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0705e-05 2024-02-05T04:54:05.627 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.627 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.627 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:05.627 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.972776+0000", 2024-02-05T04:54:05.627 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.6290000000000002e-05 2024-02-05T04:54:05.627 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.627 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.627 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:05.628 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.972786+0000", 2024-02-05T04:54:05.628 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.9420000000000007e-06 2024-02-05T04:54:05.628 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.628 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.628 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:05.628 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.982834+0000", 2024-02-05T04:54:05.628 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.010047943 2024-02-05T04:54:05.628 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.628 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.628 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:05.628 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.982946+0000", 2024-02-05T04:54:05.628 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00011206599999999999 2024-02-05T04:54:05.628 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.629 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:05.629 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.629 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.629 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.629 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.fs5 58/31 ECSubWrite(tid=104, reqid=client.4634.0:518, at_version=58'51, trim_to=0'0, roll_forward_to=57'49))", 2024-02-05T04:54:05.629 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:49.974219+0000", 2024-02-05T04:54:05.629 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 15.604740699000001, 2024-02-05T04:54:05.629 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0090656959999999998, 2024-02-05T04:54:05.629 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:05.629 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:05.629 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:05.629 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.629 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:05.629 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.974219+0000", 2024-02-05T04:54:05.629 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:05.629 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.630 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.630 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:05.630 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.974219+0000", 2024-02-05T04:54:05.630 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:05.630 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.630 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.630 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:05.630 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.974220+0000", 2024-02-05T04:54:05.630 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.6000000000000002e-07 2024-02-05T04:54:05.630 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.630 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.630 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:05.630 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.976866+0000", 2024-02-05T04:54:05.631 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.002646041 2024-02-05T04:54:05.631 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.631 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.631 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:05.631 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.976878+0000", 2024-02-05T04:54:05.631 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2157999999999999e-05 2024-02-05T04:54:05.631 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.631 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.631 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:05.631 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.976886+0000", 2024-02-05T04:54:05.631 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.3920000000000003e-06 2024-02-05T04:54:05.631 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.631 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.631 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:05.632 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.976954+0000", 2024-02-05T04:54:05.632 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.7859000000000002e-05 2024-02-05T04:54:05.632 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.632 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.632 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:05.632 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.976964+0000", 2024-02-05T04:54:05.632 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.3600000000000002e-06 2024-02-05T04:54:05.632 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.632 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.632 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:05.632 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.983255+0000", 2024-02-05T04:54:05.632 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0062918749999999997 2024-02-05T04:54:05.632 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.633 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.633 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:05.633 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.983285+0000", 2024-02-05T04:54:05.633 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.9150999999999999e-05 2024-02-05T04:54:05.633 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.633 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:05.633 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.633 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.633 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.633 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.bs2 58/57 ECSubWrite(tid=255, reqid=osd.13.0:252, at_version=0'0, trim_to=0'0, roll_forward_to=58'54))", 2024-02-05T04:54:05.633 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:49.976650+0000", 2024-02-05T04:54:05.633 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 15.602309579, 2024-02-05T04:54:05.633 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.006192197, 2024-02-05T04:54:05.633 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:05.633 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:05.634 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:05.634 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.634 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:05.634 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.976650+0000", 2024-02-05T04:54:05.634 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:05.634 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.634 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.634 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:05.634 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.976650+0000", 2024-02-05T04:54:05.634 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:05.634 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.634 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.634 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:05.634 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.976651+0000", 2024-02-05T04:54:05.634 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.9299999999999997e-07 2024-02-05T04:54:05.635 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.635 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.635 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:05.635 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.976681+0000", 2024-02-05T04:54:05.635 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.0309000000000001e-05 2024-02-05T04:54:05.635 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.635 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.635 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:05.635 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.976698+0000", 2024-02-05T04:54:05.635 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.6629e-05 2024-02-05T04:54:05.635 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.635 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.635 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:05.635 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.976707+0000", 2024-02-05T04:54:05.635 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.9940000000000006e-06 2024-02-05T04:54:05.636 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.636 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.636 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:05.636 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.976779+0000", 2024-02-05T04:54:05.636 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.2557000000000003e-05 2024-02-05T04:54:05.636 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.636 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.636 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:05.636 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.976790+0000", 2024-02-05T04:54:05.636 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0277e-05 2024-02-05T04:54:05.636 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.637 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.637 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:05.637 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.982791+0000", 2024-02-05T04:54:05.637 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0060016119999999999 2024-02-05T04:54:05.637 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.637 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.637 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:05.637 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.982842+0000", 2024-02-05T04:54:05.637 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.1026000000000002e-05 2024-02-05T04:54:05.637 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.637 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:05.637 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.637 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.637 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.637 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.ds3 63/31 ECSubWrite(tid=62, reqid=osd.7.0:62, at_version=63'41, trim_to=0'0, roll_forward_to=61'37))", 2024-02-05T04:54:05.638 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:54.977679+0000", 2024-02-05T04:54:05.638 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 10.601280232000001, 2024-02-05T04:54:05.638 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0059499339999999996, 2024-02-05T04:54:05.638 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:05.638 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:05.638 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:05.638 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.638 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:05.638 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.977679+0000", 2024-02-05T04:54:05.638 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:05.638 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.638 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.638 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:05.638 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.977679+0000", 2024-02-05T04:54:05.638 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:05.639 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.639 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.639 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:05.639 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.977680+0000", 2024-02-05T04:54:05.639 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.3200000000000005e-07 2024-02-05T04:54:05.639 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.639 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.639 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:05.639 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.977705+0000", 2024-02-05T04:54:05.639 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.4936999999999999e-05 2024-02-05T04:54:05.639 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.639 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.639 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:05.639 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.977714+0000", 2024-02-05T04:54:05.640 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.1430000000000004e-06 2024-02-05T04:54:05.640 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.640 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.640 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:05.640 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.977723+0000", 2024-02-05T04:54:05.640 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.1419999999999999e-06 2024-02-05T04:54:05.640 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.640 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.640 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:05.640 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.978346+0000", 2024-02-05T04:54:05.641 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00062224800000000005 2024-02-05T04:54:05.641 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.641 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.641 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:05.641 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.978354+0000", 2024-02-05T04:54:05.641 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.9099999999999994e-06 2024-02-05T04:54:05.641 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.641 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.641 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:05.641 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.983586+0000", 2024-02-05T04:54:05.641 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.005231767 2024-02-05T04:54:05.641 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.641 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.642 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:05.642 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.983629+0000", 2024-02-05T04:54:05.642 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.3155000000000003e-05 2024-02-05T04:54:05.642 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.642 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:05.642 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.642 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.642 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.642 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.cs3 63/31 ECSubWrite(tid=120, reqid=osd.11.0:120, at_version=63'25, trim_to=0'0, roll_forward_to=62'23))", 2024-02-05T04:54:05.642 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:54.986846+0000", 2024-02-05T04:54:05.642 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 10.59211339, 2024-02-05T04:54:05.642 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0064626040000000003, 2024-02-05T04:54:05.642 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:05.643 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:05.643 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:05.643 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.643 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:05.643 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.986846+0000", 2024-02-05T04:54:05.643 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:05.643 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.643 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.643 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:05.644 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.986846+0000", 2024-02-05T04:54:05.644 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:05.644 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.644 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.644 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:05.644 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.986847+0000", 2024-02-05T04:54:05.644 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.6000000000000003e-07 2024-02-05T04:54:05.644 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.645 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.645 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:05.645 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.986870+0000", 2024-02-05T04:54:05.645 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.3430000000000001e-05 2024-02-05T04:54:05.645 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.645 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.645 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:05.645 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.986879+0000", 2024-02-05T04:54:05.645 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.8410000000000006e-06 2024-02-05T04:54:05.646 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.646 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.646 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:05.646 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.986888+0000", 2024-02-05T04:54:05.646 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.4230000000000005e-06 2024-02-05T04:54:05.646 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.646 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.646 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:05.646 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.986985+0000", 2024-02-05T04:54:05.647 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.7695999999999994e-05 2024-02-05T04:54:05.647 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.647 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.647 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:05.647 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.987005+0000", 2024-02-05T04:54:05.647 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.9471e-05 2024-02-05T04:54:05.647 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.647 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.648 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:05.648 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.993267+0000", 2024-02-05T04:54:05.648 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.006262704 2024-02-05T04:54:05.648 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.648 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.648 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:05.648 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.993309+0000", 2024-02-05T04:54:05.648 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.1378999999999998e-05 2024-02-05T04:54:05.648 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.649 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:05.649 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.649 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.649 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.649 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.ds3 63/31 ECSubWrite(tid=63, reqid=osd.7.0:62, at_version=0'0, trim_to=0'0, roll_forward_to=63'41))", 2024-02-05T04:54:05.649 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:54.986948+0000", 2024-02-05T04:54:05.649 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 10.592011303, 2024-02-05T04:54:05.649 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0056941370000000002, 2024-02-05T04:54:05.649 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:05.650 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:05.650 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:05.650 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.650 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:05.650 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.986948+0000", 2024-02-05T04:54:05.650 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:05.650 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.650 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.651 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:05.651 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.986948+0000", 2024-02-05T04:54:05.651 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:05.651 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.651 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.651 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:05.651 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.986949+0000", 2024-02-05T04:54:05.651 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.7400000000000003e-07 2024-02-05T04:54:05.652 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.652 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.652 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:05.652 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.986958+0000", 2024-02-05T04:54:05.652 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.5119999999999997e-06 2024-02-05T04:54:05.652 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.652 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.652 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:05.653 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.986969+0000", 2024-02-05T04:54:05.653 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0677e-05 2024-02-05T04:54:05.653 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.653 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.653 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:05.653 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.986982+0000", 2024-02-05T04:54:05.653 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.3358e-05 2024-02-05T04:54:05.653 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.653 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.653 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:05.654 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.987084+0000", 2024-02-05T04:54:05.654 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000102011 2024-02-05T04:54:05.654 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.654 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.654 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:05.654 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.987098+0000", 2024-02-05T04:54:05.654 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.3725e-05 2024-02-05T04:54:05.654 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.654 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.655 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:05.655 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.992556+0000", 2024-02-05T04:54:05.655 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.005458119 2024-02-05T04:54:05.655 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.655 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.655 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:05.655 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.992642+0000", 2024-02-05T04:54:05.655 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.6161000000000004e-05 2024-02-05T04:54:05.656 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.656 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:05.656 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.656 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.656 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.656 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.fs5 63/31 ECSubWrite(tid=121, reqid=osd.11.0:121, at_version=63'54, trim_to=0'0, roll_forward_to=58'51))", 2024-02-05T04:54:05.656 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:54.988723+0000", 2024-02-05T04:54:05.656 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 10.590236192999999, 2024-02-05T04:54:05.657 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0076980920000000001, 2024-02-05T04:54:05.657 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:05.657 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:05.657 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:05.657 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.657 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:05.657 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.988723+0000", 2024-02-05T04:54:05.657 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:05.657 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.658 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.658 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:05.658 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.988723+0000", 2024-02-05T04:54:05.658 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:05.658 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.658 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.658 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:05.658 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.988724+0000", 2024-02-05T04:54:05.658 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.8899999999999999e-07 2024-02-05T04:54:05.658 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.658 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.658 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:05.658 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.988740+0000", 2024-02-05T04:54:05.658 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.5928000000000001e-05 2024-02-05T04:54:05.658 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.658 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.659 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:05.659 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.988752+0000", 2024-02-05T04:54:05.659 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2201999999999999e-05 2024-02-05T04:54:05.659 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.659 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.659 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:05.659 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.988758+0000", 2024-02-05T04:54:05.659 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.0730000000000001e-06 2024-02-05T04:54:05.659 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.659 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.659 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:05.659 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.988828+0000", 2024-02-05T04:54:05.659 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.9296999999999997e-05 2024-02-05T04:54:05.659 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.659 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.660 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:05.660 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.988836+0000", 2024-02-05T04:54:05.660 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.6449999999999994e-06 2024-02-05T04:54:05.660 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.660 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.660 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:05.660 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.996389+0000", 2024-02-05T04:54:05.660 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0075529459999999996 2024-02-05T04:54:05.660 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.660 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.660 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:05.660 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.996422+0000", 2024-02-05T04:54:05.660 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.2311999999999998e-05 2024-02-05T04:54:05.660 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.660 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:05.661 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.661 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.661 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.661 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.fs5 63/31 ECSubWrite(tid=122, reqid=osd.11.0:122, at_version=63'56, trim_to=0'0, roll_forward_to=58'51))", 2024-02-05T04:54:05.661 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:54.990220+0000", 2024-02-05T04:54:05.661 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 10.588740134, 2024-02-05T04:54:05.661 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0088437989999999994, 2024-02-05T04:54:05.661 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:05.661 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:05.661 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:05.661 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.662 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:05.662 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.990220+0000", 2024-02-05T04:54:05.662 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:05.662 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.662 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.662 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:05.662 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.990220+0000", 2024-02-05T04:54:05.662 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:05.662 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.662 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.662 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:05.662 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.990220+0000", 2024-02-05T04:54:05.662 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.4499999999999996e-07 2024-02-05T04:54:05.662 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.662 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.663 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:05.663 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.990243+0000", 2024-02-05T04:54:05.663 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.2385999999999999e-05 2024-02-05T04:54:05.663 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.663 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.663 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:05.663 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.990260+0000", 2024-02-05T04:54:05.663 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.7292999999999999e-05 2024-02-05T04:54:05.663 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.663 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.663 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:05.663 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.990269+0000", 2024-02-05T04:54:05.663 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.9160000000000007e-06 2024-02-05T04:54:05.663 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.663 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.663 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:05.664 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.990338+0000", 2024-02-05T04:54:05.664 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.8961000000000006e-05 2024-02-05T04:54:05.664 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.664 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.664 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:05.664 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.990347+0000", 2024-02-05T04:54:05.664 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.1609999999999992e-06 2024-02-05T04:54:05.664 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.664 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.664 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:05.664 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.999022+0000", 2024-02-05T04:54:05.664 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0086750279999999996 2024-02-05T04:54:05.664 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.664 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.664 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:05.665 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.999063+0000", 2024-02-05T04:54:05.665 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.1309e-05 2024-02-05T04:54:05.665 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.665 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:05.665 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.665 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.665 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.665 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.1s3 63/62 ECSubWrite(tid=265, reqid=client.4634.0:594, at_version=63'53, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:05.665 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:55.003811+0000", 2024-02-05T04:54:05.665 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 10.575148298, 2024-02-05T04:54:05.665 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.007279682, 2024-02-05T04:54:05.665 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:05.665 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:05.665 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:05.666 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.666 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:05.666 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.003811+0000", 2024-02-05T04:54:05.666 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:05.666 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.666 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.666 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:05.666 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.003811+0000", 2024-02-05T04:54:05.666 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:05.666 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.666 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.666 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:05.666 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.003812+0000", 2024-02-05T04:54:05.666 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.3299999999999998e-07 2024-02-05T04:54:05.667 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.667 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.667 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:05.667 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.004636+0000", 2024-02-05T04:54:05.667 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00082469300000000004 2024-02-05T04:54:05.667 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.667 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.667 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:05.667 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.004649+0000", 2024-02-05T04:54:05.667 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2172e-05 2024-02-05T04:54:05.667 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.667 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.667 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:05.667 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.004657+0000", 2024-02-05T04:54:05.667 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.7360000000000007e-06 2024-02-05T04:54:05.668 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.668 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.668 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:05.668 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.004716+0000", 2024-02-05T04:54:05.668 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.8650000000000003e-05 2024-02-05T04:54:05.668 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.668 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.668 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:05.668 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.004723+0000", 2024-02-05T04:54:05.668 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.9940000000000003e-06 2024-02-05T04:54:05.668 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.668 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.668 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:05.668 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.011053+0000", 2024-02-05T04:54:05.668 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0063304449999999996 2024-02-05T04:54:05.669 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.669 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.669 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:05.669 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.011091+0000", 2024-02-05T04:54:05.669 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.7659e-05 2024-02-05T04:54:05.669 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.669 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:05.669 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.669 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.669 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.669 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.bs2 69/57 ECSubWrite(tid=297, reqid=osd.13.0:294, at_version=0'0, trim_to=0'0, roll_forward_to=69'66))", 2024-02-05T04:54:05.670 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:54:00.961664+0000", 2024-02-05T04:54:05.670 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 4.6172959819999999, 2024-02-05T04:54:05.670 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.005832533, 2024-02-05T04:54:05.670 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:05.670 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:05.670 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:05.670 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.670 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:05.670 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.961664+0000", 2024-02-05T04:54:05.670 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:05.670 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.670 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.670 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:05.670 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.961664+0000", 2024-02-05T04:54:05.670 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:05.671 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.671 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.671 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:05.671 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.961664+0000", 2024-02-05T04:54:05.671 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.5699999999999998e-07 2024-02-05T04:54:05.671 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.671 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.671 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:05.671 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.961673+0000", 2024-02-05T04:54:05.671 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.5499999999999995e-06 2024-02-05T04:54:05.671 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.671 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.672 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:05.672 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.961682+0000", 2024-02-05T04:54:05.672 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.0629999999999995e-06 2024-02-05T04:54:05.672 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.672 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.672 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:05.672 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.961692+0000", 2024-02-05T04:54:05.672 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0074999999999999e-05 2024-02-05T04:54:05.672 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.672 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.672 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:05.672 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.961903+0000", 2024-02-05T04:54:05.672 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000211471 2024-02-05T04:54:05.672 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.673 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.673 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:05.673 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.961913+0000", 2024-02-05T04:54:05.673 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0239e-05 2024-02-05T04:54:05.673 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.673 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.673 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:05.673 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.967448+0000", 2024-02-05T04:54:05.673 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.005534873 2024-02-05T04:54:05.673 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.673 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.673 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:05.673 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.967496+0000", 2024-02-05T04:54:05.673 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.7905000000000003e-05 2024-02-05T04:54:05.673 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.674 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:05.674 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.674 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.674 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.674 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.1s3 69/62 ECSubWrite(tid=296, reqid=osd.13.0:296, at_version=69'68, trim_to=0'0, roll_forward_to=68'66))", 2024-02-05T04:54:05.674 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:54:00.961706+0000", 2024-02-05T04:54:05.674 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 4.6172538210000003, 2024-02-05T04:54:05.674 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.005845355, 2024-02-05T04:54:05.674 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:05.674 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:05.674 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:05.674 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.674 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:05.674 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.961706+0000", 2024-02-05T04:54:05.674 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:05.675 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.675 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.675 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:05.675 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.961706+0000", 2024-02-05T04:54:05.675 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:05.675 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.675 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.675 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:05.675 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.961706+0000", 2024-02-05T04:54:05.675 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.4700000000000002e-07 2024-02-05T04:54:05.675 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.675 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.675 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:05.675 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.961736+0000", 2024-02-05T04:54:05.675 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.9369999999999998e-05 2024-02-05T04:54:05.676 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.676 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.676 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:05.676 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.961745+0000", 2024-02-05T04:54:05.676 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.7019999999999996e-06 2024-02-05T04:54:05.676 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.676 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.676 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:05.676 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.961752+0000", 2024-02-05T04:54:05.676 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.0389999999999999e-06 2024-02-05T04:54:05.676 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.676 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.676 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:05.676 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.961807+0000", 2024-02-05T04:54:05.677 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.5136000000000002e-05 2024-02-05T04:54:05.677 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.677 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.677 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:05.677 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.961817+0000", 2024-02-05T04:54:05.677 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.5109999999999992e-06 2024-02-05T04:54:05.677 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.677 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.677 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:05.677 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.967515+0000", 2024-02-05T04:54:05.677 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0056976220000000003 2024-02-05T04:54:05.677 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.677 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.677 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:05.677 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.967551+0000", 2024-02-05T04:54:05.678 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.6628000000000002e-05 2024-02-05T04:54:05.678 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.678 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:05.678 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.678 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.678 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.678 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.2s5 69/62 ECSubWrite(tid=75, reqid=client.4634.0:641, at_version=69'55, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:05.678 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:54:01.014057+0000", 2024-02-05T04:54:05.678 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 4.5649029690000003, 2024-02-05T04:54:05.678 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0069550080000000004, 2024-02-05T04:54:05.678 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:05.678 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:05.678 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:05.678 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.678 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:05.679 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:01.014057+0000", 2024-02-05T04:54:05.679 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:05.679 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.679 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.679 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:05.679 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:01.014057+0000", 2024-02-05T04:54:05.679 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:05.679 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.679 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.679 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:05.679 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:01.014057+0000", 2024-02-05T04:54:05.679 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.0999999999999997e-07 2024-02-05T04:54:05.679 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.679 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.679 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:05.679 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:01.016905+0000", 2024-02-05T04:54:05.680 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0028475390000000001 2024-02-05T04:54:05.680 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.680 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.680 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:05.680 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:01.016940+0000", 2024-02-05T04:54:05.680 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.5176000000000003e-05 2024-02-05T04:54:05.680 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.680 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.680 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:05.680 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:01.016951+0000", 2024-02-05T04:54:05.680 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0902e-05 2024-02-05T04:54:05.680 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.680 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.680 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:05.681 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:01.017028+0000", 2024-02-05T04:54:05.681 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.6885000000000007e-05 2024-02-05T04:54:05.681 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.681 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.681 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:05.681 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:01.017041+0000", 2024-02-05T04:54:05.681 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.3424999999999999e-05 2024-02-05T04:54:05.681 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.681 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.681 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:05.681 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:01.020948+0000", 2024-02-05T04:54:05.681 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.003906952 2024-02-05T04:54:05.681 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:05.681 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:05.681 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:05.682 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:01.021012+0000", 2024-02-05T04:54:05.682 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.3318999999999995e-05 2024-02-05T04:54:05.682 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.682 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:05.682 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.682 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:05.682 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:05.682 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-02-05T04:54:05.684 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-02-05T04:54:05.684 INFO:tasks.ceph.osd.9.smithi091.stderr:2024-02-05T04:54:05.683+0000 7efc18827640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 34279) UID: 0 2024-02-05T04:54:05.784 INFO:tasks.ceph.osd.6.smithi150.stderr:2024-02-05T04:54:05.783+0000 7f6bea9f0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34015) UID: 0 2024-02-05T04:54:05.790 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-02-05T04:54:05.790 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [], 2024-02-05T04:54:05.790 INFO:teuthology.orchestra.run.smithi150.stdout: "num_ops": 0 2024-02-05T04:54:05.790 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-02-05T04:54:05.799 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-02-05T04:54:05.885 INFO:tasks.ceph.osd.2.smithi150.stderr:2024-02-05T04:54:05.884+0000 7f7337bf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 34014) UID: 0 2024-02-05T04:54:05.946 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-02-05T04:54:05.946 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [], 2024-02-05T04:54:05.946 INFO:teuthology.orchestra.run.smithi150.stdout: "complaint_time": 30, 2024-02-05T04:54:05.946 INFO:teuthology.orchestra.run.smithi150.stdout: "num_blocked_ops": 0 2024-02-05T04:54:05.946 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-02-05T04:54:05.955 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-02-05T04:54:05.986 INFO:tasks.ceph.osd.9.smithi091.stderr:2024-02-05T04:54:05.986+0000 7efc18827640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 34279) UID: 0 2024-02-05T04:54:06.036 INFO:teuthology.orchestra.run.smithi062.stderr:reweighted osd.9 to 1 (10000) 2024-02-05T04:54:06.036 INFO:teuthology.orchestra.run.smithi062.stderr:nodeep-scrub is unset 2024-02-05T04:54:06.041 INFO:tasks.rados.rados.0.smithi196.stdout:273: done (2 left) 2024-02-05T04:54:06.041 INFO:tasks.rados.rados.0.smithi196.stdout:append oid 24 current snap is 23 2024-02-05T04:54:06.041 INFO:tasks.rados.rados.0.smithi196.stdout:271: finishing rollback tid 1 to smithi19639362-50 2024-02-05T04:54:06.041 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 50 v 24 (ObjNum 353 snap 0 seq_num 3699138549) dirty dne 2024-02-05T04:54:06.041 INFO:tasks.rados.rados.0.smithi196.stdout:274: seq_num 123 ranges {1277952=524288,1802240=573440,2375680=688128,3063808=294912} 2024-02-05T04:54:06.052 INFO:tasks.rados.rados.0.smithi196.stdout:274: writing smithi19639362-24 from 1277952 to 1802240 tid 1 2024-02-05T04:54:06.055 INFO:tasks.rados.rados.0.smithi196.stdout:274: writing smithi19639362-24 from 1802240 to 2375680 tid 2 2024-02-05T04:54:06.059 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 10 1 2024-02-05T04:54:06.060 INFO:tasks.rados.rados.0.smithi196.stdout:274: writing smithi19639362-24 from 2375680 to 3063808 tid 3 2024-02-05T04:54:06.061 INFO:tasks.rados.rados.0.smithi196.stdout:274: writing smithi19639362-24 from 3063808 to 3358720 tid 4 2024-02-05T04:54:06.062 INFO:tasks.rados.rados.0.smithi196.stdout:271: done (2 left) 2024-02-05T04:54:06.062 INFO:tasks.rados.rados.0.smithi196.stdout:append oid 28 current snap is 23 2024-02-05T04:54:06.062 INFO:tasks.rados.rados.0.smithi196.stdout:275: seq_num 124 ranges {1277952=540672,1818624=720896,2539520=786432,3325952=49152} 2024-02-05T04:54:06.072 INFO:tasks.rados.rados.0.smithi196.stdout:275: writing smithi19639362-28 from 1277952 to 1818624 tid 1 2024-02-05T04:54:06.077 INFO:tasks.rados.rados.0.smithi196.stdout:275: writing smithi19639362-28 from 1818624 to 2539520 tid 2 2024-02-05T04:54:06.082 INFO:tasks.rados.rados.0.smithi196.stdout:275: writing smithi19639362-28 from 2539520 to 3325952 tid 3 2024-02-05T04:54:06.083 INFO:tasks.rados.rados.0.smithi196.stdout:275: writing smithi19639362-28 from 3325952 to 3375104 tid 4 2024-02-05T04:54:06.083 INFO:tasks.rados.rados.0.smithi196.stdout:276: delete oid 12 current snap is 23 2024-02-05T04:54:06.083 INFO:tasks.rados.rados.0.smithi196.stdout:272: expect (ObjNum 122 snap 22 seq_num 122) 2024-02-05T04:54:06.087 INFO:tasks.ceph.osd.13.smithi091.stderr:2024-02-05T04:54:06.086+0000 7f526fd4d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 34281) UID: 0 2024-02-05T04:54:06.118 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-02-05T04:54:06.118 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 20, 2024-02-05T04:54:06.118 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 600, 2024-02-05T04:54:06.118 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [ 2024-02-05T04:54:06.118 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.118 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.1s6 43/31 ECSubWrite(tid=140, reqid=osd.13.0:140, at_version=43'29, trim_to=0'0, roll_forward_to=42'27))", 2024-02-05T04:54:06.118 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:35.182104+0000", 2024-02-05T04:54:06.118 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 30.934025482999999, 2024-02-05T04:54:06.118 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0056319439999999998, 2024-02-05T04:54:06.119 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:06.119 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:06.119 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:06.119 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.119 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:06.119 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.182104+0000", 2024-02-05T04:54:06.119 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:06.119 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.119 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.119 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:06.119 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.182104+0000", 2024-02-05T04:54:06.119 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:06.119 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.119 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.119 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:06.120 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.182104+0000", 2024-02-05T04:54:06.120 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.3699999999999999e-07 2024-02-05T04:54:06.120 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.120 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.120 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:06.120 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.182127+0000", 2024-02-05T04:54:06.120 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.2994999999999999e-05 2024-02-05T04:54:06.120 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.120 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.120 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:06.120 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.182140+0000", 2024-02-05T04:54:06.120 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2965999999999999e-05 2024-02-05T04:54:06.120 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.120 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.121 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:06.121 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.182150+0000", 2024-02-05T04:54:06.121 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0423e-05 2024-02-05T04:54:06.121 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.121 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.121 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:06.121 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.182216+0000", 2024-02-05T04:54:06.121 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.5209999999999994e-05 2024-02-05T04:54:06.121 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.121 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.121 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:06.121 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.182225+0000", 2024-02-05T04:54:06.121 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.4250000000000004e-06 2024-02-05T04:54:06.121 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.122 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.122 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:06.122 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.187667+0000", 2024-02-05T04:54:06.122 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0054421160000000003 2024-02-05T04:54:06.122 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.122 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.122 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:06.122 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.187736+0000", 2024-02-05T04:54:06.122 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.8571999999999993e-05 2024-02-05T04:54:06.122 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:06.122 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:06.122 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:06.122 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.122 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.122 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.ds2 43/31 ECSubWrite(tid=44, reqid=client.4634.0:401, at_version=43'24, trim_to=0'0, roll_forward_to=38'22))", 2024-02-05T04:54:06.122 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:35.183613+0000", 2024-02-05T04:54:06.123 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 30.932516189000001, 2024-02-05T04:54:06.123 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0072371980000000002, 2024-02-05T04:54:06.123 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:06.123 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:06.123 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:06.123 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.123 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:06.123 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.183613+0000", 2024-02-05T04:54:06.123 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:06.123 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.124 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.124 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:06.124 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.183613+0000", 2024-02-05T04:54:06.124 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:06.124 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.124 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.124 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:06.125 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.183613+0000", 2024-02-05T04:54:06.125 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.5600000000000002e-07 2024-02-05T04:54:06.125 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.125 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.125 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:06.125 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.183636+0000", 2024-02-05T04:54:06.125 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.2248000000000001e-05 2024-02-05T04:54:06.126 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.126 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.126 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:06.126 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.183648+0000", 2024-02-05T04:54:06.126 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2666e-05 2024-02-05T04:54:06.126 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.127 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.127 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:06.127 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.183659+0000", 2024-02-05T04:54:06.127 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1131999999999999e-05 2024-02-05T04:54:06.127 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.127 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.127 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:06.128 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.183713+0000", 2024-02-05T04:54:06.128 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.3400999999999999e-05 2024-02-05T04:54:06.128 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.128 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.128 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:06.128 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.183721+0000", 2024-02-05T04:54:06.129 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.1999999999999994e-06 2024-02-05T04:54:06.129 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.129 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.129 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:06.129 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.190813+0000", 2024-02-05T04:54:06.129 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0070917660000000002 2024-02-05T04:54:06.129 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.130 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.130 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:06.130 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.190850+0000", 2024-02-05T04:54:06.130 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.7528999999999999e-05 2024-02-05T04:54:06.130 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:06.130 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:06.131 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:06.131 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.131 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.131 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.1s6 43/31 ECSubWrite(tid=142, reqid=osd.13.0:142, at_version=43'31, trim_to=0'0, roll_forward_to=42'27))", 2024-02-05T04:54:06.131 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:35.183877+0000", 2024-02-05T04:54:06.131 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 30.932252055999999, 2024-02-05T04:54:06.131 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.006942312, 2024-02-05T04:54:06.132 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:06.132 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:06.132 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:06.132 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.132 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:06.132 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.183877+0000", 2024-02-05T04:54:06.133 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:06.133 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.133 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.133 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:06.133 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.183877+0000", 2024-02-05T04:54:06.133 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:06.134 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.134 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.134 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:06.134 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.183878+0000", 2024-02-05T04:54:06.134 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.5000000000000002e-07 2024-02-05T04:54:06.134 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.134 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.134 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:06.134 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.183901+0000", 2024-02-05T04:54:06.135 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.2895999999999999e-05 2024-02-05T04:54:06.135 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.135 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.135 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:06.135 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.183917+0000", 2024-02-05T04:54:06.135 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.6345e-05 2024-02-05T04:54:06.136 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.136 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.136 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:06.136 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.183930+0000", 2024-02-05T04:54:06.136 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.3308e-05 2024-02-05T04:54:06.136 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.136 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.137 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:06.137 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.183993+0000", 2024-02-05T04:54:06.137 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.2979999999999997e-05 2024-02-05T04:54:06.137 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.137 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.137 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:06.138 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.184008+0000", 2024-02-05T04:54:06.138 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.4861000000000001e-05 2024-02-05T04:54:06.138 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.138 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.138 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:06.138 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.190786+0000", 2024-02-05T04:54:06.139 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0067771979999999999 2024-02-05T04:54:06.139 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.139 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.139 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:06.139 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:35.190820+0000", 2024-02-05T04:54:06.139 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.3973999999999997e-05 2024-02-05T04:54:06.140 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:06.140 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:06.140 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:06.140 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.140 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.141 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.fs2 52/31 ECSubWrite(tid=87, reqid=osd.11.0:85, at_version=0'0, trim_to=0'0, roll_forward_to=52'40))", 2024-02-05T04:54:06.141 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:43.961881+0000", 2024-02-05T04:54:06.141 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 22.154247793, 2024-02-05T04:54:06.141 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0069676850000000004, 2024-02-05T04:54:06.141 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:06.142 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:06.142 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:06.142 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.142 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:06.142 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.961881+0000", 2024-02-05T04:54:06.142 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:06.143 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.143 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.143 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:06.143 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.961881+0000", 2024-02-05T04:54:06.143 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:06.143 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.144 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.144 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:06.144 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.961882+0000", 2024-02-05T04:54:06.144 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.8599999999999998e-07 2024-02-05T04:54:06.144 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.145 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.145 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:06.145 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.961893+0000", 2024-02-05T04:54:06.145 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1019000000000001e-05 2024-02-05T04:54:06.145 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.145 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.145 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:06.146 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.961903+0000", 2024-02-05T04:54:06.146 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.8840000000000006e-06 2024-02-05T04:54:06.146 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.146 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.146 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:06.146 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.961910+0000", 2024-02-05T04:54:06.147 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.1579999999999999e-06 2024-02-05T04:54:06.147 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.147 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.147 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:06.147 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.961981+0000", 2024-02-05T04:54:06.147 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.1058999999999998e-05 2024-02-05T04:54:06.148 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.148 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.148 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:06.148 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.961990+0000", 2024-02-05T04:54:06.148 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.9979999999999992e-06 2024-02-05T04:54:06.148 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.148 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.149 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:06.149 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.968815+0000", 2024-02-05T04:54:06.149 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0068249920000000002 2024-02-05T04:54:06.149 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.149 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.149 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:06.149 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:43.968849+0000", 2024-02-05T04:54:06.149 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.3989000000000003e-05 2024-02-05T04:54:06.150 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:06.150 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:06.150 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:06.150 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.150 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.150 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.ds2 58/31 ECSubWrite(tid=56, reqid=client.4634.0:526, at_version=58'35, trim_to=0'0, roll_forward_to=57'34))", 2024-02-05T04:54:06.150 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:49.970359+0000", 2024-02-05T04:54:06.150 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 16.145770560999999, 2024-02-05T04:54:06.150 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0098177070000000002, 2024-02-05T04:54:06.151 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:06.151 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:06.151 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:06.151 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.151 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:06.151 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.970359+0000", 2024-02-05T04:54:06.151 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:06.151 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.152 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.152 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:06.152 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.970359+0000", 2024-02-05T04:54:06.152 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:06.152 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.152 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.153 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:06.153 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.970359+0000", 2024-02-05T04:54:06.153 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1000000000000001e-07 2024-02-05T04:54:06.153 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.153 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.153 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:06.153 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.972816+0000", 2024-02-05T04:54:06.153 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0024574549999999999 2024-02-05T04:54:06.153 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.153 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.154 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:06.154 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.972826+0000", 2024-02-05T04:54:06.154 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.7759999999999992e-06 2024-02-05T04:54:06.154 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.154 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.154 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:06.154 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.972832+0000", 2024-02-05T04:54:06.154 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.818e-06 2024-02-05T04:54:06.154 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.154 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.154 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:06.154 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.972914+0000", 2024-02-05T04:54:06.154 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.2150999999999993e-05 2024-02-05T04:54:06.154 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.155 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.155 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:06.155 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.972924+0000", 2024-02-05T04:54:06.155 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.9189999999999994e-06 2024-02-05T04:54:06.155 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.155 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.155 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:06.155 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.980132+0000", 2024-02-05T04:54:06.155 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0072077340000000004 2024-02-05T04:54:06.155 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.155 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.155 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:06.156 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.980176+0000", 2024-02-05T04:54:06.156 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.4743999999999997e-05 2024-02-05T04:54:06.156 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:06.156 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:06.156 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:06.156 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.156 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.156 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.fs2 58/31 ECSubWrite(tid=104, reqid=client.4634.0:518, at_version=58'51, trim_to=0'0, roll_forward_to=57'49))", 2024-02-05T04:54:06.156 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:49.974049+0000", 2024-02-05T04:54:06.156 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 16.142079791, 2024-02-05T04:54:06.156 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0089817149999999995, 2024-02-05T04:54:06.156 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:06.156 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:06.157 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:06.157 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.157 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:06.157 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.974049+0000", 2024-02-05T04:54:06.157 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:06.157 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.157 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.157 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:06.157 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.974049+0000", 2024-02-05T04:54:06.157 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:06.157 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.157 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.157 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:06.157 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.974050+0000", 2024-02-05T04:54:06.158 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.8700000000000004e-07 2024-02-05T04:54:06.158 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.158 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.158 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:06.158 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.976606+0000", 2024-02-05T04:54:06.158 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0025551010000000002 2024-02-05T04:54:06.158 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.158 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.158 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:06.158 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.976629+0000", 2024-02-05T04:54:06.158 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.3411e-05 2024-02-05T04:54:06.158 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.158 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.159 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:06.159 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.976641+0000", 2024-02-05T04:54:06.159 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1721e-05 2024-02-05T04:54:06.159 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.159 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.159 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:06.159 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.976716+0000", 2024-02-05T04:54:06.159 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.5259999999999994e-05 2024-02-05T04:54:06.159 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.159 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.159 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:06.159 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.976727+0000", 2024-02-05T04:54:06.159 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0828e-05 2024-02-05T04:54:06.159 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.159 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.159 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:06.160 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.982990+0000", 2024-02-05T04:54:06.160 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0062629080000000002 2024-02-05T04:54:06.160 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.160 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.160 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:06.160 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.983031+0000", 2024-02-05T04:54:06.160 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.1498999999999997e-05 2024-02-05T04:54:06.160 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:06.160 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:06.160 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:06.160 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.160 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.160 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.ds2 63/31 ECSubWrite(tid=63, reqid=osd.7.0:62, at_version=0'0, trim_to=0'0, roll_forward_to=63'41))", 2024-02-05T04:54:06.160 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:54.987215+0000", 2024-02-05T04:54:06.161 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 11.128914589000001, 2024-02-05T04:54:06.161 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0058779009999999996, 2024-02-05T04:54:06.161 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:06.161 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:06.161 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:06.161 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.161 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:06.161 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.987215+0000", 2024-02-05T04:54:06.161 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:06.161 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.161 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.161 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:06.161 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.987215+0000", 2024-02-05T04:54:06.161 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:06.161 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.161 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.162 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:06.162 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.987215+0000", 2024-02-05T04:54:06.162 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.5000000000000002e-07 2024-02-05T04:54:06.162 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.162 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.162 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:06.162 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.987228+0000", 2024-02-05T04:54:06.162 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2588999999999999e-05 2024-02-05T04:54:06.163 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.163 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.163 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:06.163 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.987240+0000", 2024-02-05T04:54:06.163 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2408e-05 2024-02-05T04:54:06.163 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.163 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.163 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:06.163 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.987249+0000", 2024-02-05T04:54:06.164 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.1009999999999998e-06 2024-02-05T04:54:06.164 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.164 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.164 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:06.164 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.987364+0000", 2024-02-05T04:54:06.164 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000114597 2024-02-05T04:54:06.164 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.164 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.164 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:06.165 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.987383+0000", 2024-02-05T04:54:06.165 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.931e-05 2024-02-05T04:54:06.165 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.165 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.165 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:06.165 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.993039+0000", 2024-02-05T04:54:06.165 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0056555199999999998 2024-02-05T04:54:06.165 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.165 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.166 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:06.166 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.993093+0000", 2024-02-05T04:54:06.166 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.3726e-05 2024-02-05T04:54:06.166 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:06.166 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:06.166 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:06.166 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.166 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.166 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.8s4 63/62 ECSubWrite(tid=80, reqid=client.4634.0:591, at_version=63'25, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:06.166 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:54.987369+0000", 2024-02-05T04:54:06.166 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 11.128760182000001, 2024-02-05T04:54:06.166 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0058481719999999996, 2024-02-05T04:54:06.167 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:06.167 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:06.167 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:06.167 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.167 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:06.167 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.987369+0000", 2024-02-05T04:54:06.168 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:06.168 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.168 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.168 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:06.168 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.987369+0000", 2024-02-05T04:54:06.168 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:06.168 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.168 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.168 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:06.168 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.987370+0000", 2024-02-05T04:54:06.168 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.9500000000000003e-07 2024-02-05T04:54:06.168 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.168 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.169 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:06.169 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.987393+0000", 2024-02-05T04:54:06.169 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.3603000000000001e-05 2024-02-05T04:54:06.169 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.169 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.169 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:06.169 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.987403+0000", 2024-02-05T04:54:06.169 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.4909999999999994e-06 2024-02-05T04:54:06.169 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.169 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.169 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:06.169 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.987410+0000", 2024-02-05T04:54:06.169 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.8110000000000003e-06 2024-02-05T04:54:06.169 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.170 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.170 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:06.170 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.987469+0000", 2024-02-05T04:54:06.170 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.8526999999999997e-05 2024-02-05T04:54:06.170 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.170 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.170 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:06.170 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.987483+0000", 2024-02-05T04:54:06.170 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.3917000000000001e-05 2024-02-05T04:54:06.170 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.170 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.170 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:06.170 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.993180+0000", 2024-02-05T04:54:06.170 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0056972120000000001 2024-02-05T04:54:06.170 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.170 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.171 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:06.171 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.993217+0000", 2024-02-05T04:54:06.171 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.7116000000000002e-05 2024-02-05T04:54:06.171 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:06.171 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:06.171 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:06.171 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.171 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.171 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.fs2 63/31 ECSubWrite(tid=121, reqid=osd.11.0:121, at_version=63'54, trim_to=0'0, roll_forward_to=58'51))", 2024-02-05T04:54:06.171 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:54.988829+0000", 2024-02-05T04:54:06.171 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 11.127300422999999, 2024-02-05T04:54:06.171 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0065760719999999996, 2024-02-05T04:54:06.171 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:06.171 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:06.171 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:06.171 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.172 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:06.172 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.988829+0000", 2024-02-05T04:54:06.172 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:06.172 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.172 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.172 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:06.172 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.988829+0000", 2024-02-05T04:54:06.172 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:06.172 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.172 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.172 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:06.172 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.988829+0000", 2024-02-05T04:54:06.172 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.7999999999999995e-07 2024-02-05T04:54:06.172 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.172 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.173 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:06.173 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.988848+0000", 2024-02-05T04:54:06.173 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.8831e-05 2024-02-05T04:54:06.173 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.173 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.173 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:06.173 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.988868+0000", 2024-02-05T04:54:06.173 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.927e-05 2024-02-05T04:54:06.173 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.173 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.173 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:06.173 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.988876+0000", 2024-02-05T04:54:06.173 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.2779999999999993e-06 2024-02-05T04:54:06.173 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.174 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.174 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:06.174 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.988947+0000", 2024-02-05T04:54:06.174 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.1499000000000001e-05 2024-02-05T04:54:06.174 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.174 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.174 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:06.174 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.988956+0000", 2024-02-05T04:54:06.174 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.4630000000000001e-06 2024-02-05T04:54:06.174 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.174 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.174 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:06.174 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.995372+0000", 2024-02-05T04:54:06.174 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0064165710000000003 2024-02-05T04:54:06.174 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.175 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.175 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:06.175 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.995405+0000", 2024-02-05T04:54:06.175 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.2580000000000003e-05 2024-02-05T04:54:06.175 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:06.175 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:06.175 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:06.175 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.175 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.175 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.8s4 63/62 ECSubWrite(tid=81, reqid=client.4634.0:592, at_version=63'26, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:06.175 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:54.989568+0000", 2024-02-05T04:54:06.175 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 11.126561019, 2024-02-05T04:54:06.175 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0058713589999999996, 2024-02-05T04:54:06.175 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:06.176 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:06.176 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:06.176 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.176 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:06.176 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.989568+0000", 2024-02-05T04:54:06.176 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:06.176 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.176 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.176 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:06.176 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.989568+0000", 2024-02-05T04:54:06.176 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:06.176 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.176 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.176 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:06.176 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.989569+0000", 2024-02-05T04:54:06.177 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.5899999999999995e-07 2024-02-05T04:54:06.177 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.177 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.177 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:06.177 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.989588+0000", 2024-02-05T04:54:06.177 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.9431000000000001e-05 2024-02-05T04:54:06.177 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.177 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.177 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:06.177 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.989601+0000", 2024-02-05T04:54:06.177 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2542e-05 2024-02-05T04:54:06.177 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.177 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.178 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:06.178 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.989610+0000", 2024-02-05T04:54:06.178 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.3759999999999997e-06 2024-02-05T04:54:06.178 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.179 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.179 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:06.179 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.990071+0000", 2024-02-05T04:54:06.179 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00046041300000000001 2024-02-05T04:54:06.179 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.179 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.179 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:06.179 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.990080+0000", 2024-02-05T04:54:06.179 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.6590000000000002e-06 2024-02-05T04:54:06.179 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.179 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.179 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:06.179 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.995408+0000", 2024-02-05T04:54:06.179 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0053277369999999999 2024-02-05T04:54:06.180 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.180 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.180 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:06.180 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.995440+0000", 2024-02-05T04:54:06.180 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.1442000000000002e-05 2024-02-05T04:54:06.180 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:06.180 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:06.180 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:06.180 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.180 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.180 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.fs2 63/31 ECSubWrite(tid=122, reqid=osd.11.0:122, at_version=63'56, trim_to=0'0, roll_forward_to=58'51))", 2024-02-05T04:54:06.180 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:54.989886+0000", 2024-02-05T04:54:06.180 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 11.12624293, 2024-02-05T04:54:06.180 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.010970194000000001, 2024-02-05T04:54:06.180 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:06.181 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:06.181 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:06.181 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.181 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:06.181 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.989886+0000", 2024-02-05T04:54:06.181 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:06.181 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.181 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.181 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:06.181 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.989886+0000", 2024-02-05T04:54:06.181 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:06.181 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.181 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.181 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:06.182 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.989887+0000", 2024-02-05T04:54:06.182 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.7899999999999998e-07 2024-02-05T04:54:06.182 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.182 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.182 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:06.182 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.989907+0000", 2024-02-05T04:54:06.182 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.0228999999999999e-05 2024-02-05T04:54:06.182 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.182 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.182 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:06.182 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.989916+0000", 2024-02-05T04:54:06.182 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.217e-06 2024-02-05T04:54:06.182 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.182 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.182 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:06.183 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.989924+0000", 2024-02-05T04:54:06.183 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.374e-06 2024-02-05T04:54:06.183 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.183 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.183 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:06.183 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.990166+0000", 2024-02-05T04:54:06.183 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00024187300000000001 2024-02-05T04:54:06.183 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.183 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.183 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:06.183 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.990176+0000", 2024-02-05T04:54:06.183 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0587e-05 2024-02-05T04:54:06.183 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.183 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.184 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:06.184 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.000816+0000", 2024-02-05T04:54:06.184 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.010639684 2024-02-05T04:54:06.184 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.184 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.184 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:06.184 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.000857+0000", 2024-02-05T04:54:06.184 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.0651000000000001e-05 2024-02-05T04:54:06.184 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:06.184 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:06.184 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:06.184 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.184 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.184 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.8s4 63/62 ECSubWrite(tid=84, reqid=client.4634.0:593, at_version=0'0, trim_to=0'0, roll_forward_to=63'27))", 2024-02-05T04:54:06.184 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:55.002637+0000", 2024-02-05T04:54:06.185 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 11.113492469000001, 2024-02-05T04:54:06.185 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0091868599999999998, 2024-02-05T04:54:06.185 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:06.185 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:06.185 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:06.185 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.185 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:06.185 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.002637+0000", 2024-02-05T04:54:06.185 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:06.185 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.185 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.185 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:06.185 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.002637+0000", 2024-02-05T04:54:06.185 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:06.186 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.186 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.186 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:06.186 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.002638+0000", 2024-02-05T04:54:06.186 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.9899999999999999e-07 2024-02-05T04:54:06.186 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.186 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.186 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:06.186 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.002669+0000", 2024-02-05T04:54:06.186 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.0969000000000001e-05 2024-02-05T04:54:06.186 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.186 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.186 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:06.186 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.002681+0000", 2024-02-05T04:54:06.186 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.269e-05 2024-02-05T04:54:06.187 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.187 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.187 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:06.187 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.002690+0000", 2024-02-05T04:54:06.187 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.5930000000000006e-06 2024-02-05T04:54:06.187 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.187 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.187 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:06.187 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.006573+0000", 2024-02-05T04:54:06.187 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0038832430000000002 2024-02-05T04:54:06.187 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.187 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.187 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:06.187 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.006584+0000", 2024-02-05T04:54:06.187 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0835999999999999e-05 2024-02-05T04:54:06.188 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.188 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.188 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:06.188 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.011794+0000", 2024-02-05T04:54:06.188 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0052104300000000003 2024-02-05T04:54:06.188 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.188 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.188 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:06.188 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.011824+0000", 2024-02-05T04:54:06.188 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.9300000000000001e-05 2024-02-05T04:54:06.188 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:06.188 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:06.188 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:06.188 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.189 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.189 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.1s6 63/62 ECSubWrite(tid=265, reqid=client.4634.0:594, at_version=63'53, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:06.189 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:55.003909+0000", 2024-02-05T04:54:06.189 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 11.112220391999999, 2024-02-05T04:54:06.189 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0080433959999999995, 2024-02-05T04:54:06.189 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:06.189 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:06.189 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:06.189 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.189 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:06.189 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.003909+0000", 2024-02-05T04:54:06.189 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:06.189 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.189 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.189 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:06.190 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.003909+0000", 2024-02-05T04:54:06.190 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:06.190 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.190 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.190 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:06.190 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.003910+0000", 2024-02-05T04:54:06.190 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.6000000000000003e-07 2024-02-05T04:54:06.190 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.190 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.190 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:06.190 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.004506+0000", 2024-02-05T04:54:06.190 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00059623499999999997 2024-02-05T04:54:06.190 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.190 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.191 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:06.191 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.004518+0000", 2024-02-05T04:54:06.191 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2428e-05 2024-02-05T04:54:06.191 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.191 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.191 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:06.191 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.004528+0000", 2024-02-05T04:54:06.191 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.2590000000000006e-06 2024-02-05T04:54:06.191 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.191 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.191 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:06.191 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.004594+0000", 2024-02-05T04:54:06.191 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.6225999999999999e-05 2024-02-05T04:54:06.191 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.191 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.191 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:06.192 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.004609+0000", 2024-02-05T04:54:06.192 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.5517000000000001e-05 2024-02-05T04:54:06.192 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.192 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.192 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:06.192 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.011932+0000", 2024-02-05T04:54:06.192 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0073225770000000003 2024-02-05T04:54:06.192 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.192 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.192 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:06.192 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.011952+0000", 2024-02-05T04:54:06.192 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.0394000000000002e-05 2024-02-05T04:54:06.192 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:06.192 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:06.193 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:06.193 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.193 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.193 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 63/62 ECSubWriteReply(tid=54, last_complete=63'8, committed=1, applied=1))", 2024-02-05T04:54:06.193 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:55.006941+0000", 2024-02-05T04:54:06.193 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 11.109188395, 2024-02-05T04:54:06.193 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0051003239999999998, 2024-02-05T04:54:06.193 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:06.193 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:06.193 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:06.193 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.193 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:06.193 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.006941+0000", 2024-02-05T04:54:06.193 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:06.193 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.194 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.194 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:06.194 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.006941+0000", 2024-02-05T04:54:06.194 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:06.194 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.194 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.194 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:06.194 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.006942+0000", 2024-02-05T04:54:06.194 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.5899999999999996e-07 2024-02-05T04:54:06.194 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.194 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.194 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:06.194 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.006948+0000", 2024-02-05T04:54:06.194 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.7360000000000004e-06 2024-02-05T04:54:06.194 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.194 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.195 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:06.195 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.006960+0000", 2024-02-05T04:54:06.195 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1917e-05 2024-02-05T04:54:06.195 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.195 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.195 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:06.195 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.006966+0000", 2024-02-05T04:54:06.195 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.6470000000000003e-06 2024-02-05T04:54:06.195 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.195 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.195 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:06.195 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.012025+0000", 2024-02-05T04:54:06.195 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0050594960000000001 2024-02-05T04:54:06.195 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.195 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.196 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:06.196 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.012041+0000", 2024-02-05T04:54:06.196 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.5869e-05 2024-02-05T04:54:06.196 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:06.196 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:06.196 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:06.196 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.196 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.196 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.1s6 63/62 ECSubWrite(tid=267, reqid=client.4634.0:595, at_version=63'54, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:06.196 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:55.010077+0000", 2024-02-05T04:54:06.196 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 11.106052736000001, 2024-02-05T04:54:06.196 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0051989840000000002, 2024-02-05T04:54:06.196 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:06.197 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:06.197 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:06.197 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.197 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:06.197 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.010077+0000", 2024-02-05T04:54:06.197 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:06.197 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.197 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.197 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:06.197 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.010077+0000", 2024-02-05T04:54:06.197 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:06.197 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.197 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.197 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:06.197 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.010077+0000", 2024-02-05T04:54:06.198 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.8400000000000003e-07 2024-02-05T04:54:06.198 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.198 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.198 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:06.198 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.010619+0000", 2024-02-05T04:54:06.198 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000542038 2024-02-05T04:54:06.198 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.198 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.198 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:06.198 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.010633+0000", 2024-02-05T04:54:06.198 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.3730999999999999e-05 2024-02-05T04:54:06.198 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.198 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.198 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:06.198 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.010642+0000", 2024-02-05T04:54:06.199 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.4700000000000002e-06 2024-02-05T04:54:06.199 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.199 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.199 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:06.199 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.010867+0000", 2024-02-05T04:54:06.199 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00022510400000000001 2024-02-05T04:54:06.199 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.199 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.199 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:06.199 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.010877+0000", 2024-02-05T04:54:06.199 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0173999999999999e-05 2024-02-05T04:54:06.199 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.199 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.199 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:06.199 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.015243+0000", 2024-02-05T04:54:06.200 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0043660189999999996 2024-02-05T04:54:06.200 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.200 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.200 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:06.200 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.015276+0000", 2024-02-05T04:54:06.200 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.2564000000000002e-05 2024-02-05T04:54:06.200 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:06.200 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:06.200 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:06.200 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.200 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.200 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.14s1 68/67 ECSubWrite(tid=15, reqid=osd.3.0:15, at_version=68'75, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:06.200 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:59.963702+0000", 2024-02-05T04:54:06.200 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 6.1524269409999999, 2024-02-05T04:54:06.201 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0061511170000000002, 2024-02-05T04:54:06.201 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:06.201 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:06.201 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:06.201 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.201 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:06.201 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:59.963702+0000", 2024-02-05T04:54:06.201 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:06.201 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.201 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.201 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:06.201 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:59.963702+0000", 2024-02-05T04:54:06.201 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:06.201 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.201 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.202 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:06.202 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:59.963703+0000", 2024-02-05T04:54:06.202 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.4799999999999997e-07 2024-02-05T04:54:06.202 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.202 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.202 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:06.202 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:59.963730+0000", 2024-02-05T04:54:06.202 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.6951999999999999e-05 2024-02-05T04:54:06.202 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.202 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.202 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:06.202 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:59.963744+0000", 2024-02-05T04:54:06.202 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.4059e-05 2024-02-05T04:54:06.202 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.202 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.202 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:06.203 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:59.963753+0000", 2024-02-05T04:54:06.203 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.1030000000000008e-06 2024-02-05T04:54:06.203 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.203 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.203 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:06.203 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:59.963803+0000", 2024-02-05T04:54:06.203 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.9387e-05 2024-02-05T04:54:06.203 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.203 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.203 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:06.203 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:59.963809+0000", 2024-02-05T04:54:06.203 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.5400000000000001e-06 2024-02-05T04:54:06.203 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.203 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.204 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:06.204 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:59.969807+0000", 2024-02-05T04:54:06.204 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0059979689999999997 2024-02-05T04:54:06.204 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.204 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.204 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:06.204 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:59.969853+0000", 2024-02-05T04:54:06.204 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.6359000000000001e-05 2024-02-05T04:54:06.204 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:06.204 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:06.204 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:06.204 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.204 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.205 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.14s1 68/67 ECSubWrite(tid=16, reqid=osd.3.0:16, at_version=68'77, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:06.205 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:59.963773+0000", 2024-02-05T04:54:06.205 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 6.1523561559999997, 2024-02-05T04:54:06.205 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0091058300000000005, 2024-02-05T04:54:06.205 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:06.205 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:06.205 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:06.205 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.205 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:06.205 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:59.963773+0000", 2024-02-05T04:54:06.205 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:06.205 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.205 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.205 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:06.205 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:59.963773+0000", 2024-02-05T04:54:06.206 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:06.206 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.206 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.206 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:06.206 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:59.963773+0000", 2024-02-05T04:54:06.206 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.7300000000000002e-07 2024-02-05T04:54:06.206 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.206 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.206 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:06.206 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:59.963792+0000", 2024-02-05T04:54:06.206 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.8663000000000001e-05 2024-02-05T04:54:06.206 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.206 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.206 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:06.207 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:59.963801+0000", 2024-02-05T04:54:06.207 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.3890000000000005e-06 2024-02-05T04:54:06.207 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.207 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.207 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:06.207 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:59.963807+0000", 2024-02-05T04:54:06.207 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.9639999999999997e-06 2024-02-05T04:54:06.207 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.207 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.207 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:06.207 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:59.964069+0000", 2024-02-05T04:54:06.207 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00026121199999999998 2024-02-05T04:54:06.207 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.207 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.208 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:06.208 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:59.964076+0000", 2024-02-05T04:54:06.208 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.2429999999999999e-06 2024-02-05T04:54:06.208 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.208 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.208 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:06.208 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:59.972840+0000", 2024-02-05T04:54:06.208 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0087637370000000006 2024-02-05T04:54:06.208 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.208 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.208 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:06.208 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:59.972879+0000", 2024-02-05T04:54:06.208 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.9249000000000003e-05 2024-02-05T04:54:06.208 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:06.209 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:06.209 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:06.209 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.209 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.209 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.1s6 68/62 ECSubWrite(tid=293, reqid=client.4634.0:623, at_version=0'0, trim_to=0'0, roll_forward_to=68'66))", 2024-02-05T04:54:06.209 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:59.966322+0000", 2024-02-05T04:54:06.209 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 6.1498068789999998, 2024-02-05T04:54:06.209 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0061223220000000004, 2024-02-05T04:54:06.209 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:06.209 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:06.209 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:06.209 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.209 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:06.210 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:59.966322+0000", 2024-02-05T04:54:06.210 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:06.210 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.210 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.210 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:06.210 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:59.966322+0000", 2024-02-05T04:54:06.210 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:06.210 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.210 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.210 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:06.210 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:59.966323+0000", 2024-02-05T04:54:06.210 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.2500000000000005e-07 2024-02-05T04:54:06.210 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.210 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.211 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:06.211 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:59.966333+0000", 2024-02-05T04:54:06.211 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.9320000000000008e-06 2024-02-05T04:54:06.211 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.211 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.211 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:06.211 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:59.966343+0000", 2024-02-05T04:54:06.211 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0403e-05 2024-02-05T04:54:06.211 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.211 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.211 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:06.211 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:59.966355+0000", 2024-02-05T04:54:06.211 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1423999999999999e-05 2024-02-05T04:54:06.211 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.212 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.212 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:06.212 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:59.966402+0000", 2024-02-05T04:54:06.212 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.6712999999999998e-05 2024-02-05T04:54:06.212 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.212 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.212 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:06.212 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:59.966411+0000", 2024-02-05T04:54:06.212 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.5019999999999998e-06 2024-02-05T04:54:06.212 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.212 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.212 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:06.212 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:59.972408+0000", 2024-02-05T04:54:06.212 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0059965940000000001 2024-02-05T04:54:06.212 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.213 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.213 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:06.213 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:59.972445+0000", 2024-02-05T04:54:06.213 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.7029000000000001e-05 2024-02-05T04:54:06.213 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:06.213 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:06.213 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:06.213 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.213 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.213 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.8s4 69/62 ECSubWrite(tid=93, reqid=osd.4.0:88, at_version=0'0, trim_to=0'0, roll_forward_to=69'31))", 2024-02-05T04:54:06.213 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:54:00.972971+0000", 2024-02-05T04:54:06.213 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 5.1431581059999996, 2024-02-05T04:54:06.213 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0059465899999999999, 2024-02-05T04:54:06.213 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:06.213 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:06.214 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:06.214 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.214 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:06.214 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.972971+0000", 2024-02-05T04:54:06.214 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:06.214 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.214 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.214 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:06.214 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.972971+0000", 2024-02-05T04:54:06.214 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:06.214 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.214 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.214 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:06.214 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.972972+0000", 2024-02-05T04:54:06.214 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.018e-06 2024-02-05T04:54:06.215 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.215 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.215 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:06.215 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.972984+0000", 2024-02-05T04:54:06.215 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1416e-05 2024-02-05T04:54:06.215 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.215 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.215 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:06.215 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.973007+0000", 2024-02-05T04:54:06.215 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.3881000000000001e-05 2024-02-05T04:54:06.215 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.215 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.215 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:06.215 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.973017+0000", 2024-02-05T04:54:06.215 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.9939999999999995e-06 2024-02-05T04:54:06.216 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.216 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.216 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:06.216 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.973129+0000", 2024-02-05T04:54:06.216 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000112003 2024-02-05T04:54:06.216 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.216 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.216 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:06.216 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.973141+0000", 2024-02-05T04:54:06.216 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1409e-05 2024-02-05T04:54:06.216 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.216 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.216 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:06.216 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.978866+0000", 2024-02-05T04:54:06.217 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0057253419999999996 2024-02-05T04:54:06.217 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.217 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.217 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:06.217 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.978918+0000", 2024-02-05T04:54:06.217 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.1527000000000003e-05 2024-02-05T04:54:06.217 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:06.217 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:06.217 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:06.217 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.217 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.217 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpRead(3.ds2 69/31 ECSubRead(tid=72, to_read={3:b7ad4782:::smithi19639362-40:head=0,667648,0}, subchunks={3:b7ad4782:::smithi19639362-40:head=[0,1]}, attrs_to_read=))", 2024-02-05T04:54:06.217 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:54:00.999954+0000", 2024-02-05T04:54:06.217 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 5.1161753130000003, 2024-02-05T04:54:06.217 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0050629230000000004, 2024-02-05T04:54:06.217 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:06.218 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:06.218 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:06.218 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.218 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:06.218 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.999954+0000", 2024-02-05T04:54:06.218 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:06.218 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.218 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.218 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:06.218 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.999954+0000", 2024-02-05T04:54:06.218 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:06.218 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.218 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.218 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:06.218 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.999954+0000", 2024-02-05T04:54:06.219 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1899999999999999e-07 2024-02-05T04:54:06.219 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.219 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.219 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:06.219 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.999962+0000", 2024-02-05T04:54:06.219 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.0010000000000001e-06 2024-02-05T04:54:06.219 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.219 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.219 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:06.219 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.999973+0000", 2024-02-05T04:54:06.219 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1192999999999999e-05 2024-02-05T04:54:06.219 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.219 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.219 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:06.220 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.999980+0000", 2024-02-05T04:54:06.220 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.1720000000000001e-06 2024-02-05T04:54:06.220 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.220 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.220 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:06.220 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:01.000044+0000", 2024-02-05T04:54:06.220 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.3705000000000001e-05 2024-02-05T04:54:06.220 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:06.220 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:06.220 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:06.220 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:01.005017+0000", 2024-02-05T04:54:06.220 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.004972733 2024-02-05T04:54:06.220 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:06.220 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:06.220 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:06.221 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:06.221 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:06.221 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-02-05T04:54:06.222 INFO:tasks.rados.rados.0.smithi196.stdout:274: finishing write tid 1 to smithi19639362-24 2024-02-05T04:54:06.222 INFO:tasks.rados.rados.0.smithi196.stdout:274: finishing write tid 2 to smithi19639362-24 2024-02-05T04:54:06.222 INFO:tasks.rados.rados.0.smithi196.stdout:272: done (3 left) 2024-02-05T04:54:06.222 INFO:tasks.rados.rados.0.smithi196.stdout:276: done (2 left) 2024-02-05T04:54:06.222 INFO:tasks.rados.rados.0.smithi196.stdout:277: snap_remove snap 19 2024-02-05T04:54:06.223 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-02-05T04:54:06.223 INFO:tasks.ceph.osd.14.smithi150.stderr:2024-02-05T04:54:06.221+0000 7f5f81083640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 34020) UID: 0 2024-02-05T04:54:06.323 INFO:tasks.ceph.osd.11.smithi196.stderr:2024-02-05T04:54:06.322+0000 7f85a8d86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 34609) UID: 0 2024-02-05T04:54:06.331 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T04:54:06.331 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-02-05T04:54:06.331 INFO:teuthology.orchestra.run.smithi196.stdout: "num_ops": 0 2024-02-05T04:54:06.331 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T04:54:06.340 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-02-05T04:54:06.424 INFO:tasks.ceph.osd.14.smithi150.stderr:2024-02-05T04:54:06.423+0000 7f5f81083640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 34020) UID: 0 2024-02-05T04:54:06.486 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T04:54:06.486 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-02-05T04:54:06.486 INFO:teuthology.orchestra.run.smithi196.stdout: "complaint_time": 30, 2024-02-05T04:54:06.486 INFO:teuthology.orchestra.run.smithi196.stdout: "num_blocked_ops": 0 2024-02-05T04:54:06.486 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T04:54:06.495 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-02-05T04:54:06.526 INFO:tasks.ceph.osd.9.smithi091.stderr:2024-02-05T04:54:06.526+0000 7efc18827640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 34279) UID: 0 2024-02-05T04:54:06.602 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T04:54:06.602 INFO:teuthology.orchestra.run.smithi196.stdout: "size": 20, 2024-02-05T04:54:06.602 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 600, 2024-02-05T04:54:06.602 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-02-05T04:54:06.602 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.602 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.ds1 57/31 ECSubWrite(tid=54, reqid=client.4634.0:525, at_version=57'34, trim_to=0'0, roll_forward_to=57'32))", 2024-02-05T04:54:06.602 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:48.959354+0000", 2024-02-05T04:54:06.602 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 17.640579275, 2024-02-05T04:54:06.602 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0066299729999999999, 2024-02-05T04:54:06.602 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:06.602 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:06.602 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:06.602 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.603 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:06.603 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.959354+0000", 2024-02-05T04:54:06.603 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:06.603 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.603 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.603 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:06.603 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.959354+0000", 2024-02-05T04:54:06.603 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:06.603 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.603 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.603 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:06.603 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.959355+0000", 2024-02-05T04:54:06.603 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5600000000000005e-07 2024-02-05T04:54:06.603 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.603 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.604 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:06.604 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.959379+0000", 2024-02-05T04:54:06.604 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3366e-05 2024-02-05T04:54:06.604 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.604 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.604 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:06.604 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.959391+0000", 2024-02-05T04:54:06.604 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2672e-05 2024-02-05T04:54:06.604 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.604 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.604 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:06.604 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.959398+0000", 2024-02-05T04:54:06.604 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3450000000000002e-06 2024-02-05T04:54:06.604 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.604 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.605 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:06.605 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.959467+0000", 2024-02-05T04:54:06.605 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9665000000000005e-05 2024-02-05T04:54:06.605 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.605 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.605 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:06.605 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.959477+0000", 2024-02-05T04:54:06.605 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6930000000000002e-06 2024-02-05T04:54:06.605 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.605 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.606 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:06.606 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.965940+0000", 2024-02-05T04:54:06.606 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.006463112 2024-02-05T04:54:06.606 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.606 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.606 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:06.606 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.965984+0000", 2024-02-05T04:54:06.606 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4363999999999997e-05 2024-02-05T04:54:06.606 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:06.606 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:06.606 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:06.606 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.606 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.606 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.3s6 57/31 ECSubWrite(tid=77, reqid=osd.4.0:76, at_version=0'0, trim_to=0'0, roll_forward_to=57'35))", 2024-02-05T04:54:06.606 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:48.960703+0000", 2024-02-05T04:54:06.606 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 17.639230942000001, 2024-02-05T04:54:06.607 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.014641594000000001, 2024-02-05T04:54:06.607 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:06.607 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:06.607 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:06.607 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.607 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:06.607 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.960703+0000", 2024-02-05T04:54:06.607 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:06.607 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.607 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.607 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:06.607 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.960703+0000", 2024-02-05T04:54:06.607 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:06.607 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.607 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.608 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:06.608 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.960704+0000", 2024-02-05T04:54:06.608 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1399999999999995e-07 2024-02-05T04:54:06.608 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.608 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.608 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:06.608 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.960715+0000", 2024-02-05T04:54:06.608 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1534e-05 2024-02-05T04:54:06.608 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.608 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.608 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:06.608 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.960734+0000", 2024-02-05T04:54:06.608 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9264999999999999e-05 2024-02-05T04:54:06.608 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.608 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.609 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:06.609 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.960744+0000", 2024-02-05T04:54:06.609 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.7610000000000002e-06 2024-02-05T04:54:06.609 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.609 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.609 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:06.609 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.960808+0000", 2024-02-05T04:54:06.609 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4257000000000005e-05 2024-02-05T04:54:06.609 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.609 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.609 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:06.609 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.960819+0000", 2024-02-05T04:54:06.609 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0158e-05 2024-02-05T04:54:06.609 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.609 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.610 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:06.610 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.975292+0000", 2024-02-05T04:54:06.610 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.014473803 2024-02-05T04:54:06.610 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.610 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.610 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:06.610 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.975344+0000", 2024-02-05T04:54:06.610 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1901999999999999e-05 2024-02-05T04:54:06.610 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:06.610 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:06.610 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:06.610 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.610 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.610 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.ds1 57/31 ECSubWrite(tid=55, reqid=client.4634.0:525, at_version=0'0, trim_to=0'0, roll_forward_to=57'34))", 2024-02-05T04:54:06.610 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:48.966615+0000", 2024-02-05T04:54:06.610 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 17.633318710000001, 2024-02-05T04:54:06.611 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0093695359999999995, 2024-02-05T04:54:06.611 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:06.611 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:06.611 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:06.611 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.611 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:06.611 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.966615+0000", 2024-02-05T04:54:06.611 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:06.611 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.611 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.611 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:06.611 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.966615+0000", 2024-02-05T04:54:06.611 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:06.611 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.611 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.612 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:06.612 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.966616+0000", 2024-02-05T04:54:06.612 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.7000000000000003e-07 2024-02-05T04:54:06.612 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.612 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.612 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:06.612 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.966638+0000", 2024-02-05T04:54:06.612 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1980999999999998e-05 2024-02-05T04:54:06.612 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.612 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.612 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:06.612 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.966656+0000", 2024-02-05T04:54:06.612 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8550999999999999e-05 2024-02-05T04:54:06.612 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.612 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.613 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:06.613 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.966674+0000", 2024-02-05T04:54:06.613 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7510999999999999e-05 2024-02-05T04:54:06.613 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.613 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.613 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:06.613 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.966728+0000", 2024-02-05T04:54:06.613 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4001e-05 2024-02-05T04:54:06.613 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.613 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.613 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:06.613 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.966737+0000", 2024-02-05T04:54:06.613 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.6759999999999996e-06 2024-02-05T04:54:06.613 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.613 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.614 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:06.614 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.975954+0000", 2024-02-05T04:54:06.614 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0092173919999999996 2024-02-05T04:54:06.614 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.614 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.614 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:06.614 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.975984+0000", 2024-02-05T04:54:06.614 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.0454e-05 2024-02-05T04:54:06.614 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:06.614 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:06.614 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:06.614 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.614 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.614 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.5s3 58/57 ECSubWrite(tid=57, reqid=osd.15.0:57, at_version=58'36, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:06.614 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:49.955234+0000", 2024-02-05T04:54:06.614 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 16.644700094000001, 2024-02-05T04:54:06.615 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0090458080000000007, 2024-02-05T04:54:06.615 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:06.615 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:06.615 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:06.615 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.615 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:06.615 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.955234+0000", 2024-02-05T04:54:06.615 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:06.615 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.615 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.615 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:06.615 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.955234+0000", 2024-02-05T04:54:06.615 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:06.615 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.616 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.616 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:06.616 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.955234+0000", 2024-02-05T04:54:06.616 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1699999999999997e-07 2024-02-05T04:54:06.616 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.616 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.616 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:06.616 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.955259+0000", 2024-02-05T04:54:06.616 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.4264e-05 2024-02-05T04:54:06.616 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.616 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.616 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:06.616 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.955275+0000", 2024-02-05T04:54:06.616 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5896000000000002e-05 2024-02-05T04:54:06.616 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.617 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.617 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:06.617 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.955285+0000", 2024-02-05T04:54:06.617 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0643e-05 2024-02-05T04:54:06.617 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.617 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.617 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:06.617 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.955388+0000", 2024-02-05T04:54:06.617 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010303599999999999 2024-02-05T04:54:06.617 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.617 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.617 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:06.617 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.955400+0000", 2024-02-05T04:54:06.617 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1582e-05 2024-02-05T04:54:06.617 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.617 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.618 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:06.618 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.964224+0000", 2024-02-05T04:54:06.618 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0088238989999999996 2024-02-05T04:54:06.618 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.618 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.618 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:06.618 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.964279+0000", 2024-02-05T04:54:06.618 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5571e-05 2024-02-05T04:54:06.618 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:06.618 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:06.618 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:06.618 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.618 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.618 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.5s3 58/57 ECSubWrite(tid=58, reqid=osd.15.0:58, at_version=58'38, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:06.618 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:49.956373+0000", 2024-02-05T04:54:06.619 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 16.643560372, 2024-02-05T04:54:06.619 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.010807686, 2024-02-05T04:54:06.619 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:06.619 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:06.619 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:06.619 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.619 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:06.619 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.956373+0000", 2024-02-05T04:54:06.619 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:06.619 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.619 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.619 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:06.619 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.956373+0000", 2024-02-05T04:54:06.619 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:06.619 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.620 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.620 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:06.620 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.956374+0000", 2024-02-05T04:54:06.620 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.9899999999999999e-07 2024-02-05T04:54:06.620 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.620 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.620 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:06.620 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.956401+0000", 2024-02-05T04:54:06.620 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.6404e-05 2024-02-05T04:54:06.620 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.620 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.620 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:06.620 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.956417+0000", 2024-02-05T04:54:06.620 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.613e-05 2024-02-05T04:54:06.621 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.621 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.621 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:06.621 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.956441+0000", 2024-02-05T04:54:06.621 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3836e-05 2024-02-05T04:54:06.621 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.621 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.621 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:06.621 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.956524+0000", 2024-02-05T04:54:06.621 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3902999999999999e-05 2024-02-05T04:54:06.621 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.621 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.621 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:06.621 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.956536+0000", 2024-02-05T04:54:06.621 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.118e-05 2024-02-05T04:54:06.622 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.622 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.622 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:06.622 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.967117+0000", 2024-02-05T04:54:06.622 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.010581620999999999 2024-02-05T04:54:06.622 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.622 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.622 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:06.622 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.967181+0000", 2024-02-05T04:54:06.622 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3713000000000005e-05 2024-02-05T04:54:06.622 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:06.622 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:06.622 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:06.622 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.622 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.622 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.5s3 58/57 ECSubWrite(tid=60, reqid=osd.15.0:60, at_version=58'40, trim_to=0'0, roll_forward_to=58'38))", 2024-02-05T04:54:06.623 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:49.969331+0000", 2024-02-05T04:54:06.623 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 16.630602674999999, 2024-02-05T04:54:06.623 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0094108439999999998, 2024-02-05T04:54:06.623 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:06.623 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:06.623 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:06.623 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.623 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:06.623 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.969331+0000", 2024-02-05T04:54:06.623 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:06.623 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.623 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.623 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:06.623 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.969331+0000", 2024-02-05T04:54:06.623 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:06.623 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.624 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.624 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:06.624 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.969332+0000", 2024-02-05T04:54:06.624 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2419999999999999e-06 2024-02-05T04:54:06.624 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.624 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.624 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:06.624 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.969362+0000", 2024-02-05T04:54:06.624 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.9696000000000002e-05 2024-02-05T04:54:06.624 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.624 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.624 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:06.624 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.969382+0000", 2024-02-05T04:54:06.624 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9945999999999998e-05 2024-02-05T04:54:06.624 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.625 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.625 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:06.625 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.969393+0000", 2024-02-05T04:54:06.625 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1015999999999999e-05 2024-02-05T04:54:06.625 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.625 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.625 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:06.625 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.969485+0000", 2024-02-05T04:54:06.625 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2377e-05 2024-02-05T04:54:06.625 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.625 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.625 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:06.625 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.969496+0000", 2024-02-05T04:54:06.625 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.065e-05 2024-02-05T04:54:06.625 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.626 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.626 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:06.626 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.978687+0000", 2024-02-05T04:54:06.626 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0091914130000000007 2024-02-05T04:54:06.626 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.626 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.626 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:06.626 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.978742+0000", 2024-02-05T04:54:06.626 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4503999999999999e-05 2024-02-05T04:54:06.626 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:06.626 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:06.626 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:06.626 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.626 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.626 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.ds1 58/31 ECSubWrite(tid=56, reqid=client.4634.0:526, at_version=58'35, trim_to=0'0, roll_forward_to=57'34))", 2024-02-05T04:54:06.626 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:49.970064+0000", 2024-02-05T04:54:06.627 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 16.62986991, 2024-02-05T04:54:06.627 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.011474428, 2024-02-05T04:54:06.627 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:06.627 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:06.627 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:06.627 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.627 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:06.627 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.970064+0000", 2024-02-05T04:54:06.627 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:06.627 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.627 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.627 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:06.627 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.970064+0000", 2024-02-05T04:54:06.627 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:06.627 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.628 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.628 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:06.628 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.970064+0000", 2024-02-05T04:54:06.628 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4099999999999998e-07 2024-02-05T04:54:06.628 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.628 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.628 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:06.628 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.970616+0000", 2024-02-05T04:54:06.628 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00055171500000000004 2024-02-05T04:54:06.628 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.628 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.628 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:06.628 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.970636+0000", 2024-02-05T04:54:06.629 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0244000000000001e-05 2024-02-05T04:54:06.629 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.629 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.629 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:06.629 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.970648+0000", 2024-02-05T04:54:06.629 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1678000000000001e-05 2024-02-05T04:54:06.629 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.629 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.629 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:06.629 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.970755+0000", 2024-02-05T04:54:06.629 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000107188 2024-02-05T04:54:06.629 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.629 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.629 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:06.629 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.970769+0000", 2024-02-05T04:54:06.629 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3346000000000001e-05 2024-02-05T04:54:06.630 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.630 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.630 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:06.630 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.981495+0000", 2024-02-05T04:54:06.630 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.010726352999999999 2024-02-05T04:54:06.630 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.630 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.630 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:06.630 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.981538+0000", 2024-02-05T04:54:06.630 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3163e-05 2024-02-05T04:54:06.630 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:06.630 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:06.630 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:06.630 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.630 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.630 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.12s5 68/67 ECSubWrite(tid=85, reqid=osd.2.0:85, at_version=68'57, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:06.631 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:59.961270+0000", 2024-02-05T04:54:06.631 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 6.6386639929999998, 2024-02-05T04:54:06.631 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0062076639999999999, 2024-02-05T04:54:06.631 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:06.631 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:06.631 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:06.631 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.631 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:06.631 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.961270+0000", 2024-02-05T04:54:06.631 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:06.631 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.631 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.631 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:06.631 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.961270+0000", 2024-02-05T04:54:06.631 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:06.632 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.632 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.632 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:06.632 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.961270+0000", 2024-02-05T04:54:06.632 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3099999999999997e-07 2024-02-05T04:54:06.632 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.632 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.632 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:06.632 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.961288+0000", 2024-02-05T04:54:06.632 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7640000000000001e-05 2024-02-05T04:54:06.632 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.632 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.632 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:06.632 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.961300+0000", 2024-02-05T04:54:06.632 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2245999999999999e-05 2024-02-05T04:54:06.632 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.633 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.633 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:06.633 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.961309+0000", 2024-02-05T04:54:06.633 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.738e-06 2024-02-05T04:54:06.633 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.633 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.633 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:06.633 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.961373+0000", 2024-02-05T04:54:06.633 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3801999999999997e-05 2024-02-05T04:54:06.633 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.633 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.633 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:06.633 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.961383+0000", 2024-02-05T04:54:06.633 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0037999999999999e-05 2024-02-05T04:54:06.633 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.634 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.634 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:06.634 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.967435+0000", 2024-02-05T04:54:06.634 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0060525559999999997 2024-02-05T04:54:06.634 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.634 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.634 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:06.634 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.967477+0000", 2024-02-05T04:54:06.634 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2012999999999999e-05 2024-02-05T04:54:06.634 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:06.634 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:06.634 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:06.634 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.634 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.634 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.12s5 68/67 ECSubWrite(tid=86, reqid=osd.2.0:85, at_version=0'0, trim_to=0'0, roll_forward_to=68'57))", 2024-02-05T04:54:06.634 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:59.970672+0000", 2024-02-05T04:54:06.635 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 6.6292619820000001, 2024-02-05T04:54:06.635 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0058002239999999997, 2024-02-05T04:54:06.635 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:06.635 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:06.635 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:06.635 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.635 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:06.635 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.970672+0000", 2024-02-05T04:54:06.635 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:06.635 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.635 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.635 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:06.635 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.970672+0000", 2024-02-05T04:54:06.635 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:06.635 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.635 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.636 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:06.636 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.970673+0000", 2024-02-05T04:54:06.636 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.5499999999999997e-07 2024-02-05T04:54:06.636 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.636 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.636 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:06.636 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.970683+0000", 2024-02-05T04:54:06.636 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0499000000000001e-05 2024-02-05T04:54:06.636 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.636 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.636 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:06.636 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.970694+0000", 2024-02-05T04:54:06.636 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.094e-05 2024-02-05T04:54:06.636 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.636 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.636 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:06.637 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.970702+0000", 2024-02-05T04:54:06.637 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9160000000000001e-06 2024-02-05T04:54:06.637 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.637 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.637 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:06.637 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.970845+0000", 2024-02-05T04:54:06.637 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00014292299999999999 2024-02-05T04:54:06.637 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.637 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.637 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:06.637 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.970854+0000", 2024-02-05T04:54:06.637 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1989999999999995e-06 2024-02-05T04:54:06.637 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.637 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.637 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:06.637 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.976434+0000", 2024-02-05T04:54:06.638 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0055798000000000002 2024-02-05T04:54:06.638 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.638 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.638 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:06.638 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.976472+0000", 2024-02-05T04:54:06.638 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8092000000000001e-05 2024-02-05T04:54:06.638 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:06.638 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:06.638 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:06.638 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.638 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.638 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.5s3 69/62 ECSubWrite(tid=71, reqid=client.4634.0:638, at_version=0'0, trim_to=0'0, roll_forward_to=69'50))", 2024-02-05T04:54:06.638 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:54:00.999685+0000", 2024-02-05T04:54:06.638 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 5.6002490510000005, 2024-02-05T04:54:06.638 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0060564049999999999, 2024-02-05T04:54:06.639 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:06.639 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:06.639 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:06.639 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.639 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:06.639 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:00.999685+0000", 2024-02-05T04:54:06.639 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:06.639 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.639 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.639 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:06.639 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:00.999685+0000", 2024-02-05T04:54:06.639 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:06.639 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.639 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.639 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:06.639 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:00.999685+0000", 2024-02-05T04:54:06.640 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2799999999999995e-07 2024-02-05T04:54:06.640 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.640 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.640 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:06.640 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:00.999695+0000", 2024-02-05T04:54:06.640 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.9550000000000004e-06 2024-02-05T04:54:06.640 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.640 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.640 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:06.640 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:00.999708+0000", 2024-02-05T04:54:06.640 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3095000000000001e-05 2024-02-05T04:54:06.640 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.640 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.640 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:06.640 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:00.999719+0000", 2024-02-05T04:54:06.641 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0643e-05 2024-02-05T04:54:06.641 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.641 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.641 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:06.641 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:00.999773+0000", 2024-02-05T04:54:06.641 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3596e-05 2024-02-05T04:54:06.641 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.641 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.641 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:06.641 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:00.999781+0000", 2024-02-05T04:54:06.641 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.6049999999999998e-06 2024-02-05T04:54:06.641 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.641 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.641 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:06.641 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:01.005720+0000", 2024-02-05T04:54:06.641 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0059392409999999996 2024-02-05T04:54:06.642 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.642 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.642 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:06.642 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:01.005741+0000", 2024-02-05T04:54:06.642 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0542000000000001e-05 2024-02-05T04:54:06.642 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:06.642 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:06.642 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:06.642 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.642 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.642 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.2s2 69/62 ECSubWrite(tid=75, reqid=client.4634.0:641, at_version=69'55, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:06.642 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:54:01.014043+0000", 2024-02-05T04:54:06.642 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 5.5858904320000002, 2024-02-05T04:54:06.642 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0081968869999999999, 2024-02-05T04:54:06.642 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:06.642 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:06.643 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:06.643 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.643 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:06.643 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:01.014043+0000", 2024-02-05T04:54:06.643 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:06.643 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.643 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.643 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:06.643 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:01.014043+0000", 2024-02-05T04:54:06.643 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:06.643 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.643 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.643 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:06.643 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:01.014043+0000", 2024-02-05T04:54:06.643 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.16e-07 2024-02-05T04:54:06.643 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.644 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.644 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:06.644 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:01.017527+0000", 2024-02-05T04:54:06.644 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0034836089999999999 2024-02-05T04:54:06.644 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.644 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.644 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:06.644 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:01.017541+0000", 2024-02-05T04:54:06.644 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3816e-05 2024-02-05T04:54:06.644 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.644 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.644 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:06.644 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:01.017549+0000", 2024-02-05T04:54:06.644 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8110000000000003e-06 2024-02-05T04:54:06.644 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.645 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.645 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:06.645 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:01.017615+0000", 2024-02-05T04:54:06.645 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6582999999999996e-05 2024-02-05T04:54:06.645 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.645 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.645 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:06.645 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:01.017626+0000", 2024-02-05T04:54:06.645 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1165000000000001e-05 2024-02-05T04:54:06.645 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.645 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.645 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:06.645 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:01.022186+0000", 2024-02-05T04:54:06.645 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0045597789999999999 2024-02-05T04:54:06.645 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.645 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.646 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:06.646 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:01.022240+0000", 2024-02-05T04:54:06.646 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3908e-05 2024-02-05T04:54:06.646 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:06.646 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:06.646 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:06.646 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.646 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.646 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.3s6 74/62 ECSubWrite(tid=105, reqid=osd.4.0:105, at_version=74'47, trim_to=0'0, roll_forward_to=72'45))", 2024-02-05T04:54:06.646 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:54:06.042188+0000", 2024-02-05T04:54:06.646 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.55774532300000002, 2024-02-05T04:54:06.646 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0074863799999999999, 2024-02-05T04:54:06.646 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:06.646 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:06.646 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:06.646 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.647 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:06.647 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.042188+0000", 2024-02-05T04:54:06.647 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:06.647 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.647 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.647 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:06.647 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.042188+0000", 2024-02-05T04:54:06.647 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:06.647 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.647 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.647 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:06.647 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.042189+0000", 2024-02-05T04:54:06.647 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0100000000000005e-07 2024-02-05T04:54:06.647 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.647 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.648 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:06.648 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.042217+0000", 2024-02-05T04:54:06.648 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.7657e-05 2024-02-05T04:54:06.648 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.648 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.648 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:06.648 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.042226+0000", 2024-02-05T04:54:06.648 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5710000000000004e-06 2024-02-05T04:54:06.648 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.648 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.648 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:06.648 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.042234+0000", 2024-02-05T04:54:06.648 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9279999999999993e-06 2024-02-05T04:54:06.648 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.648 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.648 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:06.649 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.042323+0000", 2024-02-05T04:54:06.649 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8428000000000001e-05 2024-02-05T04:54:06.649 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.649 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.649 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:06.649 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.042339+0000", 2024-02-05T04:54:06.649 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5994e-05 2024-02-05T04:54:06.649 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.649 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.649 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:06.649 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.049625+0000", 2024-02-05T04:54:06.649 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0072861799999999997 2024-02-05T04:54:06.649 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.649 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.649 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:06.649 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.049675+0000", 2024-02-05T04:54:06.650 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0021000000000001e-05 2024-02-05T04:54:06.650 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:06.650 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:06.650 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:06.650 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.650 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.650 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.17s1 74/67 ECSubWrite(tid=4, reqid=osd.0.0:4, at_version=74'32, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:06.650 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:54:06.045938+0000", 2024-02-05T04:54:06.650 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.55399560199999998, 2024-02-05T04:54:06.650 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0070776980000000003, 2024-02-05T04:54:06.650 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:06.650 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:06.650 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:06.650 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.650 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:06.651 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.045938+0000", 2024-02-05T04:54:06.651 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:06.651 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.651 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.651 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:06.651 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.045938+0000", 2024-02-05T04:54:06.651 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:06.651 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.651 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.651 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:06.651 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.045938+0000", 2024-02-05T04:54:06.651 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.27e-07 2024-02-05T04:54:06.651 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.651 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.651 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:06.651 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.045963+0000", 2024-02-05T04:54:06.652 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.4834999999999999e-05 2024-02-05T04:54:06.652 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.652 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.652 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:06.652 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.045973+0000", 2024-02-05T04:54:06.652 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0059e-05 2024-02-05T04:54:06.652 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.652 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.652 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:06.652 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.045982+0000", 2024-02-05T04:54:06.652 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.7860000000000002e-06 2024-02-05T04:54:06.652 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.652 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.652 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:06.652 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.046058+0000", 2024-02-05T04:54:06.653 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5958999999999996e-05 2024-02-05T04:54:06.653 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.653 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.653 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:06.653 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.046068+0000", 2024-02-05T04:54:06.653 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0407e-05 2024-02-05T04:54:06.653 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.653 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.653 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:06.653 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.052966+0000", 2024-02-05T04:54:06.653 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.006897668 2024-02-05T04:54:06.653 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.653 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.653 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:06.653 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.053016+0000", 2024-02-05T04:54:06.654 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9657000000000002e-05 2024-02-05T04:54:06.654 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:06.654 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:06.654 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:06.654 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.654 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.654 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWriteReply(3.14s0 74/67 ECSubWriteReply(tid=22, last_complete=74'83, committed=1, applied=1))", 2024-02-05T04:54:06.654 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:54:06.048543+0000", 2024-02-05T04:54:06.654 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.55139032799999999, 2024-02-05T04:54:06.654 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0075604620000000004, 2024-02-05T04:54:06.654 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:06.654 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:06.654 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:06.654 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.654 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:06.654 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.048543+0000", 2024-02-05T04:54:06.655 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:06.655 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.655 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.655 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:06.655 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.048543+0000", 2024-02-05T04:54:06.655 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:06.655 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.655 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.655 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:06.655 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.048544+0000", 2024-02-05T04:54:06.655 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.06e-07 2024-02-05T04:54:06.655 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.655 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.655 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:06.655 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.048559+0000", 2024-02-05T04:54:06.655 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4912999999999999e-05 2024-02-05T04:54:06.656 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.656 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.656 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:06.656 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.048569+0000", 2024-02-05T04:54:06.656 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0186999999999999e-05 2024-02-05T04:54:06.656 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.656 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.656 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:06.656 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.048577+0000", 2024-02-05T04:54:06.656 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7689999999999996e-06 2024-02-05T04:54:06.656 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.656 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.656 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:06.656 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.056078+0000", 2024-02-05T04:54:06.656 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0075011210000000004 2024-02-05T04:54:06.657 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.657 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.657 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:06.657 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.056104+0000", 2024-02-05T04:54:06.657 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5965999999999998e-05 2024-02-05T04:54:06.657 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:06.657 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:06.657 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:06.657 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.657 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.657 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWriteReply(3.14s0 74/67 ECSubWriteReply(tid=22, last_complete=74'83, committed=1, applied=1))", 2024-02-05T04:54:06.657 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:54:06.049955+0000", 2024-02-05T04:54:06.657 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.54997856700000003, 2024-02-05T04:54:06.657 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0061941080000000003, 2024-02-05T04:54:06.657 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:06.658 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:06.658 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:06.658 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.658 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:06.658 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.049955+0000", 2024-02-05T04:54:06.658 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:06.658 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.658 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.658 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:06.658 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.049955+0000", 2024-02-05T04:54:06.658 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:06.658 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.658 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.658 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:06.658 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.049956+0000", 2024-02-05T04:54:06.659 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.4e-07 2024-02-05T04:54:06.659 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.659 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.659 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:06.659 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.049963+0000", 2024-02-05T04:54:06.659 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.199e-06 2024-02-05T04:54:06.659 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.659 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.659 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:06.659 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.049976+0000", 2024-02-05T04:54:06.659 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2806000000000001e-05 2024-02-05T04:54:06.659 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.659 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.659 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:06.659 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.049984+0000", 2024-02-05T04:54:06.659 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8040000000000002e-06 2024-02-05T04:54:06.660 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.660 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.660 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:06.660 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.056132+0000", 2024-02-05T04:54:06.660 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0061485259999999996 2024-02-05T04:54:06.660 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.660 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.660 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:06.660 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.056149+0000", 2024-02-05T04:54:06.660 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6833e-05 2024-02-05T04:54:06.660 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:06.661 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:06.661 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:06.661 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.661 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.661 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWriteReply(3.14s0 74/67 ECSubWriteReply(tid=22, last_complete=74'83, committed=1, applied=1))", 2024-02-05T04:54:06.661 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:54:06.050537+0000", 2024-02-05T04:54:06.661 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.54939631899999997, 2024-02-05T04:54:06.661 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0056738969999999998, 2024-02-05T04:54:06.661 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:06.661 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:06.661 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:06.661 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.661 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:06.661 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.050537+0000", 2024-02-05T04:54:06.661 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:06.662 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.662 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.662 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:06.662 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.050537+0000", 2024-02-05T04:54:06.662 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:06.662 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.662 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.662 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:06.662 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.050538+0000", 2024-02-05T04:54:06.662 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0380000000000001e-06 2024-02-05T04:54:06.662 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.662 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.662 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:06.662 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.050544+0000", 2024-02-05T04:54:06.662 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0739999999999998e-06 2024-02-05T04:54:06.663 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.663 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.663 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:06.663 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.050558+0000", 2024-02-05T04:54:06.663 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3515e-05 2024-02-05T04:54:06.663 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.663 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.663 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:06.663 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.050566+0000", 2024-02-05T04:54:06.663 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.1599999999999998e-06 2024-02-05T04:54:06.663 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.663 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.663 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:06.663 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.056190+0000", 2024-02-05T04:54:06.663 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0056241499999999996 2024-02-05T04:54:06.663 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.664 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.664 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:06.664 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.056211+0000", 2024-02-05T04:54:06.664 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0959999999999999e-05 2024-02-05T04:54:06.664 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:06.664 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:06.664 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:06.664 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.664 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.664 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.2s2 74/62 ECSubWrite(tid=89, reqid=osd.12.0:89, at_version=74'69, trim_to=0'0, roll_forward_to=72'67))", 2024-02-05T04:54:06.664 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:54:06.050676+0000", 2024-02-05T04:54:06.664 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.54925725000000003, 2024-02-05T04:54:06.664 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0079223910000000008, 2024-02-05T04:54:06.664 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:06.664 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:06.665 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:06.665 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.665 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:06.665 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.050676+0000", 2024-02-05T04:54:06.665 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:06.665 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.665 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.665 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:06.665 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.050676+0000", 2024-02-05T04:54:06.665 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:06.665 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.665 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.665 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:06.665 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.050677+0000", 2024-02-05T04:54:06.665 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6899999999999996e-07 2024-02-05T04:54:06.665 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.666 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.666 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:06.666 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.050702+0000", 2024-02-05T04:54:06.666 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.4457e-05 2024-02-05T04:54:06.666 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.666 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.666 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:06.666 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.050717+0000", 2024-02-05T04:54:06.666 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4701e-05 2024-02-05T04:54:06.666 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.666 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.666 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:06.666 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.050726+0000", 2024-02-05T04:54:06.666 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8260000000000005e-06 2024-02-05T04:54:06.666 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.667 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.667 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:06.667 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.050807+0000", 2024-02-05T04:54:06.667 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0669999999999998e-05 2024-02-05T04:54:06.667 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.667 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.667 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:06.667 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.050817+0000", 2024-02-05T04:54:06.667 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0383999999999999e-05 2024-02-05T04:54:06.667 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.667 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.667 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:06.667 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.058551+0000", 2024-02-05T04:54:06.667 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0077333640000000004 2024-02-05T04:54:06.667 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.668 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.668 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:06.668 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.058599+0000", 2024-02-05T04:54:06.668 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8019999999999998e-05 2024-02-05T04:54:06.668 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:06.668 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:06.668 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:06.668 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.668 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.668 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.2s2 74/62 ECSubWrite(tid=90, reqid=osd.12.0:90, at_version=74'71, trim_to=0'0, roll_forward_to=72'67))", 2024-02-05T04:54:06.668 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:54:06.051210+0000", 2024-02-05T04:54:06.668 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.54872322500000004, 2024-02-05T04:54:06.668 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.010150836, 2024-02-05T04:54:06.668 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:06.669 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:06.669 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:06.669 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.669 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:06.669 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.051210+0000", 2024-02-05T04:54:06.669 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:06.669 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.669 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.669 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:06.669 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.051210+0000", 2024-02-05T04:54:06.669 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:06.669 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.669 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.669 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:06.669 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.051211+0000", 2024-02-05T04:54:06.669 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.1299999999999999e-07 2024-02-05T04:54:06.670 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.670 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.670 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:06.670 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.051237+0000", 2024-02-05T04:54:06.670 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5746e-05 2024-02-05T04:54:06.670 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.670 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.670 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:06.670 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.051249+0000", 2024-02-05T04:54:06.670 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2398999999999999e-05 2024-02-05T04:54:06.670 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.670 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.670 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:06.670 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.051260+0000", 2024-02-05T04:54:06.670 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0473e-05 2024-02-05T04:54:06.671 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.671 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.671 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:06.671 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.053539+0000", 2024-02-05T04:54:06.671 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0022794960000000002 2024-02-05T04:54:06.671 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.671 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.671 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:06.671 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.053549+0000", 2024-02-05T04:54:06.671 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.7510000000000003e-06 2024-02-05T04:54:06.671 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.671 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.671 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:06.671 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.061320+0000", 2024-02-05T04:54:06.671 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0077706099999999998 2024-02-05T04:54:06.671 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.672 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.672 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:06.672 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.061361+0000", 2024-02-05T04:54:06.672 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1548000000000003e-05 2024-02-05T04:54:06.672 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:06.672 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:06.672 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:06.672 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.672 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.672 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.5s3 74/62 ECSubWrite(tid=79, reqid=osd.15.0:79, at_version=74'59, trim_to=0'0, roll_forward_to=72'57))", 2024-02-05T04:54:06.672 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:54:06.054480+0000", 2024-02-05T04:54:06.672 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.54545385499999999, 2024-02-05T04:54:06.672 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0068285220000000001, 2024-02-05T04:54:06.672 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:06.672 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:06.672 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:06.673 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.673 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:06.673 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.054480+0000", 2024-02-05T04:54:06.673 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:06.673 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.673 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.673 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:06.673 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.054480+0000", 2024-02-05T04:54:06.673 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:06.673 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.673 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.673 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:06.673 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.054481+0000", 2024-02-05T04:54:06.673 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1310000000000001e-06 2024-02-05T04:54:06.673 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.674 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.674 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:06.674 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.054506+0000", 2024-02-05T04:54:06.674 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5406e-05 2024-02-05T04:54:06.674 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.674 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.674 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:06.674 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.054528+0000", 2024-02-05T04:54:06.674 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1211000000000001e-05 2024-02-05T04:54:06.674 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.674 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.674 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:06.674 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.054547+0000", 2024-02-05T04:54:06.674 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9548e-05 2024-02-05T04:54:06.674 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.674 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.675 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:06.675 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.054649+0000", 2024-02-05T04:54:06.675 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000102268 2024-02-05T04:54:06.675 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.675 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.675 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:06.675 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.054659+0000", 2024-02-05T04:54:06.675 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.008e-05 2024-02-05T04:54:06.675 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.675 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.675 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:06.675 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.061263+0000", 2024-02-05T04:54:06.675 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0066037329999999997 2024-02-05T04:54:06.675 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.675 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.676 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:06.676 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.061308+0000", 2024-02-05T04:54:06.676 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5145000000000002e-05 2024-02-05T04:54:06.676 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:06.676 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:06.676 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:06.676 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.676 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.676 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.17s1 74/67 ECSubWrite(tid=5, reqid=osd.0.0:4, at_version=0'0, trim_to=0'0, roll_forward_to=74'32))", 2024-02-05T04:54:06.676 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:54:06.055499+0000", 2024-02-05T04:54:06.676 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.54443447599999995, 2024-02-05T04:54:06.676 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0058280470000000003, 2024-02-05T04:54:06.676 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:06.676 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:06.676 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:06.676 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.677 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:06.677 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.055499+0000", 2024-02-05T04:54:06.677 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:06.677 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.677 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.677 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:06.677 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.055499+0000", 2024-02-05T04:54:06.677 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:06.677 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.677 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.677 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:06.677 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.055500+0000", 2024-02-05T04:54:06.677 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.0699999999999996e-07 2024-02-05T04:54:06.677 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.677 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.678 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:06.678 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.055512+0000", 2024-02-05T04:54:06.678 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1552999999999999e-05 2024-02-05T04:54:06.678 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.678 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.678 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:06.678 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.055555+0000", 2024-02-05T04:54:06.678 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3591000000000003e-05 2024-02-05T04:54:06.678 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.678 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.678 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:06.678 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.055569+0000", 2024-02-05T04:54:06.678 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.34e-05 2024-02-05T04:54:06.678 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.678 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.679 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:06.679 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.055629+0000", 2024-02-05T04:54:06.679 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0089e-05 2024-02-05T04:54:06.679 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.679 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.679 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:06.679 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.055639+0000", 2024-02-05T04:54:06.679 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0030000000000001e-05 2024-02-05T04:54:06.679 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.679 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.679 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:06.679 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.061284+0000", 2024-02-05T04:54:06.679 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0056451100000000001 2024-02-05T04:54:06.679 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:06.679 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:06.679 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:06.680 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.061327+0000", 2024-02-05T04:54:06.680 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3367e-05 2024-02-05T04:54:06.680 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:06.680 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:06.680 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:06.680 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:06.680 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:06.680 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T04:54:06.681 INFO:tasks.ceph.osd.1.smithi091.stderr:2024-02-05T04:54:06.681+0000 7ff9dc283640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34275) UID: 0 2024-02-05T04:54:06.682 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-02-05T04:54:06.754 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T04:54:06.782 INFO:tasks.ceph.osd.13.smithi091.stderr:2024-02-05T04:54:06.782+0000 7f526fd4d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 34281) UID: 0 2024-02-05T04:54:06.788 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T04:54:06.788 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-02-05T04:54:06.788 INFO:teuthology.orchestra.run.smithi196.stdout: "num_ops": 0 2024-02-05T04:54:06.789 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T04:54:06.795 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-02-05T04:54:06.881 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0: {} 2024-02-05T04:54:06.882 INFO:teuthology.orchestra.run.smithi062.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T04:54:06.883 INFO:tasks.ceph.osd.3.smithi196.stderr:2024-02-05T04:54:06.882+0000 7fc8817c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34605) UID: 0 2024-02-05T04:54:06.900 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1: {} 2024-02-05T04:54:06.900 INFO:teuthology.orchestra.run.smithi062.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T04:54:06.901 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T04:54:06.901 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-02-05T04:54:06.901 INFO:teuthology.orchestra.run.smithi196.stdout: "complaint_time": 30, 2024-02-05T04:54:06.901 INFO:teuthology.orchestra.run.smithi196.stdout: "num_blocked_ops": 0 2024-02-05T04:54:06.902 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T04:54:06.909 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-02-05T04:54:06.918 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2: {} 2024-02-05T04:54:06.918 INFO:teuthology.orchestra.run.smithi062.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T04:54:06.936 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3: {} 2024-02-05T04:54:06.937 INFO:teuthology.orchestra.run.smithi062.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T04:54:06.952 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4: {} 2024-02-05T04:54:06.952 INFO:teuthology.orchestra.run.smithi062.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T04:54:06.970 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5: {} 2024-02-05T04:54:06.970 INFO:teuthology.orchestra.run.smithi062.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T04:54:06.983 INFO:tasks.ceph.osd.4.smithi062.stderr:2024-02-05T04:54:06.982+0000 7f708c308640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34689) UID: 0 2024-02-05T04:54:06.987 INFO:teuthology.orchestra.run.smithi062.stdout:osd.6: {} 2024-02-05T04:54:06.988 INFO:teuthology.orchestra.run.smithi062.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T04:54:07.006 INFO:teuthology.orchestra.run.smithi062.stdout:osd.7: {} 2024-02-05T04:54:07.006 INFO:teuthology.orchestra.run.smithi062.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T04:54:07.022 INFO:teuthology.orchestra.run.smithi062.stdout:osd.8: {} 2024-02-05T04:54:07.022 INFO:teuthology.orchestra.run.smithi062.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-05T04:54:07.037 INFO:teuthology.orchestra.run.smithi062.stderr:reweighted osd.10 to 1 (10000) 2024-02-05T04:54:07.042 INFO:teuthology.orchestra.run.smithi062.stdout:osd.9: {} 2024-02-05T04:54:07.043 INFO:tasks.rados.rados.0.smithi196.stdout:277: done (2 left) 2024-02-05T04:54:07.043 INFO:tasks.rados.rados.0.smithi196.stdout:append oid 7 current snap is 23 2024-02-05T04:54:07.043 INFO:tasks.rados.rados.0.smithi196.stdout:274: finishing write tid 3 to smithi19639362-24 2024-02-05T04:54:07.043 INFO:tasks.rados.rados.0.smithi196.stdout:278: seq_num 126 ranges {114688=573440,688128=622592,1310720=622592,1933312=311296} 2024-02-05T04:54:07.043 INFO:teuthology.orchestra.run.smithi062.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-05T04:54:07.045 INFO:tasks.rados.rados.0.smithi196.stdout:278: writing smithi19639362-7 from 114688 to 688128 tid 1 2024-02-05T04:54:07.050 INFO:tasks.rados.rados.0.smithi196.stdout:278: writing smithi19639362-7 from 688128 to 1310720 tid 2 2024-02-05T04:54:07.054 INFO:tasks.rados.rados.0.smithi196.stdout:278: writing smithi19639362-7 from 1310720 to 1933312 tid 3 2024-02-05T04:54:07.056 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 11 1 2024-02-05T04:54:07.061 INFO:tasks.rados.rados.0.smithi196.stdout:278: writing smithi19639362-7 from 1933312 to 2244608 tid 4 2024-02-05T04:54:07.061 INFO:tasks.rados.rados.0.smithi196.stdout:279: rmattr oid 21 current snap is 23 2024-02-05T04:54:07.061 INFO:tasks.rados.rados.0.smithi196.stdout:274: finishing write tid 4 to smithi19639362-24 2024-02-05T04:54:07.062 INFO:tasks.rados.rados.0.smithi196.stdout:274: finishing write tid 5 to smithi19639362-24 2024-02-05T04:54:07.062 INFO:tasks.rados.rados.0.smithi196.stdout:274: finishing write tid 6 to smithi19639362-24 2024-02-05T04:54:07.062 INFO:tasks.rados.rados.0.smithi196.stdout:274: oid 24 updating version 0 to 94 2024-02-05T04:54:07.063 INFO:tasks.rados.rados.0.smithi196.stdout:274: oid 24 updating version 94 to 96 2024-02-05T04:54:07.063 INFO:tasks.rados.rados.0.smithi196.stdout:274: oid 24 version 96 is already newer than 93 2024-02-05T04:54:07.063 INFO:tasks.rados.rados.0.smithi196.stdout:274: oid 24 version 96 is already newer than 92 2024-02-05T04:54:07.063 INFO:tasks.rados.rados.0.smithi196.stdout:274: oid 24 version 96 is already newer than 95 2024-02-05T04:54:07.063 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 24 v 96 (ObjNum 123 snap 23 seq_num 123) dirty exists 2024-02-05T04:54:07.064 INFO:tasks.rados.rados.0.smithi196.stdout:274: left oid 24 (ObjNum 123 snap 23 seq_num 123) 2024-02-05T04:54:07.064 INFO:tasks.rados.rados.0.smithi196.stdout:275: finishing write tid 1 to smithi19639362-28 2024-02-05T04:54:07.064 INFO:tasks.rados.rados.0.smithi196.stdout:275: finishing write tid 2 to smithi19639362-28 2024-02-05T04:54:07.064 INFO:tasks.rados.rados.0.smithi196.stdout:275: finishing write tid 3 to smithi19639362-28 2024-02-05T04:54:07.064 INFO:tasks.rados.rados.0.smithi196.stdout:275: finishing write tid 4 to smithi19639362-28 2024-02-05T04:54:07.064 INFO:tasks.rados.rados.0.smithi196.stdout:275: finishing write tid 5 to smithi19639362-28 2024-02-05T04:54:07.065 INFO:tasks.rados.rados.0.smithi196.stdout:275: finishing write tid 6 to smithi19639362-28 2024-02-05T04:54:07.065 INFO:tasks.rados.rados.0.smithi196.stdout:275: oid 28 updating version 0 to 37 2024-02-05T04:54:07.065 INFO:tasks.rados.rados.0.smithi196.stdout:275: oid 28 updating version 37 to 38 2024-02-05T04:54:07.065 INFO:tasks.rados.rados.0.smithi196.stdout:275: oid 28 version 38 is already newer than 34 2024-02-05T04:54:07.066 INFO:tasks.rados.rados.0.smithi196.stdout:275: oid 28 version 38 is already newer than 35 2024-02-05T04:54:07.066 INFO:tasks.rados.rados.0.smithi196.stdout:275: oid 28 version 38 is already newer than 36 2024-02-05T04:54:07.066 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 28 v 38 (ObjNum 124 snap 23 seq_num 124) dirty exists 2024-02-05T04:54:07.066 INFO:tasks.rados.rados.0.smithi196.stdout:275: left oid 28 (ObjNum 124 snap 23 seq_num 124) 2024-02-05T04:54:07.066 INFO:tasks.rados.rados.0.smithi196.stdout:274: done (3 left) 2024-02-05T04:54:07.066 INFO:tasks.rados.rados.0.smithi196.stdout:275: done (2 left) 2024-02-05T04:54:07.067 INFO:tasks.rados.rados.0.smithi196.stdout:280: copy_from oid 36 from oid 47 current snap is 23 2024-02-05T04:54:07.067 INFO:tasks.rados.rados.0.smithi196.stdout:281: rollback oid 32 current snap is 23 2024-02-05T04:54:07.067 INFO:tasks.rados.rados.0.smithi196.stdout:rollback oid 32 to 21 2024-02-05T04:54:07.067 INFO:tasks.rados.rados.0.smithi196.stdout:282: rollback oid 28 current snap is 23 2024-02-05T04:54:07.067 INFO:tasks.rados.rados.0.smithi196.stdout:rollback oid 28 to 20 2024-02-05T04:54:07.068 INFO:tasks.rados.rados.0.smithi196.stdout:283: read oid 37 snap -1 2024-02-05T04:54:07.068 INFO:tasks.rados.rados.0.smithi196.stdout:283: expect deleted 2024-02-05T04:54:07.068 INFO:tasks.rados.rados.0.smithi196.stdout:284: rmattr oid 50 current snap is 23 2024-02-05T04:54:07.068 INFO:tasks.rados.rados.0.smithi196.stdout:284: done (6 left) 2024-02-05T04:54:07.068 INFO:tasks.rados.rados.0.smithi196.stdout:285: read oid 44 snap -1 2024-02-05T04:54:07.069 INFO:tasks.rados.rados.0.smithi196.stdout:285: expect (ObjNum 68 snap 2 seq_num 68) 2024-02-05T04:54:07.069 INFO:tasks.rados.rados.0.smithi196.stdout:286: rollback oid 31 current snap is 23 2024-02-05T04:54:07.069 INFO:tasks.rados.rados.0.smithi196.stdout:rollback oid 31 to 16 2024-02-05T04:54:07.069 INFO:tasks.rados.rados.0.smithi196.stdout:287: read oid 34 snap -1 2024-02-05T04:54:07.069 INFO:tasks.rados.rados.0.smithi196.stdout:287: expect (ObjNum 0 snap 0 seq_num 0) 2024-02-05T04:54:07.070 INFO:tasks.rados.rados.0.smithi196.stdout:288: read oid 40 snap -1 2024-02-05T04:54:07.070 INFO:tasks.rados.rados.0.smithi196.stdout:288: expect (ObjNum 73 snap 3 seq_num 73) 2024-02-05T04:54:07.070 INFO:tasks.rados.rados.0.smithi196.stdout:append oid 17 current snap is 23 2024-02-05T04:54:07.070 INFO:tasks.rados.rados.0.smithi196.stdout:289: seq_num 127 ranges {1474560=589824,2064384=753664,2818048=475136,3293184=327680} 2024-02-05T04:54:07.072 INFO:teuthology.orchestra.run.smithi062.stdout:osd.10: {} 2024-02-05T04:54:07.072 INFO:tasks.rados.rados.0.smithi196.stdout:289: writing smithi19639362-17 from 1474560 to 2064384 tid 1 2024-02-05T04:54:07.073 INFO:teuthology.orchestra.run.smithi062.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-05T04:54:07.074 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:07.074 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:07.075 INFO:tasks.rados.rados.0.smithi196.stdout:289: writing smithi19639362-17 from 2064384 to 2818048 tid 2 2024-02-05T04:54:07.077 INFO:tasks.rados.rados.0.smithi196.stdout:289: writing smithi19639362-17 from 2818048 to 3293184 tid 3 2024-02-05T04:54:07.081 INFO:tasks.rados.rados.0.smithi196.stdout:289: writing smithi19639362-17 from 3293184 to 3620864 tid 4 2024-02-05T04:54:07.081 INFO:tasks.rados.rados.0.smithi196.stdout:290: rollback oid 26 current snap is 23 2024-02-05T04:54:07.081 INFO:tasks.rados.rados.0.smithi196.stdout:rollback oid 26 to 22 2024-02-05T04:54:07.081 INFO:tasks.rados.rados.0.smithi196.stdout:278: finishing write tid 1 to smithi19639362-7 2024-02-05T04:54:07.081 INFO:tasks.rados.rados.0.smithi196.stdout:287: expect (ObjNum 0 snap 0 seq_num 0) 2024-02-05T04:54:07.084 INFO:tasks.ceph.osd.9.smithi091.stderr:2024-02-05T04:54:07.084+0000 7efc18827640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 34279) UID: 0 2024-02-05T04:54:07.085 INFO:tasks.rados.rados.0.smithi196.stdout:278: finishing write tid 2 to smithi19639362-7 2024-02-05T04:54:07.085 INFO:tasks.rados.rados.0.smithi196.stdout:278: finishing write tid 3 to smithi19639362-7 2024-02-05T04:54:07.085 INFO:tasks.rados.rados.0.smithi196.stdout:285: expect (ObjNum 68 snap 2 seq_num 68) 2024-02-05T04:54:07.090 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-02-05T04:54:07.095 INFO:teuthology.orchestra.run.smithi062.stdout:osd.11: {} 2024-02-05T04:54:07.095 INFO:teuthology.orchestra.run.smithi062.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-05T04:54:07.112 INFO:teuthology.orchestra.run.smithi062.stdout:osd.12: {} 2024-02-05T04:54:07.113 INFO:teuthology.orchestra.run.smithi062.stderr:osd.12: osd_enable_op_tracker = '' 2024-02-05T04:54:07.130 INFO:teuthology.orchestra.run.smithi062.stdout:osd.13: {} 2024-02-05T04:54:07.131 INFO:teuthology.orchestra.run.smithi062.stderr:osd.13: osd_enable_op_tracker = '' 2024-02-05T04:54:07.148 INFO:teuthology.orchestra.run.smithi062.stdout:osd.14: {} 2024-02-05T04:54:07.148 INFO:teuthology.orchestra.run.smithi062.stderr:osd.14: osd_enable_op_tracker = '' 2024-02-05T04:54:07.149 INFO:tasks.rados.rados.0.smithi196.stdout:281: finishing rollback tid 0 to smithi19639362-32 2024-02-05T04:54:07.150 INFO:tasks.rados.rados.0.smithi196.stdout:280: finishing copy_from racing read to smithi19639362-36 2024-02-05T04:54:07.150 INFO:tasks.rados.rados.0.smithi196.stdout:281: finishing rollback tid 1 to smithi19639362-32 2024-02-05T04:54:07.150 INFO:tasks.rados.rados.0.smithi196.stdout:281: finishing rollback tid 2 to smithi19639362-32 2024-02-05T04:54:07.150 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 32 v 25 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-02-05T04:54:07.150 INFO:tasks.rados.rados.0.smithi196.stdout:278: finishing write tid 4 to smithi19639362-7 2024-02-05T04:54:07.150 INFO:tasks.rados.rados.0.smithi196.stdout:278: finishing write tid 5 to smithi19639362-7 2024-02-05T04:54:07.150 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 21 v 82 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-02-05T04:54:07.150 INFO:tasks.rados.rados.0.smithi196.stdout:286: finishing rollback tid 0 to smithi19639362-31 2024-02-05T04:54:07.150 INFO:tasks.rados.rados.0.smithi196.stdout:286: finishing rollback tid 1 to smithi19639362-31 2024-02-05T04:54:07.150 INFO:tasks.rados.rados.0.smithi196.stdout:286: finishing rollback tid 2 to smithi19639362-31 2024-02-05T04:54:07.150 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 31 v 74 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-02-05T04:54:07.150 INFO:tasks.rados.rados.0.smithi196.stdout:279: done (11 left) 2024-02-05T04:54:07.150 INFO:tasks.rados.rados.0.smithi196.stdout:281: done (10 left) 2024-02-05T04:54:07.150 INFO:tasks.rados.rados.0.smithi196.stdout:283: done (9 left) 2024-02-05T04:54:07.150 INFO:tasks.rados.rados.0.smithi196.stdout:285: done (8 left) 2024-02-05T04:54:07.151 INFO:tasks.rados.rados.0.smithi196.stdout:286: done (7 left) 2024-02-05T04:54:07.151 INFO:tasks.rados.rados.0.smithi196.stdout:287: done (6 left) 2024-02-05T04:54:07.151 INFO:tasks.rados.rados.0.smithi196.stdout:291: snap_create 2024-02-05T04:54:07.151 INFO:tasks.rados.rados.0.smithi196.stdout:282: finishing rollback tid 0 to smithi19639362-28 2024-02-05T04:54:07.151 INFO:tasks.rados.rados.0.smithi196.stdout:282: finishing rollback tid 1 to smithi19639362-28 2024-02-05T04:54:07.151 INFO:tasks.rados.rados.0.smithi196.stdout:282: finishing rollback tid 2 to smithi19639362-28 2024-02-05T04:54:07.151 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 28 v 41 (ObjNum 60 snap 2 seq_num 60) dirty exists 2024-02-05T04:54:07.151 INFO:tasks.rados.rados.0.smithi196.stdout:280: finishing copy_from to smithi19639362-36 2024-02-05T04:54:07.151 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 36 v 12 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-02-05T04:54:07.152 INFO:tasks.rados.rados.0.smithi196.stdout:288: expect (ObjNum 73 snap 3 seq_num 73) 2024-02-05T04:54:07.166 INFO:teuthology.orchestra.run.smithi062.stdout:osd.15: {} 2024-02-05T04:54:07.166 INFO:teuthology.orchestra.run.smithi062.stderr:osd.15: osd_enable_op_tracker = '' 2024-02-05T04:54:07.185 INFO:tasks.ceph.osd.1.smithi091.stderr:2024-02-05T04:54:07.185+0000 7ff9dc283640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34275) UID: 0 2024-02-05T04:54:07.197 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:07.197 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:07.207 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-02-05T04:54:07.249 INFO:tasks.rados.rados.0.smithi196.stdout:278: finishing write tid 6 to smithi19639362-7 2024-02-05T04:54:07.249 INFO:tasks.rados.rados.0.smithi196.stdout:278: oid 7 updating version 0 to 11 2024-02-05T04:54:07.249 INFO:tasks.rados.rados.0.smithi196.stdout:278: oid 7 version 11 is already newer than 9 2024-02-05T04:54:07.249 INFO:tasks.rados.rados.0.smithi196.stdout:278: oid 7 version 11 is already newer than 8 2024-02-05T04:54:07.249 INFO:tasks.rados.rados.0.smithi196.stdout:278: oid 7 version 11 is already newer than 10 2024-02-05T04:54:07.249 INFO:tasks.rados.rados.0.smithi196.stdout:278: oid 7 version 11 is already newer than 7 2024-02-05T04:54:07.249 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 7 v 11 (ObjNum 126 snap 23 seq_num 126) dirty exists 2024-02-05T04:54:07.249 INFO:tasks.rados.rados.0.smithi196.stdout:278: left oid 7 (ObjNum 126 snap 23 seq_num 126) 2024-02-05T04:54:07.249 INFO:tasks.rados.rados.0.smithi196.stdout:289: finishing write tid 1 to smithi19639362-17 2024-02-05T04:54:07.250 INFO:tasks.rados.rados.0.smithi196.stdout:290: finishing rollback tid 0 to smithi19639362-26 2024-02-05T04:54:07.250 INFO:tasks.rados.rados.0.smithi196.stdout:289: finishing write tid 2 to smithi19639362-17 2024-02-05T04:54:07.250 INFO:tasks.rados.rados.0.smithi196.stdout:290: finishing rollback tid 1 to smithi19639362-26 2024-02-05T04:54:07.250 INFO:tasks.rados.rados.0.smithi196.stdout:290: finishing rollback tid 2 to smithi19639362-26 2024-02-05T04:54:07.250 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 26 v 44 (ObjNum 118 snap 21 seq_num 118) dirty exists 2024-02-05T04:54:07.250 INFO:tasks.rados.rados.0.smithi196.stdout:289: finishing write tid 3 to smithi19639362-17 2024-02-05T04:54:07.250 INFO:tasks.rados.rados.0.smithi196.stdout:289: finishing write tid 4 to smithi19639362-17 2024-02-05T04:54:07.250 INFO:tasks.rados.rados.0.smithi196.stdout:289: finishing write tid 5 to smithi19639362-17 2024-02-05T04:54:07.250 INFO:tasks.rados.rados.0.smithi196.stdout:289: finishing write tid 6 to smithi19639362-17 2024-02-05T04:54:07.250 INFO:tasks.rados.rados.0.smithi196.stdout:289: oid 17 updating version 0 to 82 2024-02-05T04:54:07.250 INFO:tasks.rados.rados.0.smithi196.stdout:289: oid 17 version 82 is already newer than 79 2024-02-05T04:54:07.250 INFO:tasks.rados.rados.0.smithi196.stdout:289: oid 17 version 82 is already newer than 81 2024-02-05T04:54:07.250 INFO:tasks.rados.rados.0.smithi196.stdout:289: oid 17 version 82 is already newer than 80 2024-02-05T04:54:07.250 INFO:tasks.rados.rados.0.smithi196.stdout:289: oid 17 version 82 is already newer than 78 2024-02-05T04:54:07.250 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 17 v 82 (ObjNum 127 snap 23 seq_num 127) dirty exists 2024-02-05T04:54:07.251 INFO:tasks.rados.rados.0.smithi196.stdout:289: left oid 17 (ObjNum 127 snap 23 seq_num 127) 2024-02-05T04:54:07.286 INFO:tasks.ceph.osd.10.smithi150.stderr:2024-02-05T04:54:07.285+0000 7f3075a4d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 34018) UID: 0 2024-02-05T04:54:07.311 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:07.312 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:07.320 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-02-05T04:54:07.387 INFO:tasks.ceph.osd.12.smithi062.stderr:2024-02-05T04:54:07.386+0000 7f5e5847e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 34695) UID: 0 2024-02-05T04:54:07.466 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:07.466 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:07.475 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-02-05T04:54:07.488 INFO:tasks.ceph.osd.12.smithi062.stderr:2024-02-05T04:54:07.487+0000 7f5e5847e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 34695) UID: 0 2024-02-05T04:54:07.580 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:07.580 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:07.589 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-02-05T04:54:07.590 INFO:tasks.ceph.osd.4.smithi062.stderr:2024-02-05T04:54:07.588+0000 7f708c308640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34689) UID: 0 2024-02-05T04:54:07.690 INFO:tasks.ceph.osd.6.smithi150.stderr:2024-02-05T04:54:07.689+0000 7f6bea9f0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34015) UID: 0 2024-02-05T04:54:07.694 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:07.694 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:07.703 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-02-05T04:54:07.791 INFO:tasks.ceph.osd.3.smithi196.stderr:2024-02-05T04:54:07.790+0000 7fc8817c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34605) UID: 0 2024-02-05T04:54:07.810 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:07.810 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:07.820 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-02-05T04:54:07.892 INFO:tasks.ceph.osd.0.smithi062.stderr:2024-02-05T04:54:07.891+0000 7fece8e03640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34688) UID: 0 2024-02-05T04:54:07.931 INFO:teuthology.orchestra.run.smithi062.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:07.931 INFO:teuthology.orchestra.run.smithi062.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:07.940 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-02-05T04:54:08.024 INFO:tasks.ceph.osd.0.smithi062.stderr:2024-02-05T04:54:08.023+0000 7fece8e03640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34688) UID: 0 2024-02-05T04:54:08.046 INFO:teuthology.orchestra.run.smithi062.stderr:reweighted osd.11 to 1 (10000) 2024-02-05T04:54:08.057 INFO:tasks.rados.rados.0.smithi196.stdout:278: done (6 left) 2024-02-05T04:54:08.057 INFO:tasks.rados.rados.0.smithi196.stdout:280: done (5 left) 2024-02-05T04:54:08.058 INFO:tasks.rados.rados.0.smithi196.stdout:282: done (4 left) 2024-02-05T04:54:08.058 INFO:tasks.rados.rados.0.smithi196.stdout:288: done (3 left) 2024-02-05T04:54:08.058 INFO:tasks.rados.rados.0.smithi196.stdout:289: done (2 left) 2024-02-05T04:54:08.058 INFO:tasks.rados.rados.0.smithi196.stdout:290: done (1 left) 2024-02-05T04:54:08.058 INFO:tasks.rados.rados.0.smithi196.stdout:291: done (0 left) 2024-02-05T04:54:08.058 INFO:tasks.rados.rados.0.smithi196.stdout:292: read oid 3 snap 5 2024-02-05T04:54:08.058 INFO:tasks.rados.rados.0.smithi196.stdout:292: expect deleted 2024-02-05T04:54:08.058 INFO:tasks.rados.rados.0.smithi196.stdout:293: rmattr oid 18 current snap is 24 2024-02-05T04:54:08.058 INFO:tasks.rados.rados.0.smithi196.stdout:294: copy_from oid 12 from oid 34 current snap is 24 2024-02-05T04:54:08.058 INFO:tasks.rados.rados.0.smithi196.stdout:295: read oid 13 snap -1 2024-02-05T04:54:08.058 INFO:tasks.rados.rados.0.smithi196.stdout:295: expect (ObjNum 60 snap 2 seq_num 60) 2024-02-05T04:54:08.058 INFO:tasks.rados.rados.0.smithi196.stdout:296: rmattr oid 19 current snap is 24 2024-02-05T04:54:08.059 INFO:tasks.rados.rados.0.smithi196.stdout:296: done (4 left) 2024-02-05T04:54:08.059 INFO:tasks.rados.rados.0.smithi196.stdout:297: read oid 49 snap 20 2024-02-05T04:54:08.059 INFO:tasks.rados.rados.0.smithi196.stdout:297: expect (ObjNum 48 snap 0 seq_num 48) 2024-02-05T04:54:08.059 INFO:tasks.rados.rados.0.smithi196.stdout:append oid 43 current snap is 24 2024-02-05T04:54:08.059 INFO:tasks.rados.rados.0.smithi196.stdout:298: seq_num 128 ranges {2064384=606208,2670592=507904,3178496=573440,3751936=475136} 2024-02-05T04:54:08.059 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-02-05T04:54:08.061 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 12 1 2024-02-05T04:54:08.071 INFO:tasks.rados.rados.0.smithi196.stdout:298: writing smithi19639362-43 from 2064384 to 2670592 tid 1 2024-02-05T04:54:08.074 INFO:tasks.rados.rados.0.smithi196.stdout:298: writing smithi19639362-43 from 2670592 to 3178496 tid 2 2024-02-05T04:54:08.077 INFO:tasks.rados.rados.0.smithi196.stdout:298: writing smithi19639362-43 from 3178496 to 3751936 tid 3 2024-02-05T04:54:08.080 INFO:tasks.rados.rados.0.smithi196.stdout:298: writing smithi19639362-43 from 3751936 to 4227072 tid 4 2024-02-05T04:54:08.080 INFO:tasks.rados.rados.0.smithi196.stdout:299: read oid 42 snap -1 2024-02-05T04:54:08.081 INFO:tasks.rados.rados.0.smithi196.stdout:299: expect (ObjNum 116 snap 21 seq_num 116) 2024-02-05T04:54:08.081 INFO:tasks.rados.rados.0.smithi196.stdout:300: snap_create 2024-02-05T04:54:08.081 INFO:tasks.rados.rados.0.smithi196.stdout:297: expect (ObjNum 48 snap 0 seq_num 48) 2024-02-05T04:54:08.093 INFO:tasks.ceph.osd.1.smithi091.stderr:2024-02-05T04:54:08.092+0000 7ff9dc283640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34275) UID: 0 2024-02-05T04:54:08.097 INFO:teuthology.orchestra.run.smithi062.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:08.098 INFO:teuthology.orchestra.run.smithi062.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:08.105 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 18 v 83 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-02-05T04:54:08.105 INFO:tasks.rados.rados.0.smithi196.stdout:294: finishing copy_from racing read to smithi19639362-12 2024-02-05T04:54:08.105 INFO:tasks.rados.rados.0.smithi196.stdout:294: finishing copy_from to smithi19639362-12 2024-02-05T04:54:08.105 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 12 v 50 (ObjNum 0 snap 0 seq_num 304) dirty exists 2024-02-05T04:54:08.105 INFO:tasks.rados.rados.0.smithi196.stdout:295: expect (ObjNum 60 snap 2 seq_num 60) 2024-02-05T04:54:08.107 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-02-05T04:54:08.141 INFO:tasks.rados.rados.0.smithi196.stdout:298: finishing write tid 1 to smithi19639362-43 2024-02-05T04:54:08.141 INFO:tasks.rados.rados.0.smithi196.stdout:298: finishing write tid 2 to smithi19639362-43 2024-02-05T04:54:08.141 INFO:tasks.rados.rados.0.smithi196.stdout:298: finishing write tid 3 to smithi19639362-43 2024-02-05T04:54:08.141 INFO:tasks.rados.rados.0.smithi196.stdout:298: finishing write tid 4 to smithi19639362-43 2024-02-05T04:54:08.141 INFO:tasks.rados.rados.0.smithi196.stdout:298: finishing write tid 5 to smithi19639362-43 2024-02-05T04:54:08.141 INFO:tasks.rados.rados.0.smithi196.stdout:298: finishing write tid 6 to smithi19639362-43 2024-02-05T04:54:08.141 INFO:tasks.rados.rados.0.smithi196.stdout:298: oid 43 updating version 0 to 108 2024-02-05T04:54:08.142 INFO:tasks.rados.rados.0.smithi196.stdout:298: oid 43 updating version 108 to 109 2024-02-05T04:54:08.142 INFO:tasks.rados.rados.0.smithi196.stdout:298: oid 43 updating version 109 to 111 2024-02-05T04:54:08.142 INFO:tasks.rados.rados.0.smithi196.stdout:298: oid 43 updating version 111 to 112 2024-02-05T04:54:08.142 INFO:tasks.rados.rados.0.smithi196.stdout:298: oid 43 version 112 is already newer than 110 2024-02-05T04:54:08.142 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 43 v 112 (ObjNum 128 snap 24 seq_num 128) dirty exists 2024-02-05T04:54:08.142 INFO:tasks.rados.rados.0.smithi196.stdout:298: left oid 43 (ObjNum 128 snap 24 seq_num 128) 2024-02-05T04:54:08.142 INFO:tasks.rados.rados.0.smithi196.stdout:299: expect (ObjNum 116 snap 21 seq_num 116) 2024-02-05T04:54:08.194 INFO:tasks.ceph.osd.14.smithi150.stderr:2024-02-05T04:54:08.194+0000 7f5f81083640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 34020) UID: 0 2024-02-05T04:54:08.258 INFO:teuthology.orchestra.run.smithi062.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:08.258 INFO:teuthology.orchestra.run.smithi062.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:08.267 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-02-05T04:54:08.295 INFO:tasks.ceph.osd.2.smithi150.stderr:2024-02-05T04:54:08.295+0000 7f7337bf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 34014) UID: 0 2024-02-05T04:54:08.376 INFO:teuthology.orchestra.run.smithi062.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:08.376 INFO:teuthology.orchestra.run.smithi062.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:08.385 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-02-05T04:54:08.396 INFO:tasks.ceph.osd.6.smithi150.stderr:2024-02-05T04:54:08.395+0000 7f6bea9f0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34015) UID: 0 2024-02-05T04:54:08.492 INFO:teuthology.orchestra.run.smithi062.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:08.492 INFO:teuthology.orchestra.run.smithi062.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:08.497 INFO:tasks.ceph.osd.4.smithi062.stderr:2024-02-05T04:54:08.496+0000 7f708c308640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34689) UID: 0 2024-02-05T04:54:08.500 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-02-05T04:54:08.599 INFO:tasks.ceph.osd.3.smithi196.stderr:2024-02-05T04:54:08.598+0000 7fc8817c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34605) UID: 0 2024-02-05T04:54:08.648 INFO:teuthology.orchestra.run.smithi062.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:08.648 INFO:teuthology.orchestra.run.smithi062.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:08.656 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-02-05T04:54:08.700 INFO:tasks.ceph.osd.1.smithi091.stderr:2024-02-05T04:54:08.700+0000 7ff9dc283640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34275) UID: 0 2024-02-05T04:54:08.764 INFO:teuthology.orchestra.run.smithi062.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:08.764 INFO:teuthology.orchestra.run.smithi062.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:08.773 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-02-05T04:54:08.801 INFO:tasks.ceph.osd.1.smithi091.stderr:2024-02-05T04:54:08.800+0000 7ff9dc283640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34275) UID: 0 2024-02-05T04:54:08.880 INFO:teuthology.orchestra.run.smithi062.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:08.880 INFO:teuthology.orchestra.run.smithi062.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:08.889 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-02-05T04:54:08.933 INFO:tasks.ceph.osd.4.smithi062.stderr:2024-02-05T04:54:08.932+0000 7f708c308640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34689) UID: 0 2024-02-05T04:54:08.947 INFO:teuthology.orchestra.run.smithi062.stderr:noscrub is set 2024-02-05T04:54:08.949 INFO:teuthology.orchestra.run.smithi062.stderr:reweighted osd.12 to 1 (10000) 2024-02-05T04:54:08.950 INFO:tasks.rados.rados.0.smithi196.stdout:292: done (7 left) 2024-02-05T04:54:08.950 INFO:tasks.rados.rados.0.smithi196.stdout:293: done (6 left) 2024-02-05T04:54:08.950 INFO:tasks.rados.rados.0.smithi196.stdout:294: done (5 left) 2024-02-05T04:54:08.950 INFO:tasks.rados.rados.0.smithi196.stdout:295: done (4 left) 2024-02-05T04:54:08.950 INFO:tasks.rados.rados.0.smithi196.stdout:297: done (3 left) 2024-02-05T04:54:08.950 INFO:tasks.rados.rados.0.smithi196.stdout:298: done (2 left) 2024-02-05T04:54:08.950 INFO:tasks.rados.rados.0.smithi196.stdout:299: done (1 left) 2024-02-05T04:54:08.951 INFO:tasks.rados.rados.0.smithi196.stdout:300: done (0 left) 2024-02-05T04:54:08.951 INFO:tasks.rados.rados.0.smithi196.stdout:301: read oid 44 snap -1 2024-02-05T04:54:08.951 INFO:tasks.rados.rados.0.smithi196.stdout:301: expect (ObjNum 68 snap 2 seq_num 68) 2024-02-05T04:54:08.951 INFO:tasks.rados.rados.0.smithi196.stdout:append oid 43 current snap is 25 2024-02-05T04:54:08.951 INFO:tasks.rados.rados.0.smithi196.stdout:302: seq_num 129 ranges {4227072=622592,4849664=655360,5505024=688128,6193152=212992} 2024-02-05T04:54:08.964 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 13 1 2024-02-05T04:54:08.977 INFO:tasks.rados.rados.0.smithi196.stdout:302: writing smithi19639362-43 from 4227072 to 4849664 tid 1 2024-02-05T04:54:08.980 INFO:tasks.rados.rados.0.smithi196.stdout:302: writing smithi19639362-43 from 4849664 to 5505024 tid 2 2024-02-05T04:54:08.984 INFO:tasks.rados.rados.0.smithi196.stdout:302: writing smithi19639362-43 from 5505024 to 6193152 tid 3 2024-02-05T04:54:08.986 INFO:tasks.rados.rados.0.smithi196.stdout:302: writing smithi19639362-43 from 6193152 to 6406144 tid 4 2024-02-05T04:54:08.987 INFO:tasks.rados.rados.0.smithi196.stdout:303: read oid 34 snap -1 2024-02-05T04:54:08.987 INFO:tasks.rados.rados.0.smithi196.stdout:303: expect (ObjNum 0 snap 0 seq_num 0) 2024-02-05T04:54:08.987 INFO:tasks.rados.rados.0.smithi196.stdout:304: rmattr oid 45 current snap is 25 2024-02-05T04:54:08.987 INFO:tasks.rados.rados.0.smithi196.stdout:304: done (3 left) 2024-02-05T04:54:08.987 INFO:tasks.rados.rados.0.smithi196.stdout:append oid 31 current snap is 25 2024-02-05T04:54:08.987 INFO:tasks.rados.rados.0.smithi196.stdout:305: seq_num 130 ranges {622592=638976,1261568=671744,1933312=802816,2736128=81920} 2024-02-05T04:54:08.993 INFO:tasks.rados.rados.0.smithi196.stdout:305: writing smithi19639362-31 from 622592 to 1261568 tid 1 2024-02-05T04:54:08.998 INFO:tasks.rados.rados.0.smithi196.stdout:305: writing smithi19639362-31 from 1261568 to 1933312 tid 2 2024-02-05T04:54:09.002 INFO:tasks.ceph.osd.14.smithi150.stderr:2024-02-05T04:54:09.001+0000 7f5f81083640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 34020) UID: 0 2024-02-05T04:54:09.002 INFO:tasks.rados.rados.0.smithi196.stdout:305: writing smithi19639362-31 from 1933312 to 2736128 tid 3 2024-02-05T04:54:09.003 INFO:tasks.rados.rados.0.smithi196.stdout:305: writing smithi19639362-31 from 2736128 to 2818048 tid 4 2024-02-05T04:54:09.003 INFO:tasks.rados.rados.0.smithi196.stdout:306: copy_from oid 48 from oid 21 current snap is 25 2024-02-05T04:54:09.003 INFO:tasks.rados.rados.0.smithi196.stdout:301: expect (ObjNum 68 snap 2 seq_num 68) 2024-02-05T04:54:09.004 INFO:teuthology.orchestra.run.smithi062.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:09.004 INFO:teuthology.orchestra.run.smithi062.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:09.015 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-02-05T04:54:09.064 INFO:tasks.rados.rados.0.smithi196.stdout:302: finishing write tid 1 to smithi19639362-43 2024-02-05T04:54:09.064 INFO:tasks.rados.rados.0.smithi196.stdout:303: expect (ObjNum 0 snap 0 seq_num 0) 2024-02-05T04:54:09.067 INFO:tasks.rados.rados.0.smithi196.stdout:302: finishing write tid 2 to smithi19639362-43 2024-02-05T04:54:09.067 INFO:tasks.rados.rados.0.smithi196.stdout:302: finishing write tid 3 to smithi19639362-43 2024-02-05T04:54:09.067 INFO:tasks.rados.rados.0.smithi196.stdout:302: finishing write tid 4 to smithi19639362-43 2024-02-05T04:54:09.067 INFO:tasks.rados.rados.0.smithi196.stdout:302: finishing write tid 5 to smithi19639362-43 2024-02-05T04:54:09.067 INFO:tasks.rados.rados.0.smithi196.stdout:302: finishing write tid 6 to smithi19639362-43 2024-02-05T04:54:09.067 INFO:tasks.rados.rados.0.smithi196.stdout:302: oid 43 updating version 0 to 116 2024-02-05T04:54:09.067 INFO:tasks.rados.rados.0.smithi196.stdout:302: oid 43 updating version 116 to 118 2024-02-05T04:54:09.067 INFO:tasks.rados.rados.0.smithi196.stdout:302: oid 43 version 118 is already newer than 115 2024-02-05T04:54:09.067 INFO:tasks.rados.rados.0.smithi196.stdout:302: oid 43 version 118 is already newer than 114 2024-02-05T04:54:09.068 INFO:tasks.rados.rados.0.smithi196.stdout:302: oid 43 version 118 is already newer than 117 2024-02-05T04:54:09.068 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 43 v 118 (ObjNum 129 snap 25 seq_num 129) dirty exists 2024-02-05T04:54:09.068 INFO:tasks.rados.rados.0.smithi196.stdout:302: left oid 43 (ObjNum 129 snap 25 seq_num 129) 2024-02-05T04:54:09.068 INFO:tasks.rados.rados.0.smithi196.stdout:301: done (4 left) 2024-02-05T04:54:09.068 INFO:tasks.rados.rados.0.smithi196.stdout:302: done (3 left) 2024-02-05T04:54:09.068 INFO:tasks.rados.rados.0.smithi196.stdout:303: done (2 left) 2024-02-05T04:54:09.068 INFO:tasks.rados.rados.0.smithi196.stdout:307: delete oid 40 current snap is 25 2024-02-05T04:54:09.068 INFO:tasks.rados.rados.0.smithi196.stdout:305: finishing write tid 1 to smithi19639362-31 2024-02-05T04:54:09.068 INFO:tasks.rados.rados.0.smithi196.stdout:305: finishing write tid 2 to smithi19639362-31 2024-02-05T04:54:09.068 INFO:tasks.rados.rados.0.smithi196.stdout:305: finishing write tid 3 to smithi19639362-31 2024-02-05T04:54:09.068 INFO:tasks.rados.rados.0.smithi196.stdout:305: finishing write tid 4 to smithi19639362-31 2024-02-05T04:54:09.068 INFO:tasks.rados.rados.0.smithi196.stdout:306: finishing copy_from racing read to smithi19639362-48 2024-02-05T04:54:09.068 INFO:tasks.rados.rados.0.smithi196.stdout:305: finishing write tid 5 to smithi19639362-31 2024-02-05T04:54:09.068 INFO:tasks.rados.rados.0.smithi196.stdout:306: finishing copy_from to smithi19639362-48 2024-02-05T04:54:09.069 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 48 v 82 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-02-05T04:54:09.069 INFO:tasks.rados.rados.0.smithi196.stdout:305: finishing write tid 6 to smithi19639362-31 2024-02-05T04:54:09.069 INFO:tasks.rados.rados.0.smithi196.stdout:305: oid 31 updating version 0 to 84 2024-02-05T04:54:09.069 INFO:tasks.rados.rados.0.smithi196.stdout:305: oid 31 updating version 84 to 87 2024-02-05T04:54:09.069 INFO:tasks.rados.rados.0.smithi196.stdout:305: oid 31 version 87 is already newer than 86 2024-02-05T04:54:09.069 INFO:tasks.rados.rados.0.smithi196.stdout:305: oid 31 updating version 87 to 88 2024-02-05T04:54:09.069 INFO:tasks.rados.rados.0.smithi196.stdout:305: oid 31 version 88 is already newer than 85 2024-02-05T04:54:09.069 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 31 v 88 (ObjNum 130 snap 25 seq_num 130) dirty exists 2024-02-05T04:54:09.069 INFO:tasks.rados.rados.0.smithi196.stdout:305: left oid 31 (ObjNum 130 snap 25 seq_num 130) 2024-02-05T04:54:09.069 INFO:tasks.rados.rados.0.smithi196.stdout:305: done (2 left) 2024-02-05T04:54:09.070 INFO:tasks.rados.rados.0.smithi196.stdout:306: done (1 left) 2024-02-05T04:54:09.070 INFO:tasks.rados.rados.0.smithi196.stdout:307: done (0 left) 2024-02-05T04:54:09.070 INFO:tasks.rados.rados.0.smithi196.stdout:308: snap_remove snap 24 2024-02-05T04:54:09.103 INFO:tasks.ceph.osd.7.smithi196.stderr:2024-02-05T04:54:09.103+0000 7fb281419640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34606) UID: 0 2024-02-05T04:54:09.165 INFO:teuthology.orchestra.run.smithi062.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:09.165 INFO:teuthology.orchestra.run.smithi062.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:09.176 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-02-05T04:54:09.177 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T04:54:09.204 INFO:tasks.ceph.osd.5.smithi091.stderr:2024-02-05T04:54:09.203+0000 7f63983b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34277) UID: 0 2024-02-05T04:54:09.285 INFO:teuthology.orchestra.run.smithi062.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:09.285 INFO:teuthology.orchestra.run.smithi062.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:09.295 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-02-05T04:54:09.304 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0: {} 2024-02-05T04:54:09.304 INFO:teuthology.orchestra.run.smithi062.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T04:54:09.305 INFO:tasks.ceph.osd.13.smithi091.stderr:2024-02-05T04:54:09.304+0000 7f526fd4d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 34281) UID: 0 2024-02-05T04:54:09.324 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1: {} 2024-02-05T04:54:09.324 INFO:teuthology.orchestra.run.smithi062.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T04:54:09.342 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2: {} 2024-02-05T04:54:09.342 INFO:teuthology.orchestra.run.smithi062.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T04:54:09.359 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3: {} 2024-02-05T04:54:09.360 INFO:teuthology.orchestra.run.smithi062.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T04:54:09.378 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4: {} 2024-02-05T04:54:09.378 INFO:teuthology.orchestra.run.smithi062.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T04:54:09.396 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5: {} 2024-02-05T04:54:09.396 INFO:teuthology.orchestra.run.smithi062.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T04:54:09.403 INFO:teuthology.orchestra.run.smithi062.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:09.404 INFO:teuthology.orchestra.run.smithi062.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:09.405 INFO:tasks.ceph.osd.6.smithi150.stderr:2024-02-05T04:54:09.404+0000 7f6bea9f0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34015) UID: 0 2024-02-05T04:54:09.412 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-02-05T04:54:09.414 INFO:teuthology.orchestra.run.smithi062.stdout:osd.6: {} 2024-02-05T04:54:09.414 INFO:teuthology.orchestra.run.smithi062.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T04:54:09.432 INFO:teuthology.orchestra.run.smithi062.stdout:osd.7: {} 2024-02-05T04:54:09.432 INFO:teuthology.orchestra.run.smithi062.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T04:54:09.448 INFO:teuthology.orchestra.run.smithi062.stdout:osd.8: {} 2024-02-05T04:54:09.449 INFO:teuthology.orchestra.run.smithi062.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-05T04:54:09.467 INFO:teuthology.orchestra.run.smithi062.stdout:osd.9: {} 2024-02-05T04:54:09.467 INFO:teuthology.orchestra.run.smithi062.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-05T04:54:09.485 INFO:teuthology.orchestra.run.smithi062.stdout:osd.10: {} 2024-02-05T04:54:09.485 INFO:teuthology.orchestra.run.smithi062.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-05T04:54:09.503 INFO:teuthology.orchestra.run.smithi062.stdout:osd.11: {} 2024-02-05T04:54:09.503 INFO:teuthology.orchestra.run.smithi062.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-05T04:54:09.505 INFO:tasks.ceph.osd.14.smithi150.stderr:2024-02-05T04:54:09.505+0000 7f5f81083640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 34020) UID: 0 2024-02-05T04:54:09.519 INFO:teuthology.orchestra.run.smithi062.stdout:osd.12: {} 2024-02-05T04:54:09.520 INFO:teuthology.orchestra.run.smithi062.stderr:osd.12: osd_enable_op_tracker = '' 2024-02-05T04:54:09.530 INFO:teuthology.orchestra.run.smithi091.stdout:{ 2024-02-05T04:54:09.531 INFO:teuthology.orchestra.run.smithi091.stdout: "ops": [], 2024-02-05T04:54:09.531 INFO:teuthology.orchestra.run.smithi091.stdout: "num_ops": 0 2024-02-05T04:54:09.531 INFO:teuthology.orchestra.run.smithi091.stdout:} 2024-02-05T04:54:09.538 INFO:teuthology.orchestra.run.smithi062.stdout:osd.13: {} 2024-02-05T04:54:09.538 INFO:teuthology.orchestra.run.smithi062.stderr:osd.13: osd_enable_op_tracker = '' 2024-02-05T04:54:09.541 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-02-05T04:54:09.556 INFO:teuthology.orchestra.run.smithi062.stdout:osd.14: {} 2024-02-05T04:54:09.557 INFO:teuthology.orchestra.run.smithi062.stderr:osd.14: osd_enable_op_tracker = '' 2024-02-05T04:54:09.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-02-05T04:54:09.556+0000 7f66861cf640 0 -- 172.21.15.62:0/2676020965 >> [v2:172.21.15.196:6826/3187639683,v1:172.21.15.196:6827/3187639683] conn(0x7f66802224b0 msgr2=0x7f66802248a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-02-05T04:54:09.606 INFO:tasks.ceph.osd.0.smithi062.stderr:2024-02-05T04:54:09.605+0000 7fece8e03640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34688) UID: 0 2024-02-05T04:54:09.647 INFO:teuthology.orchestra.run.smithi091.stdout:{ 2024-02-05T04:54:09.647 INFO:teuthology.orchestra.run.smithi091.stdout: "ops": [], 2024-02-05T04:54:09.647 INFO:teuthology.orchestra.run.smithi091.stdout: "complaint_time": 30, 2024-02-05T04:54:09.647 INFO:teuthology.orchestra.run.smithi091.stdout: "num_blocked_ops": 0 2024-02-05T04:54:09.647 INFO:teuthology.orchestra.run.smithi091.stdout:} 2024-02-05T04:54:09.655 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-02-05T04:54:09.707 INFO:tasks.ceph.osd.5.smithi091.stderr:2024-02-05T04:54:09.706+0000 7f63983b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34277) UID: 0 2024-02-05T04:54:09.763 INFO:teuthology.orchestra.run.smithi091.stdout:{ 2024-02-05T04:54:09.763 INFO:teuthology.orchestra.run.smithi091.stdout: "size": 20, 2024-02-05T04:54:09.763 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 600, 2024-02-05T04:54:09.763 INFO:teuthology.orchestra.run.smithi091.stdout: "ops": [ 2024-02-05T04:54:09.764 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.764 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:285 3.4s0 3:264f0b4b:::smithi19639362-35:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-02-05T04:54:09.764 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:30.205648+0000", 2024-02-05T04:54:09.764 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 39.555586320000003, 2024-02-05T04:54:09.764 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0099691669999999993, 2024-02-05T04:54:09.764 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:09.764 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "started", 2024-02-05T04:54:09.764 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:54:09.764 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:54:09.764 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:09.764 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 285 2024-02-05T04:54:09.764 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.764 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:09.764 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.765 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:09.765 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.205648+0000", 2024-02-05T04:54:09.765 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:09.765 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.765 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.765 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:09.765 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.205648+0000", 2024-02-05T04:54:09.765 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:09.765 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.765 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.765 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:09.765 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.205648+0000", 2024-02-05T04:54:09.765 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.7100000000000002e-07 2024-02-05T04:54:09.765 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.766 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.766 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:09.766 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.205651+0000", 2024-02-05T04:54:09.766 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.277e-06 2024-02-05T04:54:09.766 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.766 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.766 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:09.766 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.205658+0000", 2024-02-05T04:54:09.766 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.1709999999999996e-06 2024-02-05T04:54:09.766 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.766 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.766 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:09.766 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.205663+0000", 2024-02-05T04:54:09.767 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.4009999999999996e-06 2024-02-05T04:54:09.767 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.767 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.767 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:09.767 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.208044+0000", 2024-02-05T04:54:09.767 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.002380957 2024-02-05T04:54:09.767 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.767 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.767 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "waiting for rw locks", 2024-02-05T04:54:09.767 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.208122+0000", 2024-02-05T04:54:09.767 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.7688999999999994e-05 2024-02-05T04:54:09.767 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.767 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.767 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:09.767 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.212367+0000", 2024-02-05T04:54:09.768 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0042449569999999997 2024-02-05T04:54:09.768 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.768 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.768 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:54:09.768 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.212502+0000", 2024-02-05T04:54:09.768 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00013503199999999999 2024-02-05T04:54:09.768 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.768 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.768 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:09.768 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.215617+0000", 2024-02-05T04:54:09.768 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0031151120000000002 2024-02-05T04:54:09.768 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:09.769 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:09.769 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:09.769 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.769 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.769 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:395 3.4s0 3:2c242b83:::smithi19639362-9:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-02-05T04:54:09.769 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:34.187868+0000", 2024-02-05T04:54:09.769 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 35.573366659999998, 2024-02-05T04:54:09.769 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0080281970000000008, 2024-02-05T04:54:09.769 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:09.769 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "started", 2024-02-05T04:54:09.769 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:54:09.770 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:54:09.770 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:09.770 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 395 2024-02-05T04:54:09.770 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.770 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:09.770 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.770 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:09.770 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.187868+0000", 2024-02-05T04:54:09.770 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:09.770 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.770 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.770 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:09.770 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.187868+0000", 2024-02-05T04:54:09.770 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:09.770 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.771 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.771 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:09.771 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.187868+0000", 2024-02-05T04:54:09.771 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.6499999999999999e-07 2024-02-05T04:54:09.771 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.771 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.771 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:09.771 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.187872+0000", 2024-02-05T04:54:09.771 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.9659999999999998e-06 2024-02-05T04:54:09.771 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.771 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.771 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:09.771 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.187881+0000", 2024-02-05T04:54:09.772 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.8519999999999993e-06 2024-02-05T04:54:09.772 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.772 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.772 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:09.772 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.187887+0000", 2024-02-05T04:54:09.772 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.3579999999999999e-06 2024-02-05T04:54:09.772 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.772 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.772 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:09.772 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.191261+0000", 2024-02-05T04:54:09.772 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.003373095 2024-02-05T04:54:09.772 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.772 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.772 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "waiting for rw locks", 2024-02-05T04:54:09.772 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.191336+0000", 2024-02-05T04:54:09.773 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.5731999999999999e-05 2024-02-05T04:54:09.773 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.773 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.773 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:09.773 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.194503+0000", 2024-02-05T04:54:09.773 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0031670069999999999 2024-02-05T04:54:09.773 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.773 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.773 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:54:09.773 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.194590+0000", 2024-02-05T04:54:09.773 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.6650000000000006e-05 2024-02-05T04:54:09.773 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.773 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.773 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:09.773 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.195896+0000", 2024-02-05T04:54:09.774 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.001305872 2024-02-05T04:54:09.774 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:09.774 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:09.774 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:09.774 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.774 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.774 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.6s4 43/31 ECSubWrite(tid=138, reqid=osd.13.0:138, at_version=43'24, trim_to=0'0, roll_forward_to=40'22))", 2024-02-05T04:54:09.774 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:35.172360+0000", 2024-02-05T04:54:09.774 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 34.588874455000003, 2024-02-05T04:54:09.774 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00794804, 2024-02-05T04:54:09.774 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:09.774 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:54:09.774 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:09.774 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.775 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:09.775 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.172360+0000", 2024-02-05T04:54:09.775 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:09.775 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.775 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.775 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:09.775 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.172360+0000", 2024-02-05T04:54:09.775 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:09.775 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.775 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.775 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:09.775 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.172360+0000", 2024-02-05T04:54:09.775 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.9400000000000005e-07 2024-02-05T04:54:09.775 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.775 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.776 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:09.776 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.172384+0000", 2024-02-05T04:54:09.776 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.3186000000000001e-05 2024-02-05T04:54:09.776 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.776 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.776 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:09.776 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.172425+0000", 2024-02-05T04:54:09.776 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.1069000000000003e-05 2024-02-05T04:54:09.776 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.776 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.776 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:09.776 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.172435+0000", 2024-02-05T04:54:09.776 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.9150000000000008e-06 2024-02-05T04:54:09.776 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.776 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.777 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:09.777 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.172511+0000", 2024-02-05T04:54:09.777 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.6118000000000005e-05 2024-02-05T04:54:09.777 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.777 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.777 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:09.777 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.172521+0000", 2024-02-05T04:54:09.777 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.0429e-05 2024-02-05T04:54:09.777 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.777 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.777 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:09.777 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.180273+0000", 2024-02-05T04:54:09.777 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0077523109999999996 2024-02-05T04:54:09.777 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.777 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.778 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:09.778 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.180308+0000", 2024-02-05T04:54:09.778 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.4418000000000001e-05 2024-02-05T04:54:09.778 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:09.778 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:09.778 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:09.778 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.778 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.779 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.8s1 43/31 ECSubWrite(tid=62, reqid=osd.4.0:62, at_version=43'19, trim_to=0'0, roll_forward_to=42'17))", 2024-02-05T04:54:09.779 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:35.172380+0000", 2024-02-05T04:54:09.779 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 34.588854675, 2024-02-05T04:54:09.779 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0081089249999999995, 2024-02-05T04:54:09.779 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:09.779 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:54:09.779 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:09.779 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.779 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:09.779 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.172380+0000", 2024-02-05T04:54:09.779 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:09.779 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.780 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.780 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:09.780 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.172380+0000", 2024-02-05T04:54:09.780 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:09.780 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.780 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.781 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:09.781 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.172380+0000", 2024-02-05T04:54:09.781 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.48e-07 2024-02-05T04:54:09.781 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.781 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.781 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:09.781 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.172399+0000", 2024-02-05T04:54:09.781 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.9377000000000001e-05 2024-02-05T04:54:09.781 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.781 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.781 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:09.781 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.172407+0000", 2024-02-05T04:54:09.781 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.7729999999999999e-06 2024-02-05T04:54:09.781 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.781 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.782 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:09.782 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.172415+0000", 2024-02-05T04:54:09.782 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.8010000000000004e-06 2024-02-05T04:54:09.782 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.782 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.782 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:09.782 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.172494+0000", 2024-02-05T04:54:09.782 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.9208999999999993e-05 2024-02-05T04:54:09.782 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.782 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.782 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:09.782 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.172503+0000", 2024-02-05T04:54:09.782 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.1439999999999992e-06 2024-02-05T04:54:09.782 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.782 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.783 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:09.783 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.180452+0000", 2024-02-05T04:54:09.783 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0079488509999999998 2024-02-05T04:54:09.783 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.783 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.783 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:09.783 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.180488+0000", 2024-02-05T04:54:09.783 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.6622000000000003e-05 2024-02-05T04:54:09.783 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:09.783 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:09.783 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:09.784 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.784 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.784 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:500 3.4s0 3:2c15cb2e:::smithi19639362-6:head [append 0~524288] snapc d=[d,b,9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-02-05T04:54:09.784 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:44.963298+0000", 2024-02-05T04:54:09.784 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 24.797936038, 2024-02-05T04:54:09.784 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.010488153, 2024-02-05T04:54:09.784 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:09.784 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:09.784 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:54:09.784 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:54:09.785 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:09.785 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 500 2024-02-05T04:54:09.785 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.785 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:09.785 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.785 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:09.785 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.963298+0000", 2024-02-05T04:54:09.785 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:09.785 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.785 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.785 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:09.785 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.963298+0000", 2024-02-05T04:54:09.785 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:09.785 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.786 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.786 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:09.786 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.963301+0000", 2024-02-05T04:54:09.786 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.6769999999999999e-06 2024-02-05T04:54:09.786 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.786 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.786 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:09.786 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.965096+0000", 2024-02-05T04:54:09.786 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.001794618 2024-02-05T04:54:09.786 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.786 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.786 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:09.786 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.965120+0000", 2024-02-05T04:54:09.786 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.4161000000000001e-05 2024-02-05T04:54:09.786 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.787 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.787 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:09.787 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.965132+0000", 2024-02-05T04:54:09.787 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.2658e-05 2024-02-05T04:54:09.787 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.787 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.787 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:09.787 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.965206+0000", 2024-02-05T04:54:09.787 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.3632000000000002e-05 2024-02-05T04:54:09.787 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.787 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.787 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "op must wait for map", 2024-02-05T04:54:09.787 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.965213+0000", 2024-02-05T04:54:09.787 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.8399999999999997e-06 2024-02-05T04:54:09.787 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.788 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.788 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:09.788 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.967118+0000", 2024-02-05T04:54:09.788 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.001905688 2024-02-05T04:54:09.788 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.788 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.788 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:54:09.788 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.967190+0000", 2024-02-05T04:54:09.788 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.161e-05 2024-02-05T04:54:09.788 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.788 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.788 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:09.788 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.968120+0000", 2024-02-05T04:54:09.788 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00092962600000000002 2024-02-05T04:54:09.788 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.789 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.789 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:09.789 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.973623+0000", 2024-02-05T04:54:09.789 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0055030770000000003 2024-02-05T04:54:09.789 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.789 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.789 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "commit_sent", 2024-02-05T04:54:09.789 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.973717+0000", 2024-02-05T04:54:09.789 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.3845999999999996e-05 2024-02-05T04:54:09.789 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.789 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.789 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:09.789 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.973786+0000", 2024-02-05T04:54:09.789 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.9720000000000003e-05 2024-02-05T04:54:09.789 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:09.789 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:09.790 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:09.790 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.790 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.790 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:501 3.4s0 3:2c15cb2e:::smithi19639362-6:head [append 0~753664] snapc d=[d,b,9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-02-05T04:54:09.790 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:44.967519+0000", 2024-02-05T04:54:09.790 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 24.793715414000001, 2024-02-05T04:54:09.790 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.010834552000000001, 2024-02-05T04:54:09.790 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:09.790 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:09.790 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:54:09.790 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:54:09.790 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:09.790 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 501 2024-02-05T04:54:09.790 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.791 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:09.791 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.791 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:09.791 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.967519+0000", 2024-02-05T04:54:09.791 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:09.791 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.791 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.791 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:09.791 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.967519+0000", 2024-02-05T04:54:09.791 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:09.791 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.791 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.791 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:09.791 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.967520+0000", 2024-02-05T04:54:09.791 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.142e-06 2024-02-05T04:54:09.791 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.792 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.792 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:09.792 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.969864+0000", 2024-02-05T04:54:09.792 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0023435349999999999 2024-02-05T04:54:09.792 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.792 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.792 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:09.792 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.969881+0000", 2024-02-05T04:54:09.792 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.7592999999999999e-05 2024-02-05T04:54:09.792 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.792 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.792 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:09.792 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.969890+0000", 2024-02-05T04:54:09.792 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.2609999999999999e-06 2024-02-05T04:54:09.793 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.793 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.793 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:09.793 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.969975+0000", 2024-02-05T04:54:09.793 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.4382999999999995e-05 2024-02-05T04:54:09.793 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.793 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.793 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:54:09.793 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.970100+0000", 2024-02-05T04:54:09.793 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00012555199999999999 2024-02-05T04:54:09.793 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.793 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.793 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:09.793 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.971116+0000", 2024-02-05T04:54:09.793 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.001015665 2024-02-05T04:54:09.794 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.794 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.794 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:09.794 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.977653+0000", 2024-02-05T04:54:09.794 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0065369429999999999 2024-02-05T04:54:09.794 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.794 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.794 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "commit_sent", 2024-02-05T04:54:09.794 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.978281+0000", 2024-02-05T04:54:09.794 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00062775600000000004 2024-02-05T04:54:09.794 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.795 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.795 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:09.795 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.978353+0000", 2024-02-05T04:54:09.795 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.2721999999999999e-05 2024-02-05T04:54:09.795 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:09.795 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:09.795 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:09.795 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.795 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.795 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:502 3.4s0 3:2c15cb2e:::smithi19639362-6:head [append 0~409600] snapc d=[d,b,9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-02-05T04:54:09.795 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:44.970001+0000", 2024-02-05T04:54:09.795 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 24.791232794999999, 2024-02-05T04:54:09.795 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0088644669999999991, 2024-02-05T04:54:09.795 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:09.796 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:09.796 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:54:09.796 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:54:09.796 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:09.796 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 502 2024-02-05T04:54:09.796 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.796 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:09.796 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.796 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:09.796 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.970001+0000", 2024-02-05T04:54:09.796 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:09.796 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.796 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.796 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:09.796 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.970001+0000", 2024-02-05T04:54:09.797 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:09.797 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.797 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.797 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:09.797 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.970003+0000", 2024-02-05T04:54:09.797 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.7120000000000001e-06 2024-02-05T04:54:09.797 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.797 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.797 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:09.797 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.971437+0000", 2024-02-05T04:54:09.797 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.001433911 2024-02-05T04:54:09.797 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.797 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.797 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:09.798 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.971460+0000", 2024-02-05T04:54:09.798 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.3394e-05 2024-02-05T04:54:09.798 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.798 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.798 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:09.798 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.971482+0000", 2024-02-05T04:54:09.798 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.1137000000000001e-05 2024-02-05T04:54:09.798 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.798 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.799 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:09.799 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.972029+0000", 2024-02-05T04:54:09.799 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00054736499999999996 2024-02-05T04:54:09.799 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.799 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.799 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:54:09.799 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.972141+0000", 2024-02-05T04:54:09.799 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.000111742 2024-02-05T04:54:09.799 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.799 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.799 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:09.799 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.972917+0000", 2024-02-05T04:54:09.799 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00077584400000000003 2024-02-05T04:54:09.799 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.800 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.800 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:09.800 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.978062+0000", 2024-02-05T04:54:09.800 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0051453000000000002 2024-02-05T04:54:09.800 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.800 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.800 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "commit_sent", 2024-02-05T04:54:09.800 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.978801+0000", 2024-02-05T04:54:09.800 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00073907499999999995 2024-02-05T04:54:09.800 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.800 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.800 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:09.800 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.978866+0000", 2024-02-05T04:54:09.800 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.4987000000000006e-05 2024-02-05T04:54:09.801 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:09.801 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:09.801 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:09.801 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.801 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.801 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:504 3.4s0 3:2c15cb2e:::smithi19639362-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-02-05T04:54:09.801 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:44.971539+0000", 2024-02-05T04:54:09.801 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 24.789695405, 2024-02-05T04:54:09.801 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0087845130000000007, 2024-02-05T04:54:09.801 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:09.801 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "started", 2024-02-05T04:54:09.802 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:54:09.802 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:54:09.802 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:09.802 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 504 2024-02-05T04:54:09.802 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.802 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:09.802 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.802 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:09.802 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.971539+0000", 2024-02-05T04:54:09.802 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:09.802 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.803 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.803 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:09.803 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.971539+0000", 2024-02-05T04:54:09.803 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:09.803 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.803 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.803 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:09.803 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.971539+0000", 2024-02-05T04:54:09.803 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.0500000000000003e-07 2024-02-05T04:54:09.803 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.803 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.803 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:09.803 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.971542+0000", 2024-02-05T04:54:09.803 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.6970000000000002e-06 2024-02-05T04:54:09.804 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.804 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.804 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:09.804 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.971549+0000", 2024-02-05T04:54:09.804 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.9890000000000003e-06 2024-02-05T04:54:09.804 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.804 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.804 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:09.804 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.971555+0000", 2024-02-05T04:54:09.804 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.6690000000000003e-06 2024-02-05T04:54:09.804 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.804 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.804 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:09.804 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.974686+0000", 2024-02-05T04:54:09.804 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0031309939999999998 2024-02-05T04:54:09.805 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.805 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.805 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "waiting for rw locks", 2024-02-05T04:54:09.805 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.974749+0000", 2024-02-05T04:54:09.805 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.3167e-05 2024-02-05T04:54:09.805 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.805 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.805 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:09.805 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.979371+0000", 2024-02-05T04:54:09.805 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0046216349999999998 2024-02-05T04:54:09.805 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.805 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.806 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:54:09.806 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.979458+0000", 2024-02-05T04:54:09.806 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.7661e-05 2024-02-05T04:54:09.806 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.806 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.806 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:09.806 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.980323+0000", 2024-02-05T04:54:09.806 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00086509599999999999 2024-02-05T04:54:09.806 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:09.806 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:09.806 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:09.806 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.806 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.807 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.19s1 74/67 ECSubWrite(tid=9, reqid=osd.9.0:9, at_version=74'29, trim_to=0'0, roll_forward_to=72'25))", 2024-02-05T04:54:09.807 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:54:06.044912+0000", 2024-02-05T04:54:09.807 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 3.7163219320000001, 2024-02-05T04:54:09.807 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0086155429999999998, 2024-02-05T04:54:09.807 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:09.807 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:54:09.807 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:09.807 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.807 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:09.807 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.044912+0000", 2024-02-05T04:54:09.807 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:09.807 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.807 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.807 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:09.808 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.044912+0000", 2024-02-05T04:54:09.808 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:09.808 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.808 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.808 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:09.808 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.044913+0000", 2024-02-05T04:54:09.808 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.47e-07 2024-02-05T04:54:09.808 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.808 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.808 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:09.808 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.044936+0000", 2024-02-05T04:54:09.808 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.2282000000000001e-05 2024-02-05T04:54:09.808 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.808 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.808 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:09.809 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.044946+0000", 2024-02-05T04:54:09.809 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.0438000000000001e-05 2024-02-05T04:54:09.809 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.809 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.809 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:09.809 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.044954+0000", 2024-02-05T04:54:09.809 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.1680000000000004e-06 2024-02-05T04:54:09.809 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.809 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.809 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:09.809 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.045511+0000", 2024-02-05T04:54:09.809 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00055711599999999997 2024-02-05T04:54:09.809 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.809 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.809 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:09.810 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.045522+0000", 2024-02-05T04:54:09.810 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.0601999999999999e-05 2024-02-05T04:54:09.810 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.810 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.810 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:09.810 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.053477+0000", 2024-02-05T04:54:09.810 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0079552640000000001 2024-02-05T04:54:09.810 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.810 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.810 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:09.810 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.053528+0000", 2024-02-05T04:54:09.810 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.0726000000000002e-05 2024-02-05T04:54:09.810 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:09.810 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:09.810 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:09.811 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.811 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.811 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.bs1 74/57 ECSubWrite(tid=339, reqid=osd.13.0:339, at_version=74'90, trim_to=0'0, roll_forward_to=72'86))", 2024-02-05T04:54:09.811 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:54:06.045806+0000", 2024-02-05T04:54:09.811 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 3.7154282040000002, 2024-02-05T04:54:09.811 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0077896149999999997, 2024-02-05T04:54:09.811 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:09.811 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:54:09.811 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:09.811 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.811 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:09.811 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.045806+0000", 2024-02-05T04:54:09.811 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:09.811 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.811 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.812 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:09.812 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.045806+0000", 2024-02-05T04:54:09.812 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:09.812 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.812 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.812 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:09.812 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.045807+0000", 2024-02-05T04:54:09.812 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.1900000000000002e-07 2024-02-05T04:54:09.812 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.812 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.812 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:09.812 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.045828+0000", 2024-02-05T04:54:09.812 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.1517000000000001e-05 2024-02-05T04:54:09.812 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.812 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.813 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:09.813 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.045840+0000", 2024-02-05T04:54:09.813 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.2e-05 2024-02-05T04:54:09.813 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.813 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.813 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:09.813 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.045847+0000", 2024-02-05T04:54:09.813 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.0720000000000002e-06 2024-02-05T04:54:09.813 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.813 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.813 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:09.813 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.045909+0000", 2024-02-05T04:54:09.813 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.1329999999999997e-05 2024-02-05T04:54:09.813 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.813 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.814 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:09.814 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.045933+0000", 2024-02-05T04:54:09.814 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.4057999999999999e-05 2024-02-05T04:54:09.814 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.814 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.814 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:09.814 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.053551+0000", 2024-02-05T04:54:09.814 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0076183789999999998 2024-02-05T04:54:09.814 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.814 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.814 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:09.814 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.053596+0000", 2024-02-05T04:54:09.814 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.4539999999999997e-05 2024-02-05T04:54:09.814 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:09.814 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:09.815 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:09.815 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.815 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.815 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.13s3 74/67 ECSubWrite(tid=153, reqid=osd.11.0:153, at_version=74'48, trim_to=0'0, roll_forward_to=72'46))", 2024-02-05T04:54:09.815 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:54:06.049780+0000", 2024-02-05T04:54:09.815 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 3.7114545269999999, 2024-02-05T04:54:09.815 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0098180690000000004, 2024-02-05T04:54:09.815 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:09.815 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:54:09.815 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:09.815 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.816 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:09.816 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.049780+0000", 2024-02-05T04:54:09.816 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:09.816 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.816 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.816 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:09.816 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.049780+0000", 2024-02-05T04:54:09.816 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:09.816 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.816 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.816 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:09.816 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.049780+0000", 2024-02-05T04:54:09.816 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.3e-07 2024-02-05T04:54:09.816 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.816 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.817 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:09.817 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.049803+0000", 2024-02-05T04:54:09.817 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.2160000000000001e-05 2024-02-05T04:54:09.817 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.817 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.817 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:09.817 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.049814+0000", 2024-02-05T04:54:09.817 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.1778e-05 2024-02-05T04:54:09.817 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.817 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.817 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:09.817 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.049823+0000", 2024-02-05T04:54:09.817 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.4929999999999998e-06 2024-02-05T04:54:09.817 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.817 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.817 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:09.818 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.050001+0000", 2024-02-05T04:54:09.818 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00017782299999999999 2024-02-05T04:54:09.818 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.818 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.818 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:09.818 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.050011+0000", 2024-02-05T04:54:09.818 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.046e-05 2024-02-05T04:54:09.818 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.818 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.818 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:09.818 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.059563+0000", 2024-02-05T04:54:09.818 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0095519530000000002 2024-02-05T04:54:09.818 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.818 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.818 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:09.819 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.059598+0000", 2024-02-05T04:54:09.819 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.4672000000000003e-05 2024-02-05T04:54:09.819 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:09.819 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:09.819 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:09.819 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.819 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.819 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.3s2 74/62 ECSubWrite(tid=106, reqid=osd.4.0:105, at_version=0'0, trim_to=0'0, roll_forward_to=74'47))", 2024-02-05T04:54:09.819 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:54:06.050407+0000", 2024-02-05T04:54:09.819 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 3.7108269370000002, 2024-02-05T04:54:09.819 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0091517449999999993, 2024-02-05T04:54:09.819 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:09.819 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:54:09.819 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:09.819 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.820 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:09.820 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.050407+0000", 2024-02-05T04:54:09.820 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:09.820 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.820 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.820 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:09.820 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.050407+0000", 2024-02-05T04:54:09.820 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:09.820 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.820 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.820 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:09.820 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.050408+0000", 2024-02-05T04:54:09.820 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.8400000000000003e-07 2024-02-05T04:54:09.820 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.820 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.821 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:09.821 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.050420+0000", 2024-02-05T04:54:09.821 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.1819e-05 2024-02-05T04:54:09.821 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.821 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.821 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:09.821 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.050433+0000", 2024-02-05T04:54:09.821 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.269e-05 2024-02-05T04:54:09.821 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.821 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.821 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:09.821 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.050441+0000", 2024-02-05T04:54:09.821 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.5150000000000006e-06 2024-02-05T04:54:09.821 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.821 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.822 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:09.822 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.050517+0000", 2024-02-05T04:54:09.822 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.6243999999999997e-05 2024-02-05T04:54:09.822 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.822 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.822 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:09.822 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.050527+0000", 2024-02-05T04:54:09.822 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.3810000000000005e-06 2024-02-05T04:54:09.822 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.822 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.822 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:09.822 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.059527+0000", 2024-02-05T04:54:09.822 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0089998509999999997 2024-02-05T04:54:09.822 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.822 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.823 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:09.823 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.059559+0000", 2024-02-05T04:54:09.823 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.2360999999999997e-05 2024-02-05T04:54:09.823 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:09.823 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:09.823 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:09.823 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.823 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.823 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.8s1 74/62 ECSubWrite(tid=107, reqid=osd.4.0:104, at_version=0'0, trim_to=0'0, roll_forward_to=74'37))", 2024-02-05T04:54:09.823 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:54:06.050958+0000", 2024-02-05T04:54:09.823 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 3.710276216, 2024-02-05T04:54:09.823 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0085881529999999994, 2024-02-05T04:54:09.823 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:09.823 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:54:09.824 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:09.824 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.824 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:09.824 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.050958+0000", 2024-02-05T04:54:09.824 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:09.824 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.824 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.824 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:09.824 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.050958+0000", 2024-02-05T04:54:09.824 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:09.824 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.824 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.824 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:09.824 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.050959+0000", 2024-02-05T04:54:09.824 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.2399999999999997e-07 2024-02-05T04:54:09.825 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.825 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.825 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:09.825 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.050971+0000", 2024-02-05T04:54:09.825 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.1916000000000001e-05 2024-02-05T04:54:09.825 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.825 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.825 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:09.825 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.050997+0000", 2024-02-05T04:54:09.825 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.5860999999999999e-05 2024-02-05T04:54:09.826 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.826 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.826 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:09.826 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.051005+0000", 2024-02-05T04:54:09.826 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.6209999999999993e-06 2024-02-05T04:54:09.826 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.826 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.826 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:09.826 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.051090+0000", 2024-02-05T04:54:09.826 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.4838000000000003e-05 2024-02-05T04:54:09.826 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.826 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.826 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:09.826 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.051101+0000", 2024-02-05T04:54:09.827 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.0607e-05 2024-02-05T04:54:09.827 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.827 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.827 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:09.827 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.059503+0000", 2024-02-05T04:54:09.827 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0084025599999999999 2024-02-05T04:54:09.827 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.827 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.827 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:09.827 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.059546+0000", 2024-02-05T04:54:09.827 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.2926000000000002e-05 2024-02-05T04:54:09.827 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:09.827 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:09.827 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:09.827 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.828 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.828 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.5s5 74/62 ECSubWrite(tid=79, reqid=osd.15.0:79, at_version=74'59, trim_to=0'0, roll_forward_to=72'57))", 2024-02-05T04:54:09.828 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:54:06.054419+0000", 2024-02-05T04:54:09.828 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 3.7068150920000003, 2024-02-05T04:54:09.828 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.011377302000000001, 2024-02-05T04:54:09.828 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:09.828 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:54:09.828 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:09.828 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.828 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:09.828 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.054419+0000", 2024-02-05T04:54:09.828 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:09.828 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.828 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.829 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:09.829 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.054419+0000", 2024-02-05T04:54:09.829 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:09.829 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.829 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.829 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:09.829 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.054419+0000", 2024-02-05T04:54:09.829 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.27e-07 2024-02-05T04:54:09.829 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.829 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.829 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:09.829 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.054444+0000", 2024-02-05T04:54:09.829 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.4992999999999999e-05 2024-02-05T04:54:09.829 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.829 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.830 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:09.830 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.054461+0000", 2024-02-05T04:54:09.830 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.6169999999999999e-05 2024-02-05T04:54:09.830 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.830 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.830 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:09.830 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.054480+0000", 2024-02-05T04:54:09.830 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.8967999999999999e-05 2024-02-05T04:54:09.830 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.830 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.830 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:09.830 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.054571+0000", 2024-02-05T04:54:09.830 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.1335999999999994e-05 2024-02-05T04:54:09.830 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.830 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.830 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:09.831 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.054581+0000", 2024-02-05T04:54:09.831 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.0343000000000001e-05 2024-02-05T04:54:09.831 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.831 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.831 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:09.831 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.065755+0000", 2024-02-05T04:54:09.831 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.011174132 2024-02-05T04:54:09.831 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.831 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.831 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:09.831 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.065796+0000", 2024-02-05T04:54:09.831 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.1032999999999998e-05 2024-02-05T04:54:09.831 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:09.831 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:09.832 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:09.832 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.832 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.832 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.6s4 74/62 ECSubWrite(tid=341, reqid=osd.13.0:337, at_version=0'0, trim_to=0'0, roll_forward_to=74'38))", 2024-02-05T04:54:09.832 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:54:06.054605+0000", 2024-02-05T04:54:09.832 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 3.7066287569999998, 2024-02-05T04:54:09.832 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0084369000000000006, 2024-02-05T04:54:09.832 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:09.832 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:54:09.832 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:09.832 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.832 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:09.832 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.054605+0000", 2024-02-05T04:54:09.832 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:09.832 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.833 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.833 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:09.833 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.054605+0000", 2024-02-05T04:54:09.833 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:09.833 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.833 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.833 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:09.833 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.054606+0000", 2024-02-05T04:54:09.833 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.5300000000000003e-07 2024-02-05T04:54:09.833 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.833 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.833 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:09.833 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.054619+0000", 2024-02-05T04:54:09.833 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.2401e-05 2024-02-05T04:54:09.833 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.834 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.834 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:09.834 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.054631+0000", 2024-02-05T04:54:09.834 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.2018000000000001e-05 2024-02-05T04:54:09.834 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.834 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.834 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:09.834 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.054640+0000", 2024-02-05T04:54:09.834 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.8969999999999997e-06 2024-02-05T04:54:09.834 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.834 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.834 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:09.834 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.054713+0000", 2024-02-05T04:54:09.834 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.3683000000000006e-05 2024-02-05T04:54:09.835 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.835 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.835 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:09.835 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.054727+0000", 2024-02-05T04:54:09.835 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.3633e-05 2024-02-05T04:54:09.835 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.835 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.835 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:09.835 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.063005+0000", 2024-02-05T04:54:09.835 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0082776210000000006 2024-02-05T04:54:09.835 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.835 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.835 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:09.835 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.063042+0000", 2024-02-05T04:54:09.835 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.7793999999999998e-05 2024-02-05T04:54:09.835 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:09.836 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:09.836 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:09.836 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.836 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.836 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.bs1 74/57 ECSubWrite(tid=340, reqid=client.4634.0:687, at_version=74'92, trim_to=0'0, roll_forward_to=74'88))", 2024-02-05T04:54:09.836 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:54:06.054677+0000", 2024-02-05T04:54:09.836 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 3.7065567499999998, 2024-02-05T04:54:09.836 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.014039972, 2024-02-05T04:54:09.836 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:09.836 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:54:09.836 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:09.836 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.836 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:09.836 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.054677+0000", 2024-02-05T04:54:09.836 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:09.837 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.837 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.837 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:09.837 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.054677+0000", 2024-02-05T04:54:09.837 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:09.837 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.837 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.837 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:09.837 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.054678+0000", 2024-02-05T04:54:09.837 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.46e-07 2024-02-05T04:54:09.837 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.837 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.837 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:09.837 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.054796+0000", 2024-02-05T04:54:09.837 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.000118152 2024-02-05T04:54:09.837 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.838 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.838 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:09.838 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.054808+0000", 2024-02-05T04:54:09.838 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.145e-05 2024-02-05T04:54:09.838 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.838 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.838 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:09.838 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.054815+0000", 2024-02-05T04:54:09.838 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.7980000000000006e-06 2024-02-05T04:54:09.838 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.838 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.838 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:09.838 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.054950+0000", 2024-02-05T04:54:09.838 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.000134456 2024-02-05T04:54:09.839 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.839 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.839 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:09.839 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.054968+0000", 2024-02-05T04:54:09.839 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.861e-05 2024-02-05T04:54:09.839 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.839 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.839 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:09.839 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.068684+0000", 2024-02-05T04:54:09.839 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.013715364000000001 2024-02-05T04:54:09.839 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.839 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.839 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:09.839 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.068717+0000", 2024-02-05T04:54:09.839 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.3695999999999997e-05 2024-02-05T04:54:09.839 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:09.839 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:09.840 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:09.840 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.840 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.840 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.19s1 74/67 ECSubWrite(tid=10, reqid=osd.9.0:9, at_version=0'0, trim_to=0'0, roll_forward_to=74'29))", 2024-02-05T04:54:09.840 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:54:06.054808+0000", 2024-02-05T04:54:09.840 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 3.7064257940000003, 2024-02-05T04:54:09.840 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0078967269999999992, 2024-02-05T04:54:09.840 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:09.840 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:54:09.840 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:09.840 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.840 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:09.840 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.054808+0000", 2024-02-05T04:54:09.840 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:09.840 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.841 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.841 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:09.841 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.054808+0000", 2024-02-05T04:54:09.841 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:09.841 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.841 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.841 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:09.841 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.054809+0000", 2024-02-05T04:54:09.841 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.0210000000000001e-06 2024-02-05T04:54:09.841 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.841 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.841 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:09.841 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.054821+0000", 2024-02-05T04:54:09.841 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.1464999999999999e-05 2024-02-05T04:54:09.841 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.842 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.842 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:09.842 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.054851+0000", 2024-02-05T04:54:09.842 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.9592999999999999e-05 2024-02-05T04:54:09.842 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.842 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.842 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:09.842 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.055017+0000", 2024-02-05T04:54:09.842 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.000166618 2024-02-05T04:54:09.842 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.842 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.842 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:09.842 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.055098+0000", 2024-02-05T04:54:09.842 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.0650999999999997e-05 2024-02-05T04:54:09.842 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.842 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.843 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:09.843 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.055108+0000", 2024-02-05T04:54:09.843 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.0456e-05 2024-02-05T04:54:09.843 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.843 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.843 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:09.843 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.062665+0000", 2024-02-05T04:54:09.843 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0075564350000000002 2024-02-05T04:54:09.843 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.843 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.843 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:09.843 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.062705+0000", 2024-02-05T04:54:09.843 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.0488000000000003e-05 2024-02-05T04:54:09.843 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:09.844 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:09.844 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:09.844 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.844 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.844 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.5s5 74/62 ECSubWrite(tid=80, reqid=osd.15.0:80, at_version=74'61, trim_to=0'0, roll_forward_to=72'57))", 2024-02-05T04:54:09.844 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:54:06.058060+0000", 2024-02-05T04:54:09.844 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 3.703174186, 2024-02-05T04:54:09.844 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.010553674000000001, 2024-02-05T04:54:09.844 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:09.844 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:54:09.844 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:09.844 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.844 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:09.844 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.058060+0000", 2024-02-05T04:54:09.844 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:09.845 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.845 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.845 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:09.845 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.058060+0000", 2024-02-05T04:54:09.845 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:09.845 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.845 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.845 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:09.845 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.058061+0000", 2024-02-05T04:54:09.845 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.7199999999999997e-07 2024-02-05T04:54:09.845 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.845 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.845 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:09.845 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.058084+0000", 2024-02-05T04:54:09.845 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.2657999999999999e-05 2024-02-05T04:54:09.845 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.846 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.846 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:09.846 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.058096+0000", 2024-02-05T04:54:09.846 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.2611e-05 2024-02-05T04:54:09.846 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.846 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.846 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:09.846 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.058106+0000", 2024-02-05T04:54:09.846 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.2629999999999992e-06 2024-02-05T04:54:09.846 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.846 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.846 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:09.846 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.060169+0000", 2024-02-05T04:54:09.846 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0020636539999999998 2024-02-05T04:54:09.846 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.846 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.847 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:09.847 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.060179+0000", 2024-02-05T04:54:09.847 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.6719999999999999e-06 2024-02-05T04:54:09.847 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.847 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.847 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:09.847 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.068579+0000", 2024-02-05T04:54:09.847 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0083999480000000008 2024-02-05T04:54:09.847 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.847 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.847 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:09.847 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.068614+0000", 2024-02-05T04:54:09.847 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.4895999999999999e-05 2024-02-05T04:54:09.847 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:09.848 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:09.848 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:09.848 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.848 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.848 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.16s6 74/67 ECSubWrite(tid=155, reqid=osd.11.0:155, at_version=74'34, trim_to=0'0, roll_forward_to=72'32))", 2024-02-05T04:54:09.848 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:54:06.060146+0000", 2024-02-05T04:54:09.848 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 3.7010886670000001, 2024-02-05T04:54:09.848 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0084121920000000006, 2024-02-05T04:54:09.848 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:09.848 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:54:09.848 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:09.848 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.848 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:09.848 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.060146+0000", 2024-02-05T04:54:09.848 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:09.849 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.849 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.849 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:09.849 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.060146+0000", 2024-02-05T04:54:09.849 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:09.849 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.849 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.849 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:09.849 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.060146+0000", 2024-02-05T04:54:09.849 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.1100000000000005e-07 2024-02-05T04:54:09.849 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.849 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.849 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:09.849 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.060167+0000", 2024-02-05T04:54:09.849 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.1007000000000001e-05 2024-02-05T04:54:09.850 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.850 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.850 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:09.850 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.060181+0000", 2024-02-05T04:54:09.850 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.3142999999999999e-05 2024-02-05T04:54:09.850 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.850 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.850 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:09.850 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.060189+0000", 2024-02-05T04:54:09.850 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.8580000000000006e-06 2024-02-05T04:54:09.850 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.850 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.850 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:09.850 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.060261+0000", 2024-02-05T04:54:09.850 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.1780999999999996e-05 2024-02-05T04:54:09.851 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.851 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.851 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:09.851 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.060271+0000", 2024-02-05T04:54:09.851 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.7079999999999992e-06 2024-02-05T04:54:09.851 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.851 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.851 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:09.851 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.068512+0000", 2024-02-05T04:54:09.851 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0082413220000000006 2024-02-05T04:54:09.851 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.851 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.851 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:09.851 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.068558+0000", 2024-02-05T04:54:09.851 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.5562000000000002e-05 2024-02-05T04:54:09.851 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:09.852 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:09.852 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:09.852 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.852 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.852 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.fs6 74/31 ECSubWrite(tid=157, reqid=osd.11.0:157, at_version=74'64, trim_to=0'0, roll_forward_to=72'62))", 2024-02-05T04:54:09.852 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:54:06.063677+0000", 2024-02-05T04:54:09.852 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 3.6975567809999998, 2024-02-05T04:54:09.852 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0075878940000000004, 2024-02-05T04:54:09.852 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:09.852 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:54:09.852 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:09.852 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.852 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:09.852 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.063677+0000", 2024-02-05T04:54:09.852 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:09.853 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.853 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.853 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:09.853 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.063677+0000", 2024-02-05T04:54:09.853 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:09.853 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.853 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.853 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:09.853 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.063678+0000", 2024-02-05T04:54:09.853 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.2300000000000001e-07 2024-02-05T04:54:09.853 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.853 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.853 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:09.853 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.063703+0000", 2024-02-05T04:54:09.854 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.4955000000000001e-05 2024-02-05T04:54:09.854 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.854 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.854 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:09.854 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.063713+0000", 2024-02-05T04:54:09.854 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.5850000000000006e-06 2024-02-05T04:54:09.854 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.854 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.854 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:09.854 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.063721+0000", 2024-02-05T04:54:09.854 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.5429999999999994e-06 2024-02-05T04:54:09.854 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.854 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.854 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:09.854 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.063782+0000", 2024-02-05T04:54:09.854 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.1126999999999999e-05 2024-02-05T04:54:09.855 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.855 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.855 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:09.855 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.063792+0000", 2024-02-05T04:54:09.855 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.0118999999999999e-05 2024-02-05T04:54:09.855 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.855 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.855 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:09.855 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.071229+0000", 2024-02-05T04:54:09.855 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0074366149999999997 2024-02-05T04:54:09.855 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:09.855 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:09.855 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:09.855 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.071265+0000", 2024-02-05T04:54:09.855 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.6427000000000002e-05 2024-02-05T04:54:09.855 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:09.856 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:09.856 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:09.856 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:09.856 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:09.856 INFO:teuthology.orchestra.run.smithi091.stdout:} 2024-02-05T04:54:09.857 INFO:teuthology.orchestra.run.smithi062.stdout:osd.15: {} 2024-02-05T04:54:09.858 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-02-05T04:54:09.859 INFO:teuthology.orchestra.run.smithi062.stderr:osd.15: osd_enable_op_tracker = '' 2024-02-05T04:54:09.859 INFO:tasks.ceph.osd.2.smithi150.stderr:2024-02-05T04:54:09.856+0000 7f7337bf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 34014) UID: 0 2024-02-05T04:54:09.958 INFO:tasks.ceph.osd.10.smithi150.stderr:2024-02-05T04:54:09.957+0000 7f3075a4d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 34018) UID: 0 2024-02-05T04:54:10.005 INFO:teuthology.orchestra.run.smithi091.stdout:{ 2024-02-05T04:54:10.005 INFO:teuthology.orchestra.run.smithi091.stdout: "ops": [], 2024-02-05T04:54:10.005 INFO:teuthology.orchestra.run.smithi091.stdout: "num_ops": 0 2024-02-05T04:54:10.005 INFO:teuthology.orchestra.run.smithi091.stdout:} 2024-02-05T04:54:10.013 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-02-05T04:54:10.054 INFO:teuthology.orchestra.run.smithi062.stderr:reweighted osd.13 to 1 (10000) 2024-02-05T04:54:10.059 INFO:tasks.rados.rados.0.smithi196.stdout:308: done (0 left) 2024-02-05T04:54:10.059 INFO:tasks.rados.rados.0.smithi196.stdout:append oid 31 current snap is 25 2024-02-05T04:54:10.059 INFO:tasks.rados.rados.0.smithi196.stdout:309: seq_num 132 ranges {2818048=671744,3489792=573440,4063232=753664,4816896=229376} 2024-02-05T04:54:10.060 INFO:tasks.ceph.osd.7.smithi196.stderr:2024-02-05T04:54:10.058+0000 7fb281419640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34606) UID: 0 2024-02-05T04:54:10.072 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 14 1 2024-02-05T04:54:10.079 INFO:tasks.rados.rados.0.smithi196.stdout:309: writing smithi19639362-31 from 2818048 to 3489792 tid 1 2024-02-05T04:54:10.082 INFO:tasks.rados.rados.0.smithi196.stdout:309: writing smithi19639362-31 from 3489792 to 4063232 tid 2 2024-02-05T04:54:10.087 INFO:tasks.rados.rados.0.smithi196.stdout:309: writing smithi19639362-31 from 4063232 to 4816896 tid 3 2024-02-05T04:54:10.088 INFO:tasks.rados.rados.0.smithi196.stdout:309: writing smithi19639362-31 from 4816896 to 5046272 tid 4 2024-02-05T04:54:10.088 INFO:tasks.rados.rados.0.smithi196.stdout:310: delete oid 18 current snap is 25 2024-02-05T04:54:10.088 INFO:tasks.rados.rados.0.smithi196.stdout:309: finishing write tid 1 to smithi19639362-31 2024-02-05T04:54:10.089 INFO:tasks.rados.rados.0.smithi196.stdout:309: finishing write tid 2 to smithi19639362-31 2024-02-05T04:54:10.090 INFO:tasks.rados.rados.0.smithi196.stdout:310: done (1 left) 2024-02-05T04:54:10.090 INFO:tasks.rados.rados.0.smithi196.stdout:311: snap_create 2024-02-05T04:54:10.093 INFO:tasks.rados.rados.0.smithi196.stdout:309: finishing write tid 3 to smithi19639362-31 2024-02-05T04:54:10.095 INFO:tasks.rados.rados.0.smithi196.stdout:309: finishing write tid 4 to smithi19639362-31 2024-02-05T04:54:10.096 INFO:tasks.rados.rados.0.smithi196.stdout:309: finishing write tid 5 to smithi19639362-31 2024-02-05T04:54:10.097 INFO:tasks.rados.rados.0.smithi196.stdout:309: finishing write tid 6 to smithi19639362-31 2024-02-05T04:54:10.097 INFO:tasks.rados.rados.0.smithi196.stdout:309: oid 31 updating version 0 to 95 2024-02-05T04:54:10.097 INFO:tasks.rados.rados.0.smithi196.stdout:309: oid 31 version 95 is already newer than 92 2024-02-05T04:54:10.097 INFO:tasks.rados.rados.0.smithi196.stdout:309: oid 31 version 95 is already newer than 91 2024-02-05T04:54:10.097 INFO:tasks.rados.rados.0.smithi196.stdout:309: oid 31 version 95 is already newer than 93 2024-02-05T04:54:10.098 INFO:tasks.rados.rados.0.smithi196.stdout:309: oid 31 version 95 is already newer than 94 2024-02-05T04:54:10.098 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 31 v 95 (ObjNum 132 snap 25 seq_num 132) dirty exists 2024-02-05T04:54:10.098 INFO:tasks.rados.rados.0.smithi196.stdout:309: left oid 31 (ObjNum 132 snap 25 seq_num 132) 2024-02-05T04:54:10.127 INFO:teuthology.orchestra.run.smithi091.stdout:{ 2024-02-05T04:54:10.127 INFO:teuthology.orchestra.run.smithi091.stdout: "ops": [], 2024-02-05T04:54:10.127 INFO:teuthology.orchestra.run.smithi091.stdout: "complaint_time": 30, 2024-02-05T04:54:10.127 INFO:teuthology.orchestra.run.smithi091.stdout: "num_blocked_ops": 0 2024-02-05T04:54:10.127 INFO:teuthology.orchestra.run.smithi091.stdout:} 2024-02-05T04:54:10.136 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-02-05T04:54:10.160 INFO:tasks.ceph.osd.11.smithi196.stderr:2024-02-05T04:54:10.159+0000 7f85a8d86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 34609) UID: 0 2024-02-05T04:54:10.243 INFO:teuthology.orchestra.run.smithi091.stdout:{ 2024-02-05T04:54:10.243 INFO:teuthology.orchestra.run.smithi091.stdout: "size": 20, 2024-02-05T04:54:10.243 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 600, 2024-02-05T04:54:10.243 INFO:teuthology.orchestra.run.smithi091.stdout: "ops": [ 2024-02-05T04:54:10.243 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.243 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.5s2 37/31 ECSubWrite(tid=20, reqid=osd.15.0:19, at_version=0'0, trim_to=0'0, roll_forward_to=37'12))", 2024-02-05T04:54:10.244 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:29.137152+0000", 2024-02-05T04:54:10.244 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 41.104587240000001, 2024-02-05T04:54:10.244 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0059073499999999996, 2024-02-05T04:54:10.244 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:10.244 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:54:10.244 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:10.244 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.244 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:10.244 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.137152+0000", 2024-02-05T04:54:10.244 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:10.244 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.244 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.244 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:10.245 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.137152+0000", 2024-02-05T04:54:10.245 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:10.245 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.245 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.245 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:10.245 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.137153+0000", 2024-02-05T04:54:10.245 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.8299999999999996e-07 2024-02-05T04:54:10.245 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.245 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.245 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:10.245 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.137166+0000", 2024-02-05T04:54:10.245 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.3200000000000001e-05 2024-02-05T04:54:10.245 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.245 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.245 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:10.245 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.137184+0000", 2024-02-05T04:54:10.246 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.7927999999999998e-05 2024-02-05T04:54:10.246 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.246 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.246 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:10.246 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.137191+0000", 2024-02-05T04:54:10.246 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.559e-06 2024-02-05T04:54:10.246 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.246 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.246 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:10.246 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.137379+0000", 2024-02-05T04:54:10.246 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00018750599999999999 2024-02-05T04:54:10.246 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.246 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.246 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:10.246 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.137384+0000", 2024-02-05T04:54:10.247 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.237e-06 2024-02-05T04:54:10.247 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.247 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.247 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:10.247 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.143018+0000", 2024-02-05T04:54:10.247 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0056338990000000004 2024-02-05T04:54:10.247 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.247 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.247 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:10.247 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.143059+0000", 2024-02-05T04:54:10.247 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.1338000000000003e-05 2024-02-05T04:54:10.247 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.247 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:10.247 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.247 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.248 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.248 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.es6 37/31 ECSubWrite(tid=22, reqid=client.4634.0:249, at_version=37'16, trim_to=0'0, roll_forward_to=37'12))", 2024-02-05T04:54:10.248 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:29.192133+0000", 2024-02-05T04:54:10.248 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 41.049606390000001, 2024-02-05T04:54:10.248 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0060557249999999996, 2024-02-05T04:54:10.248 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:10.248 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:54:10.248 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:10.248 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.248 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:10.248 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.192133+0000", 2024-02-05T04:54:10.248 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:10.249 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.249 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.249 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:10.249 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.192133+0000", 2024-02-05T04:54:10.249 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:10.249 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.249 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.249 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:10.249 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.192133+0000", 2024-02-05T04:54:10.249 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.6000000000000003e-07 2024-02-05T04:54:10.249 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.249 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.249 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:10.250 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.192150+0000", 2024-02-05T04:54:10.250 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.6765e-05 2024-02-05T04:54:10.250 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.250 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.250 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:10.250 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.192160+0000", 2024-02-05T04:54:10.250 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.0185e-05 2024-02-05T04:54:10.250 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.251 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.251 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:10.251 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.192166+0000", 2024-02-05T04:54:10.251 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.7030000000000003e-06 2024-02-05T04:54:10.251 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.251 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.251 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:10.251 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.192212+0000", 2024-02-05T04:54:10.252 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.5952999999999999e-05 2024-02-05T04:54:10.252 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.252 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.252 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:10.252 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.192218+0000", 2024-02-05T04:54:10.252 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.5790000000000003e-06 2024-02-05T04:54:10.252 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.252 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.252 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:10.252 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.198162+0000", 2024-02-05T04:54:10.252 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0059439580000000001 2024-02-05T04:54:10.252 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.252 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.252 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:10.253 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.198189+0000", 2024-02-05T04:54:10.253 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.6922000000000001e-05 2024-02-05T04:54:10.253 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.253 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:10.253 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.253 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.253 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.253 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.cs2 38/31 ECSubWrite(tid=41, reqid=client.4634.0:267, at_version=38'3, trim_to=0'0, roll_forward_to=33'2))", 2024-02-05T04:54:10.253 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:30.181074+0000", 2024-02-05T04:54:10.253 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 40.060665325999999, 2024-02-05T04:54:10.253 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.005888557, 2024-02-05T04:54:10.253 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:10.253 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:54:10.253 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:10.253 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.254 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:10.254 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.181074+0000", 2024-02-05T04:54:10.254 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:10.254 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.254 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.254 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:10.254 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.181074+0000", 2024-02-05T04:54:10.254 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:10.254 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.254 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.254 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:10.254 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.181074+0000", 2024-02-05T04:54:10.254 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.6400000000000002e-07 2024-02-05T04:54:10.254 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.254 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.254 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:10.255 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.181763+0000", 2024-02-05T04:54:10.255 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00068900500000000002 2024-02-05T04:54:10.255 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.255 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.255 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:10.255 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.181773+0000", 2024-02-05T04:54:10.255 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.9680000000000001e-06 2024-02-05T04:54:10.255 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.255 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.255 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:10.255 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.181780+0000", 2024-02-05T04:54:10.255 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.528e-06 2024-02-05T04:54:10.255 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.255 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.256 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:10.256 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.181845+0000", 2024-02-05T04:54:10.256 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.5100999999999999e-05 2024-02-05T04:54:10.256 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.256 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.256 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:10.256 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.181854+0000", 2024-02-05T04:54:10.256 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.6759999999999996e-06 2024-02-05T04:54:10.256 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.256 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.256 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:10.256 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.186903+0000", 2024-02-05T04:54:10.256 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0050493580000000003 2024-02-05T04:54:10.256 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.257 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.257 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:10.257 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.186962+0000", 2024-02-05T04:54:10.257 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.9357000000000001e-05 2024-02-05T04:54:10.257 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.257 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:10.257 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.257 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.257 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.257 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.cs2 38/31 ECSubWrite(tid=42, reqid=client.4634.0:268, at_version=38'4, trim_to=0'0, roll_forward_to=33'2))", 2024-02-05T04:54:10.257 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:30.183587+0000", 2024-02-05T04:54:10.257 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 40.058151813999999, 2024-02-05T04:54:10.257 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0055219010000000001, 2024-02-05T04:54:10.258 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:10.258 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:54:10.258 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:10.258 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.258 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:10.258 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.183587+0000", 2024-02-05T04:54:10.258 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:10.258 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.258 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.258 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:10.258 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.183587+0000", 2024-02-05T04:54:10.258 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:10.259 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.259 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.259 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:10.259 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.183588+0000", 2024-02-05T04:54:10.259 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.4899999999999995e-07 2024-02-05T04:54:10.259 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.259 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.259 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:10.259 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.183765+0000", 2024-02-05T04:54:10.259 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.000177257 2024-02-05T04:54:10.259 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.259 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.259 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:10.260 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.183781+0000", 2024-02-05T04:54:10.260 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.5284000000000001e-05 2024-02-05T04:54:10.260 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.260 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.260 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:10.260 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.183789+0000", 2024-02-05T04:54:10.260 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.524e-06 2024-02-05T04:54:10.260 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.260 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.260 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:10.260 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.183850+0000", 2024-02-05T04:54:10.260 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.0494000000000001e-05 2024-02-05T04:54:10.261 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.261 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.261 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:10.261 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.183859+0000", 2024-02-05T04:54:10.261 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.8749999999999995e-06 2024-02-05T04:54:10.261 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.261 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.261 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:10.261 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.189078+0000", 2024-02-05T04:54:10.261 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0052183480000000003 2024-02-05T04:54:10.261 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.261 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.262 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:10.262 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.189109+0000", 2024-02-05T04:54:10.262 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.1470000000000002e-05 2024-02-05T04:54:10.262 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.262 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:10.262 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.262 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.262 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.262 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.4s2 38/31 ECSubWrite(tid=30, reqid=client.4634.0:278, at_version=0'0, trim_to=0'0, roll_forward_to=38'17))", 2024-02-05T04:54:10.262 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:30.200115+0000", 2024-02-05T04:54:10.262 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 40.041624585000001, 2024-02-05T04:54:10.262 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0055056649999999999, 2024-02-05T04:54:10.262 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:10.263 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:54:10.263 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:10.263 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.263 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:10.263 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.200115+0000", 2024-02-05T04:54:10.263 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:10.263 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.263 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.263 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:10.263 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.200115+0000", 2024-02-05T04:54:10.263 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:10.263 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.263 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.263 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:10.264 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.200115+0000", 2024-02-05T04:54:10.264 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.6300000000000005e-07 2024-02-05T04:54:10.264 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.264 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.264 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:10.264 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.200125+0000", 2024-02-05T04:54:10.264 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.3060000000000003e-06 2024-02-05T04:54:10.264 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.264 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.264 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:10.265 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.200139+0000", 2024-02-05T04:54:10.265 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.4252e-05 2024-02-05T04:54:10.265 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.265 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.265 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:10.265 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.200148+0000", 2024-02-05T04:54:10.265 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.7050000000000005e-06 2024-02-05T04:54:10.265 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.265 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.265 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:10.265 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.200211+0000", 2024-02-05T04:54:10.266 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.3193000000000003e-05 2024-02-05T04:54:10.266 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.266 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.266 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:10.266 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.200222+0000", 2024-02-05T04:54:10.266 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.1204e-05 2024-02-05T04:54:10.266 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.266 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.266 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:10.266 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.205586+0000", 2024-02-05T04:54:10.266 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.005364382 2024-02-05T04:54:10.266 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.266 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.266 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:10.267 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.205620+0000", 2024-02-05T04:54:10.267 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.396e-05 2024-02-05T04:54:10.267 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.267 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:10.267 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.267 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.267 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.267 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.3s5 43/31 ECSubWrite(tid=63, reqid=osd.4.0:63, at_version=43'27, trim_to=0'0, roll_forward_to=40'25))", 2024-02-05T04:54:10.267 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:35.173301+0000", 2024-02-05T04:54:10.267 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 35.068438200999999, 2024-02-05T04:54:10.267 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.006998181, 2024-02-05T04:54:10.267 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:10.267 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:54:10.268 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:10.268 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.268 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:10.268 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.173301+0000", 2024-02-05T04:54:10.268 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:10.268 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.268 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.268 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:10.268 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.173301+0000", 2024-02-05T04:54:10.268 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:10.268 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.268 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.268 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:10.269 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.173301+0000", 2024-02-05T04:54:10.269 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.9600000000000001e-07 2024-02-05T04:54:10.269 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.269 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.269 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:10.269 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.173316+0000", 2024-02-05T04:54:10.269 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.5274e-05 2024-02-05T04:54:10.269 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.269 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.269 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:10.269 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.173328+0000", 2024-02-05T04:54:10.271 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.1199000000000001e-05 2024-02-05T04:54:10.271 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.271 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.271 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:10.271 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.173336+0000", 2024-02-05T04:54:10.271 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.5779999999999999e-06 2024-02-05T04:54:10.271 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.271 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.271 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:10.271 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.173389+0000", 2024-02-05T04:54:10.271 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.2818999999999997e-05 2024-02-05T04:54:10.271 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.271 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.271 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:10.272 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.173395+0000", 2024-02-05T04:54:10.272 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.0769999999999996e-06 2024-02-05T04:54:10.272 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.272 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.272 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:10.272 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.180250+0000", 2024-02-05T04:54:10.272 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0068552889999999997 2024-02-05T04:54:10.273 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.273 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.273 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:10.273 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.180299+0000", 2024-02-05T04:54:10.273 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.8748999999999997e-05 2024-02-05T04:54:10.273 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.273 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:10.273 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.273 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.274 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.274 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.7s5 57/31 ECSubWrite(tid=238, reqid=osd.13.0:238, at_version=57'27, trim_to=0'0, roll_forward_to=52'25))", 2024-02-05T04:54:10.274 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:48.946664+0000", 2024-02-05T04:54:10.274 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 21.295075154999999, 2024-02-05T04:54:10.274 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0057548510000000001, 2024-02-05T04:54:10.274 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:10.274 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:54:10.274 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:10.274 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.274 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:10.274 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.946664+0000", 2024-02-05T04:54:10.274 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:10.275 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.275 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.275 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:10.275 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.946664+0000", 2024-02-05T04:54:10.275 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:10.275 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.275 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.275 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:10.275 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.946664+0000", 2024-02-05T04:54:10.275 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.4299999999999999e-07 2024-02-05T04:54:10.275 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.275 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.275 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:10.276 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.946685+0000", 2024-02-05T04:54:10.276 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.0449999999999999e-05 2024-02-05T04:54:10.276 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.276 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.276 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:10.276 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.946700+0000", 2024-02-05T04:54:10.276 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.5608e-05 2024-02-05T04:54:10.276 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.276 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.276 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:10.276 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.946711+0000", 2024-02-05T04:54:10.276 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.0373999999999999e-05 2024-02-05T04:54:10.276 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.276 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.277 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:10.277 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.946778+0000", 2024-02-05T04:54:10.277 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.6884000000000005e-05 2024-02-05T04:54:10.277 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.277 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.277 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:10.277 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.946786+0000", 2024-02-05T04:54:10.277 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.0690000000000002e-06 2024-02-05T04:54:10.277 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.277 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.277 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:10.277 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.952361+0000", 2024-02-05T04:54:10.277 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0055754480000000002 2024-02-05T04:54:10.277 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.277 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.278 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:10.278 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.952419+0000", 2024-02-05T04:54:10.278 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.7775000000000002e-05 2024-02-05T04:54:10.278 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.278 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:10.278 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.278 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.278 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.278 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.7s5 57/31 ECSubWrite(tid=239, reqid=osd.13.0:239, at_version=57'29, trim_to=0'0, roll_forward_to=52'25))", 2024-02-05T04:54:10.278 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:48.947110+0000", 2024-02-05T04:54:10.278 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 21.294629311000001, 2024-02-05T04:54:10.278 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0062897489999999999, 2024-02-05T04:54:10.278 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:10.278 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:54:10.279 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:10.279 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.279 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:10.279 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.947110+0000", 2024-02-05T04:54:10.279 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:10.279 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.279 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.279 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:10.279 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.947110+0000", 2024-02-05T04:54:10.279 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:10.279 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.279 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.280 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:10.280 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.947110+0000", 2024-02-05T04:54:10.280 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.6899999999999998e-07 2024-02-05T04:54:10.280 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.280 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.280 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:10.280 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.947130+0000", 2024-02-05T04:54:10.280 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.9894e-05 2024-02-05T04:54:10.280 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.280 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.280 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:10.280 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.947140+0000", 2024-02-05T04:54:10.280 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.4599999999999992e-06 2024-02-05T04:54:10.281 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.281 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.281 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:10.281 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.947147+0000", 2024-02-05T04:54:10.281 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.6629999999999993e-06 2024-02-05T04:54:10.281 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.281 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.281 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:10.281 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.947202+0000", 2024-02-05T04:54:10.281 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.4768999999999997e-05 2024-02-05T04:54:10.281 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.282 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.282 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:10.282 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.947211+0000", 2024-02-05T04:54:10.282 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.9819999999999997e-06 2024-02-05T04:54:10.282 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.282 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.282 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:10.282 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.953367+0000", 2024-02-05T04:54:10.282 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0061559199999999996 2024-02-05T04:54:10.282 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.282 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.282 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:10.282 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.953400+0000", 2024-02-05T04:54:10.282 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.2592000000000002e-05 2024-02-05T04:54:10.283 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.283 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:10.283 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.283 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.283 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.283 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.ds6 58/31 ECSubWrite(tid=56, reqid=client.4634.0:526, at_version=58'35, trim_to=0'0, roll_forward_to=57'34))", 2024-02-05T04:54:10.283 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:49.970507+0000", 2024-02-05T04:54:10.283 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 20.271231749999998, 2024-02-05T04:54:10.283 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.008805551, 2024-02-05T04:54:10.283 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:10.283 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:54:10.283 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:10.283 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.283 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:10.283 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:49.970507+0000", 2024-02-05T04:54:10.284 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:10.284 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.284 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.284 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:10.284 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:49.970507+0000", 2024-02-05T04:54:10.284 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:10.284 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.284 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.284 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:10.284 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:49.970508+0000", 2024-02-05T04:54:10.284 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.8099999999999999e-07 2024-02-05T04:54:10.284 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.284 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.284 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:10.285 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:49.973063+0000", 2024-02-05T04:54:10.285 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0025555809999999999 2024-02-05T04:54:10.285 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.285 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.285 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:10.285 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:49.973074+0000", 2024-02-05T04:54:10.285 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.1029000000000001e-05 2024-02-05T04:54:10.285 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.285 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.285 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:10.285 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:49.973083+0000", 2024-02-05T04:54:10.285 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.5650000000000002e-06 2024-02-05T04:54:10.285 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.285 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.285 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:10.286 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:49.973168+0000", 2024-02-05T04:54:10.286 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.5018999999999994e-05 2024-02-05T04:54:10.286 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.286 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.286 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:10.286 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:49.973178+0000", 2024-02-05T04:54:10.286 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.9569999999999997e-06 2024-02-05T04:54:10.286 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.286 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.286 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:10.286 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:49.979277+0000", 2024-02-05T04:54:10.286 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0060991190000000001 2024-02-05T04:54:10.286 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.286 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.287 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:10.287 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:49.979313+0000", 2024-02-05T04:54:10.287 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.6100000000000003e-05 2024-02-05T04:54:10.287 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.287 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:10.287 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.287 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.287 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.287 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.cs2 62/31 ECSubWrite(tid=116, reqid=client.4634.0:586, at_version=62'23, trim_to=0'0, roll_forward_to=62'21))", 2024-02-05T04:54:10.287 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:54.009034+0000", 2024-02-05T04:54:10.287 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 16.232704968, 2024-02-05T04:54:10.287 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0054062199999999998, 2024-02-05T04:54:10.287 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:10.287 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:54:10.287 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:10.288 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.288 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:10.288 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:54.009034+0000", 2024-02-05T04:54:10.288 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:10.288 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.288 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.288 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:10.288 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:54.009034+0000", 2024-02-05T04:54:10.288 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:10.288 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.288 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.288 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:10.289 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:54.009038+0000", 2024-02-05T04:54:10.289 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.709e-06 2024-02-05T04:54:10.289 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.289 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.289 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:10.289 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:54.009054+0000", 2024-02-05T04:54:10.289 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.6124999999999999e-05 2024-02-05T04:54:10.289 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.289 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.289 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:10.289 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:54.009066+0000", 2024-02-05T04:54:10.289 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.1654999999999999e-05 2024-02-05T04:54:10.289 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.289 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.290 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:10.290 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:54.009074+0000", 2024-02-05T04:54:10.290 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.371e-06 2024-02-05T04:54:10.290 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.290 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.290 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:10.290 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:54.009279+0000", 2024-02-05T04:54:10.290 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.000204744 2024-02-05T04:54:10.290 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.290 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.290 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:10.290 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:54.009290+0000", 2024-02-05T04:54:10.290 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.1060000000000001e-05 2024-02-05T04:54:10.291 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.291 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.291 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:10.291 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:54.014399+0000", 2024-02-05T04:54:10.291 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.005108703 2024-02-05T04:54:10.291 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.291 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.291 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:10.291 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:54.014440+0000", 2024-02-05T04:54:10.291 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.1853000000000001e-05 2024-02-05T04:54:10.291 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.291 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:10.291 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.291 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.291 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.292 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.ds6 63/31 ECSubWrite(tid=62, reqid=osd.7.0:62, at_version=63'41, trim_to=0'0, roll_forward_to=61'37))", 2024-02-05T04:54:10.292 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:54.976666+0000", 2024-02-05T04:54:10.292 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 15.265073092, 2024-02-05T04:54:10.292 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0053503889999999997, 2024-02-05T04:54:10.292 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:10.292 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:54:10.292 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:10.292 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.292 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:10.292 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:54.976666+0000", 2024-02-05T04:54:10.292 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:10.292 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.292 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.293 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:10.293 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:54.976666+0000", 2024-02-05T04:54:10.293 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:10.293 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.293 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.293 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:10.293 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:54.976667+0000", 2024-02-05T04:54:10.293 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.9999999999999998e-07 2024-02-05T04:54:10.293 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.293 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.293 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:10.293 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:54.976687+0000", 2024-02-05T04:54:10.294 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.0649000000000002e-05 2024-02-05T04:54:10.294 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.294 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.294 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:10.294 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:54.976704+0000", 2024-02-05T04:54:10.294 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.6911000000000001e-05 2024-02-05T04:54:10.294 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.294 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.294 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:10.294 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:54.976714+0000", 2024-02-05T04:54:10.294 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.7890000000000006e-06 2024-02-05T04:54:10.294 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.294 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.294 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:10.294 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:54.976771+0000", 2024-02-05T04:54:10.295 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.7436999999999999e-05 2024-02-05T04:54:10.295 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.295 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.295 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:10.295 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:54.976782+0000", 2024-02-05T04:54:10.295 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.064e-05 2024-02-05T04:54:10.295 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.295 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.295 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:10.295 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:54.981971+0000", 2024-02-05T04:54:10.295 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0051887920000000002 2024-02-05T04:54:10.295 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.296 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.296 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:10.296 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:54.982017+0000", 2024-02-05T04:54:10.296 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.5670999999999997e-05 2024-02-05T04:54:10.296 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.296 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:10.296 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.296 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.296 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.296 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.12s2 68/67 ECSubWrite(tid=86, reqid=osd.2.0:85, at_version=0'0, trim_to=0'0, roll_forward_to=68'57))", 2024-02-05T04:54:10.296 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:59.970563+0000", 2024-02-05T04:54:10.296 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 10.271175925, 2024-02-05T04:54:10.296 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0054334589999999999, 2024-02-05T04:54:10.296 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:10.296 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:54:10.297 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:10.297 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.297 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:10.297 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.970563+0000", 2024-02-05T04:54:10.297 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:10.297 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.297 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.297 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:10.297 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.970563+0000", 2024-02-05T04:54:10.297 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:10.297 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.297 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.297 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:10.297 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.970564+0000", 2024-02-05T04:54:10.297 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.9199999999999999e-07 2024-02-05T04:54:10.298 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.298 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.298 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:10.298 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.970578+0000", 2024-02-05T04:54:10.298 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.3383999999999999e-05 2024-02-05T04:54:10.298 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.298 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.298 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:10.298 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.970591+0000", 2024-02-05T04:54:10.298 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.3586e-05 2024-02-05T04:54:10.298 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.298 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.298 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:10.299 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.970616+0000", 2024-02-05T04:54:10.299 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.491e-05 2024-02-05T04:54:10.299 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.299 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.299 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:10.299 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.970698+0000", 2024-02-05T04:54:10.299 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.1771e-05 2024-02-05T04:54:10.299 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.299 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.299 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:10.299 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.970709+0000", 2024-02-05T04:54:10.299 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.1385e-05 2024-02-05T04:54:10.299 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.299 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.299 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:10.300 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.975954+0000", 2024-02-05T04:54:10.300 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0052442820000000003 2024-02-05T04:54:10.300 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.300 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.300 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:10.300 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.975997+0000", 2024-02-05T04:54:10.300 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.3248999999999999e-05 2024-02-05T04:54:10.300 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.300 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:10.300 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.300 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.300 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.300 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.es6 69/57 ECSubWrite(tid=87, reqid=client.4634.0:625, at_version=69'62, trim_to=0'0, roll_forward_to=66'60))", 2024-02-05T04:54:10.300 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:54:00.976497+0000", 2024-02-05T04:54:10.301 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 9.2652425960000002, 2024-02-05T04:54:10.301 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0062653630000000004, 2024-02-05T04:54:10.301 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:10.301 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:54:10.301 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:10.301 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.301 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:10.301 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:00.976497+0000", 2024-02-05T04:54:10.301 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:10.301 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.301 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.301 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:10.301 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:00.976497+0000", 2024-02-05T04:54:10.301 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:10.301 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.302 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.302 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:10.302 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:00.976497+0000", 2024-02-05T04:54:10.302 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.0800000000000005e-07 2024-02-05T04:54:10.302 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.302 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.302 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:10.302 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:00.979680+0000", 2024-02-05T04:54:10.302 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0031828160000000002 2024-02-05T04:54:10.302 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.302 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.302 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:10.302 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:00.979693+0000", 2024-02-05T04:54:10.302 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.292e-05 2024-02-05T04:54:10.303 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.303 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.303 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:10.303 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:00.979700+0000", 2024-02-05T04:54:10.303 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.674e-06 2024-02-05T04:54:10.303 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.303 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.303 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:10.303 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:00.979761+0000", 2024-02-05T04:54:10.303 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.1810000000000006e-05 2024-02-05T04:54:10.303 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.303 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.303 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:10.304 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:00.979772+0000", 2024-02-05T04:54:10.304 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.0844e-05 2024-02-05T04:54:10.304 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.304 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.304 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:10.304 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:00.982723+0000", 2024-02-05T04:54:10.304 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0029509599999999999 2024-02-05T04:54:10.304 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.304 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.304 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:10.304 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:00.982762+0000", 2024-02-05T04:54:10.305 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.8831000000000001e-05 2024-02-05T04:54:10.305 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.305 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:10.305 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.305 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.305 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.305 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.cs2 74/31 ECSubWrite(tid=152, reqid=osd.11.0:152, at_version=74'35, trim_to=0'0, roll_forward_to=72'33))", 2024-02-05T04:54:10.305 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:54:06.048248+0000", 2024-02-05T04:54:10.305 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 4.1934916649999998, 2024-02-05T04:54:10.305 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0058901400000000003, 2024-02-05T04:54:10.305 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:10.305 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:54:10.305 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:10.306 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.306 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:10.306 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.048248+0000", 2024-02-05T04:54:10.306 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:10.306 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.306 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.306 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:10.306 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.048248+0000", 2024-02-05T04:54:10.306 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:10.306 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.306 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.306 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:10.307 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.048248+0000", 2024-02-05T04:54:10.307 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.9700000000000003e-07 2024-02-05T04:54:10.307 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.307 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.307 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:10.307 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.048275+0000", 2024-02-05T04:54:10.307 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.7070999999999999e-05 2024-02-05T04:54:10.307 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.307 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.307 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:10.307 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.048288+0000", 2024-02-05T04:54:10.308 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.3200999999999999e-05 2024-02-05T04:54:10.308 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.308 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.308 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:10.308 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.048297+0000", 2024-02-05T04:54:10.308 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.8659999999999995e-06 2024-02-05T04:54:10.308 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.308 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.308 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:10.308 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.048396+0000", 2024-02-05T04:54:10.308 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.8556999999999998e-05 2024-02-05T04:54:10.309 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.309 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.309 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:10.309 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.048406+0000", 2024-02-05T04:54:10.309 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.0732e-05 2024-02-05T04:54:10.309 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.309 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.309 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:10.309 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.054084+0000", 2024-02-05T04:54:10.309 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.005677334 2024-02-05T04:54:10.309 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.309 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.309 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:10.309 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.054138+0000", 2024-02-05T04:54:10.310 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.4082000000000002e-05 2024-02-05T04:54:10.310 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.310 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:10.310 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.310 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.310 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.310 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.13s4 74/67 ECSubWrite(tid=153, reqid=osd.11.0:153, at_version=74'48, trim_to=0'0, roll_forward_to=72'46))", 2024-02-05T04:54:10.310 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:54:06.049863+0000", 2024-02-05T04:54:10.310 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 4.1918758089999999, 2024-02-05T04:54:10.310 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0068401449999999997, 2024-02-05T04:54:10.310 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:10.310 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:54:10.310 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:10.311 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.311 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:10.311 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.049863+0000", 2024-02-05T04:54:10.311 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:10.311 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.311 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.311 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:10.311 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.049863+0000", 2024-02-05T04:54:10.311 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:10.311 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.311 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.311 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:10.311 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.049864+0000", 2024-02-05T04:54:10.311 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.0699999999999996e-07 2024-02-05T04:54:10.311 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.311 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.312 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:10.312 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.049886+0000", 2024-02-05T04:54:10.312 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.1801e-05 2024-02-05T04:54:10.312 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.312 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.312 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:10.312 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.049899+0000", 2024-02-05T04:54:10.312 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.2852e-05 2024-02-05T04:54:10.312 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.312 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.312 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:10.312 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.049908+0000", 2024-02-05T04:54:10.312 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.5880000000000004e-06 2024-02-05T04:54:10.313 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.313 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.313 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:10.313 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.049970+0000", 2024-02-05T04:54:10.313 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.1180999999999996e-05 2024-02-05T04:54:10.313 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.313 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.313 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:10.313 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.049978+0000", 2024-02-05T04:54:10.313 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.1980000000000001e-06 2024-02-05T04:54:10.313 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.313 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.313 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:10.313 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.056662+0000", 2024-02-05T04:54:10.313 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0066842100000000003 2024-02-05T04:54:10.314 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.314 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.314 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:10.314 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.056704+0000", 2024-02-05T04:54:10.314 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.1508000000000003e-05 2024-02-05T04:54:10.314 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.314 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:10.314 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.314 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.314 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.314 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.4s2 74/62 ECSubWrite(tid=132, reqid=osd.1.0:132, at_version=74'81, trim_to=0'0, roll_forward_to=72'79))", 2024-02-05T04:54:10.314 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:54:06.052052+0000", 2024-02-05T04:54:10.314 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 4.1896876150000004, 2024-02-05T04:54:10.315 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0077616869999999998, 2024-02-05T04:54:10.315 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:10.315 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:54:10.315 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:10.315 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.315 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:10.315 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.052052+0000", 2024-02-05T04:54:10.315 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:10.315 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.315 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.315 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:10.315 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.052052+0000", 2024-02-05T04:54:10.315 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:10.315 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.315 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.315 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:10.316 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.052052+0000", 2024-02-05T04:54:10.316 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.3e-07 2024-02-05T04:54:10.316 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.316 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.316 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:10.316 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.052078+0000", 2024-02-05T04:54:10.316 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.601e-05 2024-02-05T04:54:10.316 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.316 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.316 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:10.316 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.052090+0000", 2024-02-05T04:54:10.316 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.169e-05 2024-02-05T04:54:10.316 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.316 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.317 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:10.317 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.052099+0000", 2024-02-05T04:54:10.317 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.6340000000000007e-06 2024-02-05T04:54:10.317 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.317 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.317 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:10.317 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.052159+0000", 2024-02-05T04:54:10.317 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.0711000000000002e-05 2024-02-05T04:54:10.317 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.317 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.317 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:10.317 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.052169+0000", 2024-02-05T04:54:10.317 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.6719999999999999e-06 2024-02-05T04:54:10.317 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.317 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.317 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:10.318 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.059771+0000", 2024-02-05T04:54:10.318 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0076019809999999998 2024-02-05T04:54:10.318 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.318 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.318 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:10.318 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.059813+0000", 2024-02-05T04:54:10.318 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.2259000000000003e-05 2024-02-05T04:54:10.318 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.318 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:10.318 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.318 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.318 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.318 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.5s2 74/62 ECSubWrite(tid=79, reqid=osd.15.0:79, at_version=74'59, trim_to=0'0, roll_forward_to=72'57))", 2024-02-05T04:54:10.319 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:54:06.054501+0000", 2024-02-05T04:54:10.319 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 4.1872380739999997, 2024-02-05T04:54:10.319 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0081076719999999998, 2024-02-05T04:54:10.319 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:10.319 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:54:10.319 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:10.319 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.319 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:10.319 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.054501+0000", 2024-02-05T04:54:10.319 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:10.319 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.319 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.319 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:10.319 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.054501+0000", 2024-02-05T04:54:10.319 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:10.320 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.320 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.320 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:10.320 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.054501+0000", 2024-02-05T04:54:10.320 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.1e-07 2024-02-05T04:54:10.320 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.320 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.320 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:10.320 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.054524+0000", 2024-02-05T04:54:10.320 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.2997e-05 2024-02-05T04:54:10.320 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.320 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.320 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:10.320 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.054542+0000", 2024-02-05T04:54:10.320 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.7910999999999998e-05 2024-02-05T04:54:10.321 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.321 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.321 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:10.321 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.054552+0000", 2024-02-05T04:54:10.321 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.7610000000000002e-06 2024-02-05T04:54:10.321 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.321 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.321 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:10.321 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.054647+0000", 2024-02-05T04:54:10.321 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.5284000000000005e-05 2024-02-05T04:54:10.321 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.321 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.321 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:10.321 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.054658+0000", 2024-02-05T04:54:10.321 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.0702e-05 2024-02-05T04:54:10.322 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.322 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.322 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:10.322 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.062563+0000", 2024-02-05T04:54:10.322 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0079050839999999997 2024-02-05T04:54:10.322 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.322 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.322 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:10.322 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.062609+0000", 2024-02-05T04:54:10.322 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.5723000000000003e-05 2024-02-05T04:54:10.322 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.322 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:10.322 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.323 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.323 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.323 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.5s2 74/62 ECSubWrite(tid=80, reqid=osd.15.0:80, at_version=74'61, trim_to=0'0, roll_forward_to=72'57))", 2024-02-05T04:54:10.323 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:54:06.058107+0000", 2024-02-05T04:54:10.323 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 4.1836321649999997, 2024-02-05T04:54:10.323 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0075304430000000004, 2024-02-05T04:54:10.323 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:10.323 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:54:10.323 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:10.323 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.323 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:10.323 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.058107+0000", 2024-02-05T04:54:10.323 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:10.323 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.323 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.324 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:10.324 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.058107+0000", 2024-02-05T04:54:10.324 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:10.324 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.324 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.324 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:10.324 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.058108+0000", 2024-02-05T04:54:10.324 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.3799999999999996e-07 2024-02-05T04:54:10.324 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.324 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.324 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:10.324 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.058138+0000", 2024-02-05T04:54:10.324 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.0562999999999999e-05 2024-02-05T04:54:10.325 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.325 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.325 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:10.325 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.058151+0000", 2024-02-05T04:54:10.325 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.2889e-05 2024-02-05T04:54:10.325 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.325 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.325 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:10.325 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.058165+0000", 2024-02-05T04:54:10.325 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.3526999999999999e-05 2024-02-05T04:54:10.325 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.325 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.325 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:10.326 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.058232+0000", 2024-02-05T04:54:10.326 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.6765000000000002e-05 2024-02-05T04:54:10.326 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.326 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.326 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:10.326 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.058244+0000", 2024-02-05T04:54:10.326 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.198e-05 2024-02-05T04:54:10.326 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.326 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.326 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:10.326 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.065592+0000", 2024-02-05T04:54:10.326 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0073488909999999998 2024-02-05T04:54:10.326 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.326 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.326 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:10.327 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.065638+0000", 2024-02-05T04:54:10.327 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.5089999999999997e-05 2024-02-05T04:54:10.327 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.327 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:10.327 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.327 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.327 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.327 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.16s3 74/67 ECSubWrite(tid=155, reqid=osd.11.0:155, at_version=74'34, trim_to=0'0, roll_forward_to=72'32))", 2024-02-05T04:54:10.327 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:54:06.060382+0000", 2024-02-05T04:54:10.327 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 4.1813573259999997, 2024-02-05T04:54:10.327 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0079171720000000001, 2024-02-05T04:54:10.327 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:10.327 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:54:10.328 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:10.328 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.328 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:10.328 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.060382+0000", 2024-02-05T04:54:10.328 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:10.328 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.328 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.328 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:10.328 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.060382+0000", 2024-02-05T04:54:10.328 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:10.328 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.328 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.329 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:10.329 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.060383+0000", 2024-02-05T04:54:10.329 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.6099999999999999e-07 2024-02-05T04:54:10.329 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.329 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.329 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:10.329 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.060403+0000", 2024-02-05T04:54:10.329 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.0072999999999999e-05 2024-02-05T04:54:10.329 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.329 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.329 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:10.329 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.060418+0000", 2024-02-05T04:54:10.329 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.4965e-05 2024-02-05T04:54:10.329 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.329 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.330 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:10.330 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.060427+0000", 2024-02-05T04:54:10.330 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.6800000000000005e-06 2024-02-05T04:54:10.330 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.330 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.330 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:10.330 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.060508+0000", 2024-02-05T04:54:10.330 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.0111000000000006e-05 2024-02-05T04:54:10.330 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.330 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.330 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:10.330 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.060518+0000", 2024-02-05T04:54:10.330 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.0362e-05 2024-02-05T04:54:10.331 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.331 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.331 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:10.331 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.068258+0000", 2024-02-05T04:54:10.331 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0077400799999999999 2024-02-05T04:54:10.331 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.331 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.331 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:10.331 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.068299+0000", 2024-02-05T04:54:10.331 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.104e-05 2024-02-05T04:54:10.331 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.331 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:10.331 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.331 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.332 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.332 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.ds6 78/31 ECSubWrite(tid=95, reqid=osd.7.0:94, at_version=0'0, trim_to=0'0, roll_forward_to=78'58))", 2024-02-05T04:54:10.332 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:54:10.065815+0000", 2024-02-05T04:54:10.332 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 0.17592408400000001, 2024-02-05T04:54:10.332 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0054237440000000003, 2024-02-05T04:54:10.332 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:10.332 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:54:10.332 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:10.332 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.333 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:10.333 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:10.065815+0000", 2024-02-05T04:54:10.333 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:10.333 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.333 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.333 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:10.333 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:10.065815+0000", 2024-02-05T04:54:10.333 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:10.333 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.333 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.333 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:10.333 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:10.065816+0000", 2024-02-05T04:54:10.333 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.9699999999999996e-07 2024-02-05T04:54:10.333 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.334 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.334 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:10.334 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:10.065825+0000", 2024-02-05T04:54:10.334 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.5929999999999995e-06 2024-02-05T04:54:10.334 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.334 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.334 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:10.334 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:10.065836+0000", 2024-02-05T04:54:10.334 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.1090000000000001e-05 2024-02-05T04:54:10.334 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.334 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.334 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:10.334 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:10.065854+0000", 2024-02-05T04:54:10.334 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.7132000000000001e-05 2024-02-05T04:54:10.335 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.335 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.335 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:10.335 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:10.066248+0000", 2024-02-05T04:54:10.335 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.000394597 2024-02-05T04:54:10.335 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.335 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.335 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:10.335 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:10.066257+0000", 2024-02-05T04:54:10.335 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.3300000000000005e-06 2024-02-05T04:54:10.335 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.335 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.335 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:10.335 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:10.071199+0000", 2024-02-05T04:54:10.336 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0049411180000000004 2024-02-05T04:54:10.336 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.336 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.336 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:10.336 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:10.071239+0000", 2024-02-05T04:54:10.336 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.0287000000000002e-05 2024-02-05T04:54:10.336 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.336 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:10.336 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.336 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.336 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:10.336 INFO:teuthology.orchestra.run.smithi091.stdout:} 2024-02-05T04:54:10.338 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-02-05T04:54:10.339 INFO:tasks.ceph.osd.6.smithi150.stderr:2024-02-05T04:54:10.337+0000 7f6bea9f0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34015) UID: 0 2024-02-05T04:54:10.439 INFO:tasks.ceph.osd.8.smithi062.stderr:2024-02-05T04:54:10.439+0000 7f9385bf2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 34693) UID: 0 2024-02-05T04:54:10.444 INFO:teuthology.orchestra.run.smithi091.stdout:{ 2024-02-05T04:54:10.444 INFO:teuthology.orchestra.run.smithi091.stdout: "ops": [], 2024-02-05T04:54:10.444 INFO:teuthology.orchestra.run.smithi091.stdout: "num_ops": 0 2024-02-05T04:54:10.444 INFO:teuthology.orchestra.run.smithi091.stdout:} 2024-02-05T04:54:10.452 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-02-05T04:54:10.540 INFO:tasks.ceph.osd.6.smithi150.stderr:2024-02-05T04:54:10.540+0000 7f6bea9f0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34015) UID: 0 2024-02-05T04:54:10.557 INFO:teuthology.orchestra.run.smithi091.stdout:{ 2024-02-05T04:54:10.558 INFO:teuthology.orchestra.run.smithi091.stdout: "ops": [], 2024-02-05T04:54:10.558 INFO:teuthology.orchestra.run.smithi091.stdout: "complaint_time": 30, 2024-02-05T04:54:10.558 INFO:teuthology.orchestra.run.smithi091.stdout: "num_blocked_ops": 0 2024-02-05T04:54:10.558 INFO:teuthology.orchestra.run.smithi091.stdout:} 2024-02-05T04:54:10.566 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-02-05T04:54:10.641 INFO:tasks.ceph.osd.11.smithi196.stderr:2024-02-05T04:54:10.641+0000 7f85a8d86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 34609) UID: 0 2024-02-05T04:54:10.673 INFO:teuthology.orchestra.run.smithi091.stdout:{ 2024-02-05T04:54:10.673 INFO:teuthology.orchestra.run.smithi091.stdout: "size": 20, 2024-02-05T04:54:10.673 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 600, 2024-02-05T04:54:10.673 INFO:teuthology.orchestra.run.smithi091.stdout: "ops": [ 2024-02-05T04:54:10.673 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.673 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.ds4 37/31 ECSubWrite(tid=23, reqid=client.4634.0:225, at_version=37'14, trim_to=0'0, roll_forward_to=34'12))", 2024-02-05T04:54:10.673 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:29.146542+0000", 2024-02-05T04:54:10.673 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 41.524968321999999, 2024-02-05T04:54:10.674 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0049434240000000001, 2024-02-05T04:54:10.674 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:10.674 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:54:10.674 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:10.674 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.674 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:10.674 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.146542+0000", 2024-02-05T04:54:10.674 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:10.674 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.674 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.674 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:10.674 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.146542+0000", 2024-02-05T04:54:10.674 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:10.674 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.674 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.674 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:10.675 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.146542+0000", 2024-02-05T04:54:10.675 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.1199999999999999e-07 2024-02-05T04:54:10.675 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.675 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.675 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:10.675 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.147100+0000", 2024-02-05T04:54:10.675 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00055791499999999997 2024-02-05T04:54:10.675 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.675 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.675 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:10.675 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.147111+0000", 2024-02-05T04:54:10.675 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.1038e-05 2024-02-05T04:54:10.675 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.675 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.675 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:10.676 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.147119+0000", 2024-02-05T04:54:10.676 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.3389999999999993e-06 2024-02-05T04:54:10.676 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.676 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.676 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:10.676 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.147173+0000", 2024-02-05T04:54:10.676 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.3823999999999998e-05 2024-02-05T04:54:10.676 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.676 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.679 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:10.679 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.147180+0000", 2024-02-05T04:54:10.679 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.3899999999999998e-06 2024-02-05T04:54:10.680 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.680 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.680 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:10.680 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.151438+0000", 2024-02-05T04:54:10.680 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0042581229999999999 2024-02-05T04:54:10.680 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.680 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.680 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:10.680 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.151485+0000", 2024-02-05T04:54:10.680 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.7482999999999999e-05 2024-02-05T04:54:10.680 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.680 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:10.680 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.680 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.680 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.681 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.4s4 38/31 ECSubWrite(tid=30, reqid=client.4634.0:278, at_version=0'0, trim_to=0'0, roll_forward_to=38'17))", 2024-02-05T04:54:10.681 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:30.200120+0000", 2024-02-05T04:54:10.681 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 40.471390356000001, 2024-02-05T04:54:10.681 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0055354169999999999, 2024-02-05T04:54:10.681 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:10.681 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:54:10.681 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:10.681 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.681 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:10.681 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.200120+0000", 2024-02-05T04:54:10.681 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:10.681 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.681 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.681 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:10.681 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.200120+0000", 2024-02-05T04:54:10.682 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:10.682 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.682 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.682 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:10.682 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.200120+0000", 2024-02-05T04:54:10.682 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.5000000000000003e-07 2024-02-05T04:54:10.682 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.682 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.682 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:10.682 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.200127+0000", 2024-02-05T04:54:10.682 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.1570000000000003e-06 2024-02-05T04:54:10.682 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.682 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.682 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:10.682 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.200139+0000", 2024-02-05T04:54:10.682 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.1484000000000001e-05 2024-02-05T04:54:10.683 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.683 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.683 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:10.683 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.200147+0000", 2024-02-05T04:54:10.683 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.9249999999999995e-06 2024-02-05T04:54:10.683 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.683 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.683 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:10.683 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.200215+0000", 2024-02-05T04:54:10.683 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.8570000000000002e-05 2024-02-05T04:54:10.683 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.683 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.683 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:10.683 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.200224+0000", 2024-02-05T04:54:10.683 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.9670000000000007e-06 2024-02-05T04:54:10.684 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.684 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.684 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:10.684 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.205618+0000", 2024-02-05T04:54:10.684 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.005393994 2024-02-05T04:54:10.684 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.684 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.684 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:10.684 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.205655+0000", 2024-02-05T04:54:10.684 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.6770000000000002e-05 2024-02-05T04:54:10.684 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.684 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:10.684 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.684 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.684 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.685 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.ds4 43/31 ECSubWrite(tid=44, reqid=client.4634.0:401, at_version=43'24, trim_to=0'0, roll_forward_to=38'22))", 2024-02-05T04:54:10.685 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:35.183654+0000", 2024-02-05T04:54:10.685 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 35.487856409000003, 2024-02-05T04:54:10.685 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0052912829999999999, 2024-02-05T04:54:10.685 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:10.685 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:54:10.685 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:10.685 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.685 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:10.685 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.183654+0000", 2024-02-05T04:54:10.685 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:10.685 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.685 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.685 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:10.685 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.183654+0000", 2024-02-05T04:54:10.686 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:10.686 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.686 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.686 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:10.686 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.183654+0000", 2024-02-05T04:54:10.686 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.4999999999999998e-07 2024-02-05T04:54:10.686 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.686 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.686 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:10.686 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.183674+0000", 2024-02-05T04:54:10.686 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.0027e-05 2024-02-05T04:54:10.686 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.686 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.686 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:10.686 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.183684+0000", 2024-02-05T04:54:10.687 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.5000000000000005e-06 2024-02-05T04:54:10.687 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.687 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.687 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:10.687 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.183691+0000", 2024-02-05T04:54:10.687 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.1629999999999999e-06 2024-02-05T04:54:10.687 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.687 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.687 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:10.687 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.183759+0000", 2024-02-05T04:54:10.687 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.8559999999999994e-05 2024-02-05T04:54:10.687 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.687 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.687 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:10.687 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.183768+0000", 2024-02-05T04:54:10.688 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.0059999999999998e-06 2024-02-05T04:54:10.688 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.688 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.688 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:10.688 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.188906+0000", 2024-02-05T04:54:10.688 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0051378040000000002 2024-02-05T04:54:10.688 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.688 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.688 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:10.688 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.188945+0000", 2024-02-05T04:54:10.688 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.8872999999999999e-05 2024-02-05T04:54:10.688 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.688 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:10.688 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.689 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.689 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.689 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.4s4 53/31 ECSubWrite(tid=80, reqid=client.4634.0:501, at_version=53'55, trim_to=0'0, roll_forward_to=52'52))", 2024-02-05T04:54:10.689 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:44.971367+0000", 2024-02-05T04:54:10.689 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 25.700143033, 2024-02-05T04:54:10.689 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0062083889999999999, 2024-02-05T04:54:10.689 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:10.689 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:54:10.689 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:10.689 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.689 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:10.689 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.971367+0000", 2024-02-05T04:54:10.689 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:10.689 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.689 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.690 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:10.690 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.971367+0000", 2024-02-05T04:54:10.690 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:10.690 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.690 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.690 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:10.690 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.971368+0000", 2024-02-05T04:54:10.690 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.95e-07 2024-02-05T04:54:10.690 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.690 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.690 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:10.690 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.971551+0000", 2024-02-05T04:54:10.690 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00018280999999999999 2024-02-05T04:54:10.690 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.691 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.691 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:10.691 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.971566+0000", 2024-02-05T04:54:10.691 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.5654999999999998e-05 2024-02-05T04:54:10.691 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.691 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.691 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:10.691 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.971577+0000", 2024-02-05T04:54:10.691 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.0268999999999999e-05 2024-02-05T04:54:10.691 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.691 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.691 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:10.691 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.971654+0000", 2024-02-05T04:54:10.691 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.7328000000000002e-05 2024-02-05T04:54:10.691 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.692 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.692 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:10.692 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.971665+0000", 2024-02-05T04:54:10.692 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.0713000000000001e-05 2024-02-05T04:54:10.692 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.692 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.692 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:10.692 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.977541+0000", 2024-02-05T04:54:10.692 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0058759399999999996 2024-02-05T04:54:10.692 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.692 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.692 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:10.692 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.977575+0000", 2024-02-05T04:54:10.692 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.4678999999999998e-05 2024-02-05T04:54:10.692 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.693 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:10.693 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.693 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.693 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.693 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.4s4 53/31 ECSubWrite(tid=81, reqid=client.4634.0:502, at_version=53'56, trim_to=0'0, roll_forward_to=52'52))", 2024-02-05T04:54:10.693 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:44.973033+0000", 2024-02-05T04:54:10.693 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 25.698476898999999, 2024-02-05T04:54:10.693 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0055856899999999999, 2024-02-05T04:54:10.693 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:10.693 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:54:10.693 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:10.693 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.693 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:10.693 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.973033+0000", 2024-02-05T04:54:10.693 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:10.694 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.694 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.694 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:10.694 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.973033+0000", 2024-02-05T04:54:10.694 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:10.694 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.694 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.694 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:10.694 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.973034+0000", 2024-02-05T04:54:10.694 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.0400000000000005e-07 2024-02-05T04:54:10.694 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.694 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.694 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:10.694 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.973150+0000", 2024-02-05T04:54:10.694 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00011555199999999999 2024-02-05T04:54:10.694 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.695 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.695 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:10.695 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.973174+0000", 2024-02-05T04:54:10.695 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.4099000000000001e-05 2024-02-05T04:54:10.695 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.695 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.695 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:10.695 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.973183+0000", 2024-02-05T04:54:10.695 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.4609999999999997e-06 2024-02-05T04:54:10.695 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.695 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.695 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:10.695 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.973293+0000", 2024-02-05T04:54:10.695 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.000110318 2024-02-05T04:54:10.695 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.695 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.696 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:10.696 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.973307+0000", 2024-02-05T04:54:10.696 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.3613e-05 2024-02-05T04:54:10.696 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.696 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.696 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:10.696 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.978589+0000", 2024-02-05T04:54:10.696 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0052819920000000001 2024-02-05T04:54:10.696 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.696 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.696 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:10.696 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.978619+0000", 2024-02-05T04:54:10.696 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.9751e-05 2024-02-05T04:54:10.696 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.696 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:10.697 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.697 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.697 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.697 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.cs6 57/31 ECSubWrite(tid=94, reqid=osd.11.0:90, at_version=0'0, trim_to=0'0, roll_forward_to=57'13))", 2024-02-05T04:54:10.697 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:48.958470+0000", 2024-02-05T04:54:10.697 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 21.713040444000001, 2024-02-05T04:54:10.697 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00581906, 2024-02-05T04:54:10.697 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:10.697 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:54:10.697 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:10.697 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.697 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:10.697 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.958470+0000", 2024-02-05T04:54:10.697 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:10.697 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.698 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.698 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:10.698 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.958470+0000", 2024-02-05T04:54:10.698 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:10.698 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.698 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.698 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:10.698 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.958475+0000", 2024-02-05T04:54:10.698 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.1320000000000002e-06 2024-02-05T04:54:10.698 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.698 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.698 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:10.698 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.958483+0000", 2024-02-05T04:54:10.698 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.123e-06 2024-02-05T04:54:10.698 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.699 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.699 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:10.699 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.958492+0000", 2024-02-05T04:54:10.699 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.5000000000000005e-06 2024-02-05T04:54:10.699 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.699 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.699 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:10.699 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.958500+0000", 2024-02-05T04:54:10.699 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.515e-06 2024-02-05T04:54:10.699 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.699 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.699 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:10.699 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.958568+0000", 2024-02-05T04:54:10.699 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.7806999999999996e-05 2024-02-05T04:54:10.699 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.700 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.700 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:10.700 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.958577+0000", 2024-02-05T04:54:10.700 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.3880000000000006e-06 2024-02-05T04:54:10.700 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.700 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.700 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:10.700 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.964250+0000", 2024-02-05T04:54:10.700 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0056730360000000002 2024-02-05T04:54:10.700 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.700 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.700 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:10.700 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.964289+0000", 2024-02-05T04:54:10.700 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.8559000000000001e-05 2024-02-05T04:54:10.700 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.701 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:10.701 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.701 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.701 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.701 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.cs6 57/31 ECSubWrite(tid=98, reqid=client.4634.0:531, at_version=57'15, trim_to=0'0, roll_forward_to=57'13))", 2024-02-05T04:54:10.701 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:48.974750+0000", 2024-02-05T04:54:10.701 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 21.696759783000001, 2024-02-05T04:54:10.701 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.010037023000000001, 2024-02-05T04:54:10.701 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:10.701 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:54:10.701 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:10.701 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.701 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:10.701 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.974750+0000", 2024-02-05T04:54:10.701 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:10.702 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.702 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.702 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:10.702 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.974750+0000", 2024-02-05T04:54:10.702 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:10.702 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.702 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.702 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:10.702 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.974751+0000", 2024-02-05T04:54:10.702 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.0500000000000003e-07 2024-02-05T04:54:10.702 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.702 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.702 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:10.702 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.975454+0000", 2024-02-05T04:54:10.703 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00070269799999999997 2024-02-05T04:54:10.703 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.703 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.703 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:10.703 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.975467+0000", 2024-02-05T04:54:10.703 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.3737000000000001e-05 2024-02-05T04:54:10.703 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.703 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.703 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:10.703 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.975482+0000", 2024-02-05T04:54:10.703 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.4531e-05 2024-02-05T04:54:10.703 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.703 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.703 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:10.703 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.975585+0000", 2024-02-05T04:54:10.704 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00010341 2024-02-05T04:54:10.704 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.704 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.704 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:10.704 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.975596+0000", 2024-02-05T04:54:10.704 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.0621999999999999e-05 2024-02-05T04:54:10.704 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.704 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.704 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:10.704 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.984752+0000", 2024-02-05T04:54:10.704 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0091561630000000001 2024-02-05T04:54:10.704 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.704 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.704 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:10.705 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.984787+0000", 2024-02-05T04:54:10.705 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.5256999999999998e-05 2024-02-05T04:54:10.705 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.705 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:10.705 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.705 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.705 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.705 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.cs6 57/31 ECSubWrite(tid=99, reqid=client.4634.0:532, at_version=57'16, trim_to=0'0, roll_forward_to=57'13))", 2024-02-05T04:54:10.705 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:48.978847+0000", 2024-02-05T04:54:10.705 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 21.692662871, 2024-02-05T04:54:10.705 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0068457910000000004, 2024-02-05T04:54:10.705 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:10.705 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:54:10.705 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:10.705 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.706 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:10.706 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.978847+0000", 2024-02-05T04:54:10.706 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:10.706 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.706 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.706 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:10.706 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.978847+0000", 2024-02-05T04:54:10.706 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:10.706 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.706 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.706 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:10.706 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.978848+0000", 2024-02-05T04:54:10.706 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.4399999999999999e-07 2024-02-05T04:54:10.706 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.706 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.707 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:10.707 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.979286+0000", 2024-02-05T04:54:10.707 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00043832000000000002 2024-02-05T04:54:10.707 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.707 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.707 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:10.707 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.979305+0000", 2024-02-05T04:54:10.707 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.9196999999999999e-05 2024-02-05T04:54:10.707 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.707 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.707 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:10.707 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.979314+0000", 2024-02-05T04:54:10.707 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.8929999999999994e-06 2024-02-05T04:54:10.707 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.707 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.708 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:10.708 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.979368+0000", 2024-02-05T04:54:10.708 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.3378000000000003e-05 2024-02-05T04:54:10.708 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.708 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.708 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:10.708 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.979376+0000", 2024-02-05T04:54:10.708 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.5180000000000004e-06 2024-02-05T04:54:10.708 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.708 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.708 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:10.708 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.985661+0000", 2024-02-05T04:54:10.708 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0062852289999999998 2024-02-05T04:54:10.708 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.709 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.709 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:10.709 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:48.985693+0000", 2024-02-05T04:54:10.709 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.1511999999999999e-05 2024-02-05T04:54:10.709 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.709 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:10.709 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.709 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.709 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.709 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.4s4 58/57 ECSubWrite(tid=107, reqid=osd.1.0:107, at_version=58'59, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:10.709 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:49.954901+0000", 2024-02-05T04:54:10.709 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 20.716609501000001, 2024-02-05T04:54:10.709 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0054803919999999997, 2024-02-05T04:54:10.709 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:10.710 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:54:10.710 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:10.710 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.710 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:10.710 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:49.954901+0000", 2024-02-05T04:54:10.710 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:10.710 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.710 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.710 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:10.710 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:49.954901+0000", 2024-02-05T04:54:10.710 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:10.710 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.710 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.710 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:10.710 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:49.954901+0000", 2024-02-05T04:54:10.711 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.1399999999999996e-07 2024-02-05T04:54:10.711 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.711 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.711 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:10.711 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:49.954921+0000", 2024-02-05T04:54:10.711 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.9632000000000001e-05 2024-02-05T04:54:10.711 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.711 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.711 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:10.711 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:49.954932+0000", 2024-02-05T04:54:10.711 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.0753e-05 2024-02-05T04:54:10.711 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.711 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.711 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:10.711 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:49.954941+0000", 2024-02-05T04:54:10.712 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.6739999999999992e-06 2024-02-05T04:54:10.712 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.712 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.712 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:10.712 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:49.955008+0000", 2024-02-05T04:54:10.712 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.6682999999999998e-05 2024-02-05T04:54:10.712 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.712 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.712 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:10.712 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:49.955019+0000", 2024-02-05T04:54:10.712 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.1195999999999999e-05 2024-02-05T04:54:10.712 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.712 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.712 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:10.712 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:49.960340+0000", 2024-02-05T04:54:10.713 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0053212399999999997 2024-02-05T04:54:10.713 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.713 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.713 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:10.713 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:49.960381+0000", 2024-02-05T04:54:10.713 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.0500000000000002e-05 2024-02-05T04:54:10.713 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.713 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:10.713 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.713 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.713 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.713 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.4s4 58/57 ECSubWrite(tid=108, reqid=osd.1.0:108, at_version=58'61, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:10.713 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:49.955873+0000", 2024-02-05T04:54:10.713 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 20.715636931999999, 2024-02-05T04:54:10.714 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0053375979999999998, 2024-02-05T04:54:10.714 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:10.714 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:54:10.714 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:10.714 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.714 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:10.714 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:49.955873+0000", 2024-02-05T04:54:10.714 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:10.714 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.714 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.714 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:10.714 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:49.955873+0000", 2024-02-05T04:54:10.714 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:10.714 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.714 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.714 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:10.715 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:49.955874+0000", 2024-02-05T04:54:10.715 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.2200000000000004e-07 2024-02-05T04:54:10.715 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.715 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.715 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:10.715 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:49.955891+0000", 2024-02-05T04:54:10.715 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.7697999999999999e-05 2024-02-05T04:54:10.715 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.715 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.715 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:10.715 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:49.955903+0000", 2024-02-05T04:54:10.715 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.1686e-05 2024-02-05T04:54:10.715 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.715 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.715 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:10.715 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:49.955912+0000", 2024-02-05T04:54:10.716 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.8279999999999992e-06 2024-02-05T04:54:10.716 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.716 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.716 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:10.716 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:49.955974+0000", 2024-02-05T04:54:10.716 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.1557999999999996e-05 2024-02-05T04:54:10.716 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.716 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.716 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:10.716 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:49.955984+0000", 2024-02-05T04:54:10.716 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.0481e-05 2024-02-05T04:54:10.716 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.716 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.716 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:10.717 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:49.961152+0000", 2024-02-05T04:54:10.717 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0051678289999999997 2024-02-05T04:54:10.717 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.717 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.717 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:10.717 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:49.961211+0000", 2024-02-05T04:54:10.717 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.8896e-05 2024-02-05T04:54:10.717 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.717 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:10.717 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.717 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.717 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.717 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.ds4 58/31 ECSubWrite(tid=56, reqid=client.4634.0:526, at_version=58'35, trim_to=0'0, roll_forward_to=57'34))", 2024-02-05T04:54:10.717 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:49.970786+0000", 2024-02-05T04:54:10.718 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 20.700724284, 2024-02-05T04:54:10.718 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0067910610000000001, 2024-02-05T04:54:10.718 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:10.718 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:54:10.718 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:10.718 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.718 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:10.718 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:49.970786+0000", 2024-02-05T04:54:10.718 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:10.718 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.718 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.718 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:10.718 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:49.970786+0000", 2024-02-05T04:54:10.718 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:10.718 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.719 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.719 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:10.719 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:49.970786+0000", 2024-02-05T04:54:10.719 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.4400000000000001e-07 2024-02-05T04:54:10.719 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.719 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.719 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:10.719 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:49.973258+0000", 2024-02-05T04:54:10.719 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0024714020000000001 2024-02-05T04:54:10.719 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.719 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.719 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:10.719 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:49.973274+0000", 2024-02-05T04:54:10.719 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.6764000000000001e-05 2024-02-05T04:54:10.719 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.720 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.720 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:10.720 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:49.973286+0000", 2024-02-05T04:54:10.720 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.1711e-05 2024-02-05T04:54:10.720 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.720 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.720 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:10.720 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:49.973361+0000", 2024-02-05T04:54:10.720 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.5492000000000002e-05 2024-02-05T04:54:10.720 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.720 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.720 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:10.720 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:49.973373+0000", 2024-02-05T04:54:10.721 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.1941e-05 2024-02-05T04:54:10.721 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.721 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.721 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:10.721 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:49.977544+0000", 2024-02-05T04:54:10.721 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0041703690000000002 2024-02-05T04:54:10.721 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.721 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.721 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:10.721 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:49.977577+0000", 2024-02-05T04:54:10.721 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.3037999999999998e-05 2024-02-05T04:54:10.721 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.721 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:10.721 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.722 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.722 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.722 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.14s5 68/67 ECSubWrite(tid=15, reqid=osd.3.0:15, at_version=68'75, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:10.722 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:59.961441+0000", 2024-02-05T04:54:10.722 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 10.710069045999999, 2024-02-05T04:54:10.722 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0063606649999999997, 2024-02-05T04:54:10.722 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:10.722 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:54:10.722 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:10.722 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.722 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:10.722 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.961441+0000", 2024-02-05T04:54:10.722 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:10.722 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.722 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.722 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:10.723 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.961441+0000", 2024-02-05T04:54:10.723 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:10.723 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.723 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.723 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:10.723 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.961442+0000", 2024-02-05T04:54:10.723 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.6899999999999996e-07 2024-02-05T04:54:10.723 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.723 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.723 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:10.723 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.961464+0000", 2024-02-05T04:54:10.723 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.198e-05 2024-02-05T04:54:10.723 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.723 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.723 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:10.723 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.961486+0000", 2024-02-05T04:54:10.723 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.2138999999999999e-05 2024-02-05T04:54:10.724 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.724 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.724 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:10.724 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.961496+0000", 2024-02-05T04:54:10.724 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.0309000000000001e-05 2024-02-05T04:54:10.724 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.724 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.724 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:10.724 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.961558+0000", 2024-02-05T04:54:10.724 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.1995000000000005e-05 2024-02-05T04:54:10.724 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.724 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.724 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:10.724 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.961571+0000", 2024-02-05T04:54:10.724 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.2349e-05 2024-02-05T04:54:10.725 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.725 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.725 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:10.725 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.967762+0000", 2024-02-05T04:54:10.725 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0061913840000000003 2024-02-05T04:54:10.725 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.725 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.725 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:10.725 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.967802+0000", 2024-02-05T04:54:10.725 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.9740000000000002e-05 2024-02-05T04:54:10.725 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.725 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:10.725 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.725 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.725 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.725 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.14s5 68/67 ECSubWrite(tid=16, reqid=osd.3.0:16, at_version=68'77, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:10.725 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:59.963715+0000", 2024-02-05T04:54:10.725 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 10.707795503, 2024-02-05T04:54:10.726 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0069062849999999999, 2024-02-05T04:54:10.726 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:10.726 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:54:10.726 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:10.726 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.726 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:10.726 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.963715+0000", 2024-02-05T04:54:10.726 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:10.726 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.726 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.726 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:10.726 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.963715+0000", 2024-02-05T04:54:10.726 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:10.726 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.726 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.726 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:10.726 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.963715+0000", 2024-02-05T04:54:10.727 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.9500000000000003e-07 2024-02-05T04:54:10.727 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.727 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.727 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:10.727 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.963736+0000", 2024-02-05T04:54:10.727 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.0931e-05 2024-02-05T04:54:10.727 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.727 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.727 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:10.727 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.963744+0000", 2024-02-05T04:54:10.727 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.6939999999999995e-06 2024-02-05T04:54:10.727 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.727 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.727 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:10.727 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.963752+0000", 2024-02-05T04:54:10.727 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.7290000000000006e-06 2024-02-05T04:54:10.728 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.728 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.728 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:10.728 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.963822+0000", 2024-02-05T04:54:10.728 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.9478000000000001e-05 2024-02-05T04:54:10.728 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.728 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.728 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:10.728 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.963832+0000", 2024-02-05T04:54:10.728 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.0328e-05 2024-02-05T04:54:10.728 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.728 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.728 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:10.728 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.970582+0000", 2024-02-05T04:54:10.729 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0067499659999999996 2024-02-05T04:54:10.729 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.729 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.729 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:10.729 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.970621+0000", 2024-02-05T04:54:10.729 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.8664000000000001e-05 2024-02-05T04:54:10.729 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.729 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:10.729 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.729 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.729 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.729 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.14s5 68/67 ECSubWrite(tid=17, reqid=osd.3.0:16, at_version=0'0, trim_to=0'0, roll_forward_to=68'77))", 2024-02-05T04:54:10.729 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:59.973835+0000", 2024-02-05T04:54:10.729 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 10.697674772999999, 2024-02-05T04:54:10.729 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0049838080000000002, 2024-02-05T04:54:10.730 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:10.730 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:54:10.730 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:10.730 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.730 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:10.730 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.973835+0000", 2024-02-05T04:54:10.730 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:10.730 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.730 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.730 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:10.730 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.973835+0000", 2024-02-05T04:54:10.730 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:10.730 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.730 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.730 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:10.731 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.973836+0000", 2024-02-05T04:54:10.731 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.5600000000000005e-07 2024-02-05T04:54:10.731 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.731 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.731 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:10.731 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.973845+0000", 2024-02-05T04:54:10.731 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.9390000000000003e-06 2024-02-05T04:54:10.731 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.731 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.731 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:10.731 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.973857+0000", 2024-02-05T04:54:10.731 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.1754e-05 2024-02-05T04:54:10.731 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.731 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.731 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:10.732 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.973864+0000", 2024-02-05T04:54:10.732 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.4320000000000002e-06 2024-02-05T04:54:10.732 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.732 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.732 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:10.732 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.973911+0000", 2024-02-05T04:54:10.732 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.7030000000000002e-05 2024-02-05T04:54:10.732 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.732 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.732 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:10.732 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.973920+0000", 2024-02-05T04:54:10.732 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.7409999999999998e-06 2024-02-05T04:54:10.732 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.732 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.733 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:10.733 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.978786+0000", 2024-02-05T04:54:10.733 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0048664870000000001 2024-02-05T04:54:10.733 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.733 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.733 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:10.733 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.978819+0000", 2024-02-05T04:54:10.733 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.2669000000000002e-05 2024-02-05T04:54:10.733 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.733 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:10.733 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.733 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.733 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.733 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.cs6 69/31 ECSubWrite(tid=134, reqid=osd.11.0:133, at_version=0'0, trim_to=0'0, roll_forward_to=69'29))", 2024-02-05T04:54:10.733 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:54:00.969070+0000", 2024-02-05T04:54:10.734 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 9.7024402500000004, 2024-02-05T04:54:10.734 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0059456600000000002, 2024-02-05T04:54:10.734 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:10.734 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:54:10.734 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:10.734 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.734 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:10.734 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:00.969070+0000", 2024-02-05T04:54:10.734 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:10.734 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.734 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.734 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:10.734 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:00.969070+0000", 2024-02-05T04:54:10.734 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:10.734 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.734 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.735 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:10.735 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:00.969071+0000", 2024-02-05T04:54:10.735 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.0139999999999999e-06 2024-02-05T04:54:10.735 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.735 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.735 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:10.735 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:00.969087+0000", 2024-02-05T04:54:10.735 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.624e-05 2024-02-05T04:54:10.735 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.735 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.735 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:10.735 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:00.969101+0000", 2024-02-05T04:54:10.735 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.4151000000000001e-05 2024-02-05T04:54:10.735 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.735 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.736 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:10.736 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:00.969110+0000", 2024-02-05T04:54:10.736 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.2390000000000008e-06 2024-02-05T04:54:10.736 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.736 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.736 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:10.736 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:00.971116+0000", 2024-02-05T04:54:10.736 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0020058200000000002 2024-02-05T04:54:10.736 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.736 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.736 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:10.736 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:00.971124+0000", 2024-02-05T04:54:10.736 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.5630000000000003e-06 2024-02-05T04:54:10.736 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.736 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.737 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:10.737 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:00.974974+0000", 2024-02-05T04:54:10.737 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0038500660000000001 2024-02-05T04:54:10.737 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.737 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.737 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:10.737 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:00.975015+0000", 2024-02-05T04:54:10.737 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.1566999999999997e-05 2024-02-05T04:54:10.737 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.737 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:10.737 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.737 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.737 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.737 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.14s5 74/67 ECSubWrite(tid=23, reqid=osd.3.0:23, at_version=74'85, trim_to=0'0, roll_forward_to=72'81))", 2024-02-05T04:54:10.737 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:54:06.051519+0000", 2024-02-05T04:54:10.737 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 4.6199908169999997, 2024-02-05T04:54:10.738 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0053111540000000002, 2024-02-05T04:54:10.738 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:10.738 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:54:10.738 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:10.738 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.738 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:10.738 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.051519+0000", 2024-02-05T04:54:10.738 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:10.738 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.738 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.738 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:10.738 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.051519+0000", 2024-02-05T04:54:10.738 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:10.738 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.738 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.739 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:10.739 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.051520+0000", 2024-02-05T04:54:10.739 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.5099999999999998e-07 2024-02-05T04:54:10.739 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.739 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.739 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:10.739 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.051542+0000", 2024-02-05T04:54:10.739 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.1503000000000001e-05 2024-02-05T04:54:10.739 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.739 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.739 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:10.739 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.051564+0000", 2024-02-05T04:54:10.739 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.2167999999999998e-05 2024-02-05T04:54:10.739 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.739 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.739 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:10.740 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.051573+0000", 2024-02-05T04:54:10.740 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.5859999999999994e-06 2024-02-05T04:54:10.740 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.740 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.740 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:10.740 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.051656+0000", 2024-02-05T04:54:10.740 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.2789999999999998e-05 2024-02-05T04:54:10.740 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.740 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.740 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:10.740 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.051673+0000", 2024-02-05T04:54:10.740 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.6708999999999999e-05 2024-02-05T04:54:10.740 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.740 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.740 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:10.741 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.056769+0000", 2024-02-05T04:54:10.741 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0050963880000000003 2024-02-05T04:54:10.741 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.741 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.741 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:10.741 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.056830+0000", 2024-02-05T04:54:10.741 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.1159000000000002e-05 2024-02-05T04:54:10.741 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.741 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:10.741 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.741 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.741 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.741 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.4s4 74/62 ECSubWrite(tid=132, reqid=osd.1.0:132, at_version=74'81, trim_to=0'0, roll_forward_to=72'79))", 2024-02-05T04:54:10.741 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:54:06.052006+0000", 2024-02-05T04:54:10.741 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 4.6195041410000002, 2024-02-05T04:54:10.742 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0077442099999999996, 2024-02-05T04:54:10.742 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:10.742 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:54:10.742 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:10.742 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.742 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:10.742 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.052006+0000", 2024-02-05T04:54:10.742 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:10.742 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.742 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.742 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:10.742 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.052006+0000", 2024-02-05T04:54:10.742 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:10.742 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.742 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.743 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:10.743 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.052007+0000", 2024-02-05T04:54:10.743 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.0999999999999997e-07 2024-02-05T04:54:10.743 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.743 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.743 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:10.743 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.052029+0000", 2024-02-05T04:54:10.743 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.2439e-05 2024-02-05T04:54:10.743 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.743 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.743 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:10.743 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.052044+0000", 2024-02-05T04:54:10.743 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.4684999999999999e-05 2024-02-05T04:54:10.743 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.744 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.744 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:10.744 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.052053+0000", 2024-02-05T04:54:10.744 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.4960000000000002e-06 2024-02-05T04:54:10.744 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.744 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.744 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:10.744 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.052286+0000", 2024-02-05T04:54:10.744 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.000232835 2024-02-05T04:54:10.744 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.744 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.744 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:10.744 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.052297+0000", 2024-02-05T04:54:10.744 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.1027e-05 2024-02-05T04:54:10.744 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.745 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.745 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:10.745 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.059713+0000", 2024-02-05T04:54:10.745 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0074153839999999997 2024-02-05T04:54:10.745 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.745 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.745 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:10.745 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.059750+0000", 2024-02-05T04:54:10.745 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.7533999999999997e-05 2024-02-05T04:54:10.745 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.745 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:10.745 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.745 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.745 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.745 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.cs6 74/31 ECSubWrite(tid=154, reqid=osd.11.0:152, at_version=0'0, trim_to=0'0, roll_forward_to=74'35))", 2024-02-05T04:54:10.746 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:54:06.057679+0000", 2024-02-05T04:54:10.746 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 4.6138312250000002, 2024-02-05T04:54:10.746 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0056690839999999996, 2024-02-05T04:54:10.746 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:10.746 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:54:10.746 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:10.746 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.746 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:10.746 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.057679+0000", 2024-02-05T04:54:10.746 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:10.746 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.746 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.746 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:10.746 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.057679+0000", 2024-02-05T04:54:10.746 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:10.746 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.747 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.747 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:10.747 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.057679+0000", 2024-02-05T04:54:10.747 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.1500000000000004e-07 2024-02-05T04:54:10.747 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.747 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.747 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:10.747 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.057689+0000", 2024-02-05T04:54:10.747 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.1619999999999997e-06 2024-02-05T04:54:10.747 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.747 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.747 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:10.747 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.057713+0000", 2024-02-05T04:54:10.747 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.4158000000000001e-05 2024-02-05T04:54:10.747 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.747 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.748 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:10.748 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.057721+0000", 2024-02-05T04:54:10.748 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.9289999999999998e-06 2024-02-05T04:54:10.748 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.748 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.748 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:10.748 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.057792+0000", 2024-02-05T04:54:10.748 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.0964999999999996e-05 2024-02-05T04:54:10.748 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.748 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.748 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:10.748 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.057803+0000", 2024-02-05T04:54:10.748 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.1484000000000001e-05 2024-02-05T04:54:10.749 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.749 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.749 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:10.749 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.063301+0000", 2024-02-05T04:54:10.749 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0054983580000000001 2024-02-05T04:54:10.749 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.749 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.749 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:10.749 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.063348+0000", 2024-02-05T04:54:10.749 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.6412999999999998e-05 2024-02-05T04:54:10.749 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.749 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:10.749 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.749 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.749 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.749 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.16s2 74/67 ECSubWrite(tid=155, reqid=osd.11.0:155, at_version=74'34, trim_to=0'0, roll_forward_to=72'32))", 2024-02-05T04:54:10.750 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:54:06.060182+0000", 2024-02-05T04:54:10.750 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 4.6113281219999998, 2024-02-05T04:54:10.750 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0058106149999999999, 2024-02-05T04:54:10.750 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:10.750 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:54:10.750 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:10.750 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.750 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:10.750 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.060182+0000", 2024-02-05T04:54:10.750 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:10.750 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.750 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.750 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:10.750 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.060182+0000", 2024-02-05T04:54:10.750 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:10.751 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.751 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.751 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:10.751 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.060183+0000", 2024-02-05T04:54:10.751 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.5700000000000002e-07 2024-02-05T04:54:10.751 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.751 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.751 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:10.751 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.060211+0000", 2024-02-05T04:54:10.751 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.8282000000000001e-05 2024-02-05T04:54:10.751 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.751 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.751 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:10.751 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.060232+0000", 2024-02-05T04:54:10.751 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.0775e-05 2024-02-05T04:54:10.752 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.752 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.752 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:10.752 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.060241+0000", 2024-02-05T04:54:10.752 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.3600000000000002e-06 2024-02-05T04:54:10.752 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.752 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.752 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:10.752 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.060305+0000", 2024-02-05T04:54:10.752 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.4148999999999999e-05 2024-02-05T04:54:10.752 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.752 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.752 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:10.752 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.060316+0000", 2024-02-05T04:54:10.753 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.0360000000000001e-05 2024-02-05T04:54:10.753 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.753 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.753 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:10.753 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.065949+0000", 2024-02-05T04:54:10.753 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0056337160000000004 2024-02-05T04:54:10.753 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.753 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.753 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:10.753 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.065993+0000", 2024-02-05T04:54:10.753 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.3316000000000003e-05 2024-02-05T04:54:10.753 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.753 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:10.753 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.754 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.754 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.754 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.ds4 78/31 ECSubWrite(tid=94, reqid=osd.7.0:94, at_version=78'58, trim_to=0'0, roll_forward_to=77'56))", 2024-02-05T04:54:10.754 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:54:10.059344+0000", 2024-02-05T04:54:10.754 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 0.61216595900000004, 2024-02-05T04:54:10.754 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0057058389999999999, 2024-02-05T04:54:10.754 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:10.754 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:54:10.754 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:10.754 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.754 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:10.754 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:10.059344+0000", 2024-02-05T04:54:10.754 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:10.754 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.754 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.754 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:10.755 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:10.059344+0000", 2024-02-05T04:54:10.755 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:10.755 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.755 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.755 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:10.755 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:10.059345+0000", 2024-02-05T04:54:10.755 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.8100000000000002e-07 2024-02-05T04:54:10.755 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.755 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.755 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:10.755 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:10.059363+0000", 2024-02-05T04:54:10.755 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.8576e-05 2024-02-05T04:54:10.755 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.755 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.756 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:10.756 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:10.059377+0000", 2024-02-05T04:54:10.756 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.3275000000000001e-05 2024-02-05T04:54:10.756 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.756 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.756 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:10.756 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:10.059387+0000", 2024-02-05T04:54:10.756 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.0288999999999999e-05 2024-02-05T04:54:10.756 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.756 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.756 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:10.756 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:10.059446+0000", 2024-02-05T04:54:10.756 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.9468000000000001e-05 2024-02-05T04:54:10.756 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.756 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.757 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:10.757 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:10.059459+0000", 2024-02-05T04:54:10.757 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.2850999999999999e-05 2024-02-05T04:54:10.757 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.757 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.757 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:10.757 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:10.065017+0000", 2024-02-05T04:54:10.757 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0055576200000000001 2024-02-05T04:54:10.757 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:10.757 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:10.757 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:10.757 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:10.065050+0000", 2024-02-05T04:54:10.757 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.3079e-05 2024-02-05T04:54:10.757 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.757 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:10.757 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.758 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:10.758 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:10.758 INFO:teuthology.orchestra.run.smithi091.stdout:} 2024-02-05T04:54:10.759 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-02-05T04:54:10.760 INFO:tasks.ceph.osd.8.smithi062.stderr:2024-02-05T04:54:10.758+0000 7f9385bf2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 34693) UID: 0 2024-02-05T04:54:10.861 INFO:tasks.ceph.osd.15.smithi196.stderr:2024-02-05T04:54:10.860+0000 7f1facc45640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 34611) UID: 0 2024-02-05T04:54:10.864 INFO:teuthology.orchestra.run.smithi091.stdout:{ 2024-02-05T04:54:10.864 INFO:teuthology.orchestra.run.smithi091.stdout: "ops": [], 2024-02-05T04:54:10.864 INFO:teuthology.orchestra.run.smithi091.stdout: "num_ops": 0 2024-02-05T04:54:10.864 INFO:teuthology.orchestra.run.smithi091.stdout:} 2024-02-05T04:54:10.872 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-02-05T04:54:10.961 INFO:tasks.ceph.osd.13.smithi091.stderr:2024-02-05T04:54:10.961+0000 7f526fd4d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 34281) UID: 0 2024-02-05T04:54:10.964 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-02-05T04:54:10.977 INFO:teuthology.orchestra.run.smithi091.stdout:{ 2024-02-05T04:54:10.978 INFO:teuthology.orchestra.run.smithi091.stdout: "ops": [], 2024-02-05T04:54:10.978 INFO:teuthology.orchestra.run.smithi091.stdout: "complaint_time": 30, 2024-02-05T04:54:10.978 INFO:teuthology.orchestra.run.smithi091.stdout: "num_blocked_ops": 0 2024-02-05T04:54:10.978 INFO:teuthology.orchestra.run.smithi091.stdout:} 2024-02-05T04:54:10.986 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-02-05T04:54:11.055 INFO:teuthology.orchestra.run.smithi062.stderr:reweighted osd.14 to 1 (10000) 2024-02-05T04:54:11.059 INFO:tasks.rados.rados.0.smithi196.stdout:309: done (1 left) 2024-02-05T04:54:11.059 INFO:tasks.rados.rados.0.smithi196.stdout:311: done (0 left) 2024-02-05T04:54:11.059 INFO:tasks.rados.rados.0.smithi196.stdout:312: delete oid 9 current snap is 26 2024-02-05T04:54:11.068 INFO:tasks.rados.rados.0.smithi196.stdout:312: done (0 left) 2024-02-05T04:54:11.068 INFO:tasks.rados.rados.0.smithi196.stdout:append oid 15 current snap is 26 2024-02-05T04:54:11.068 INFO:tasks.rados.rados.0.smithi196.stdout:313: seq_num 135 ranges {1933312=720896,2654208=622592,3276800=688128,3964928=245760} 2024-02-05T04:54:11.070 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 15 1 2024-02-05T04:54:11.071 INFO:tasks.ceph.osd.5.smithi091.stderr:2024-02-05T04:54:11.071+0000 7f63983b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34277) UID: 0 2024-02-05T04:54:11.082 INFO:tasks.rados.rados.0.smithi196.stdout:313: writing smithi19639362-15 from 1933312 to 2654208 tid 1 2024-02-05T04:54:11.087 INFO:tasks.rados.rados.0.smithi196.stdout:313: writing smithi19639362-15 from 2654208 to 3276800 tid 2 2024-02-05T04:54:11.090 INFO:tasks.rados.rados.0.smithi196.stdout:313: writing smithi19639362-15 from 3276800 to 3964928 tid 3 2024-02-05T04:54:11.092 INFO:tasks.rados.rados.0.smithi196.stdout:313: writing smithi19639362-15 from 3964928 to 4210688 tid 4 2024-02-05T04:54:11.092 INFO:tasks.rados.rados.0.smithi196.stdout:314: copy_from oid 7 from oid 40 current snap is 26 2024-02-05T04:54:11.092 INFO:tasks.rados.rados.0.smithi196.stdout:313: finishing write tid 1 to smithi19639362-15 2024-02-05T04:54:11.092 INFO:tasks.rados.rados.0.smithi196.stdout:315: rollback oid 12 current snap is 26 2024-02-05T04:54:11.092 INFO:tasks.rados.rados.0.smithi196.stdout:rollback oid 12 to 6 2024-02-05T04:54:11.092 INFO:tasks.rados.rados.0.smithi196.stdout:316: delete oid 24 current snap is 26 2024-02-05T04:54:11.092 INFO:tasks.rados.rados.0.smithi196.stdout:313: finishing write tid 2 to smithi19639362-15 2024-02-05T04:54:11.093 INFO:tasks.rados.rados.0.smithi196.stdout:314: finishing copy_from to smithi19639362-7 2024-02-05T04:54:11.093 INFO:tasks.rados.rados.0.smithi196.stdout:314: got expected ENOENT (src dne) 2024-02-05T04:54:11.093 INFO:tasks.rados.rados.0.smithi196.stdout:314: finishing copy_from racing read to smithi19639362-7 2024-02-05T04:54:11.095 INFO:tasks.rados.rados.0.smithi196.stdout:313: finishing write tid 3 to smithi19639362-15 2024-02-05T04:54:11.096 INFO:tasks.rados.rados.0.smithi196.stdout:313: finishing write tid 4 to smithi19639362-15 2024-02-05T04:54:11.096 INFO:tasks.rados.rados.0.smithi196.stdout:315: finishing rollback tid 0 to smithi19639362-12 2024-02-05T04:54:11.097 INFO:tasks.rados.rados.0.smithi196.stdout:315: finishing rollback tid 1 to smithi19639362-12 2024-02-05T04:54:11.097 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 12 v 123 (ObjNum 49 snap 0 seq_num 3696379923) dirty dne 2024-02-05T04:54:11.097 INFO:tasks.rados.rados.0.smithi196.stdout:313: finishing write tid 5 to smithi19639362-15 2024-02-05T04:54:11.099 INFO:tasks.rados.rados.0.smithi196.stdout:314: done (3 left) 2024-02-05T04:54:11.099 INFO:tasks.rados.rados.0.smithi196.stdout:315: done (2 left) 2024-02-05T04:54:11.099 INFO:tasks.rados.rados.0.smithi196.stdout:316: done (1 left) 2024-02-05T04:54:11.099 INFO:tasks.rados.rados.0.smithi196.stdout:317: read oid 3 snap 6 2024-02-05T04:54:11.099 INFO:tasks.rados.rados.0.smithi196.stdout:317: expect deleted 2024-02-05T04:54:11.099 INFO:tasks.rados.rados.0.smithi196.stdout:318: read oid 45 snap -1 2024-02-05T04:54:11.100 INFO:tasks.rados.rados.0.smithi196.stdout:318: expect (ObjNum 89 snap 7 seq_num 89) 2024-02-05T04:54:11.100 INFO:tasks.rados.rados.0.smithi196.stdout:append oid 24 current snap is 26 2024-02-05T04:54:11.100 INFO:tasks.rados.rados.0.smithi196.stdout:319: seq_num 137 ranges {0=753664,753664=786432,1540096=655360,2195456=114688} 2024-02-05T04:54:11.103 INFO:tasks.rados.rados.0.smithi196.stdout:319: writing smithi19639362-24 from 0 to 753664 tid 1 2024-02-05T04:54:11.107 INFO:tasks.rados.rados.0.smithi196.stdout:319: writing smithi19639362-24 from 753664 to 1540096 tid 2 2024-02-05T04:54:11.111 INFO:tasks.rados.rados.0.smithi196.stdout:319: writing smithi19639362-24 from 1540096 to 2195456 tid 3 2024-02-05T04:54:11.112 INFO:tasks.rados.rados.0.smithi196.stdout:319: writing smithi19639362-24 from 2195456 to 2310144 tid 4 2024-02-05T04:54:11.112 INFO:tasks.rados.rados.0.smithi196.stdout:append oid 27 current snap is 26 2024-02-05T04:54:11.112 INFO:tasks.rados.rados.0.smithi196.stdout:313: finishing write tid 6 to smithi19639362-15 2024-02-05T04:54:11.112 INFO:tasks.rados.rados.0.smithi196.stdout:313: oid 15 updating version 0 to 53 2024-02-05T04:54:11.113 INFO:tasks.rados.rados.0.smithi196.stdout:313: oid 15 version 53 is already newer than 52 2024-02-05T04:54:11.113 INFO:tasks.rados.rados.0.smithi196.stdout:313: oid 15 updating version 53 to 54 2024-02-05T04:54:11.113 INFO:tasks.rados.rados.0.smithi196.stdout:313: oid 15 version 54 is already newer than 51 2024-02-05T04:54:11.113 INFO:tasks.rados.rados.0.smithi196.stdout:313: oid 15 updating version 54 to 55 2024-02-05T04:54:11.113 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 15 v 55 (ObjNum 135 snap 26 seq_num 135) dirty exists 2024-02-05T04:54:11.113 INFO:tasks.rados.rados.0.smithi196.stdout:313: left oid 15 (ObjNum 135 snap 26 seq_num 135) 2024-02-05T04:54:11.113 INFO:tasks.rados.rados.0.smithi196.stdout:319: finishing write tid 1 to smithi19639362-24 2024-02-05T04:54:11.113 INFO:tasks.rados.rados.0.smithi196.stdout:320: seq_num 138 ranges {3670016=770048,4440064=540672,4980736=753664,5734400=262144} 2024-02-05T04:54:11.135 INFO:tasks.rados.rados.0.smithi196.stdout:320: writing smithi19639362-27 from 3670016 to 4440064 tid 1 2024-02-05T04:54:11.139 INFO:tasks.rados.rados.0.smithi196.stdout:320: writing smithi19639362-27 from 4440064 to 4980736 tid 2 2024-02-05T04:54:11.143 INFO:tasks.rados.rados.0.smithi196.stdout:320: writing smithi19639362-27 from 4980736 to 5734400 tid 3 2024-02-05T04:54:11.145 INFO:tasks.rados.rados.0.smithi196.stdout:320: writing smithi19639362-27 from 5734400 to 5996544 tid 4 2024-02-05T04:54:11.146 INFO:tasks.rados.rados.0.smithi196.stdout:313: done (4 left) 2024-02-05T04:54:11.146 INFO:tasks.rados.rados.0.smithi196.stdout:317: done (3 left) 2024-02-05T04:54:11.146 INFO:tasks.rados.rados.0.smithi196.stdout:321: snap_create 2024-02-05T04:54:11.146 INFO:tasks.rados.rados.0.smithi196.stdout:319: finishing write tid 2 to smithi19639362-24 2024-02-05T04:54:11.146 INFO:tasks.rados.rados.0.smithi196.stdout:318: expect (ObjNum 89 snap 7 seq_num 89) 2024-02-05T04:54:11.150 INFO:teuthology.orchestra.run.smithi091.stdout:{ 2024-02-05T04:54:11.150 INFO:teuthology.orchestra.run.smithi091.stdout: "size": 20, 2024-02-05T04:54:11.150 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 600, 2024-02-05T04:54:11.150 INFO:teuthology.orchestra.run.smithi091.stdout: "ops": [ 2024-02-05T04:54:11.150 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.150 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:247 3.1s0 3:8451ec0c:::smithi19639362-8:head [copy-from ver 6] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:54:11.150 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:29.186841+0000", 2024-02-05T04:54:11.151 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 41.960435494000002, 2024-02-05T04:54:11.151 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.011720046, 2024-02-05T04:54:11.151 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:11.151 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:11.151 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:54:11.151 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:54:11.151 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:11.151 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 247 2024-02-05T04:54:11.151 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.151 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:11.151 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.151 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:11.151 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.186841+0000", 2024-02-05T04:54:11.151 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:11.151 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.152 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.152 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:11.152 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.186841+0000", 2024-02-05T04:54:11.152 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:11.152 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.152 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.152 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:11.152 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.186841+0000", 2024-02-05T04:54:11.152 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.0900000000000002e-07 2024-02-05T04:54:11.152 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.152 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.152 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:11.152 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.186844+0000", 2024-02-05T04:54:11.153 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.0970000000000002e-06 2024-02-05T04:54:11.153 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.153 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.153 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:11.153 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.186849+0000", 2024-02-05T04:54:11.153 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.9150000000000004e-06 2024-02-05T04:54:11.153 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.153 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.153 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:11.153 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.186853+0000", 2024-02-05T04:54:11.153 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.41e-06 2024-02-05T04:54:11.153 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.153 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.153 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:11.154 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.186906+0000", 2024-02-05T04:54:11.154 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.3505999999999999e-05 2024-02-05T04:54:11.154 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.154 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.154 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:54:11.154 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.187011+0000", 2024-02-05T04:54:11.154 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.000104478 2024-02-05T04:54:11.154 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.154 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.154 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:11.154 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.192835+0000", 2024-02-05T04:54:11.154 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0058247689999999996 2024-02-05T04:54:11.154 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.154 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.154 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:11.155 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.198154+0000", 2024-02-05T04:54:11.155 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0053189470000000001 2024-02-05T04:54:11.155 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.155 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.155 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "commit_sent", 2024-02-05T04:54:11.155 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.198508+0000", 2024-02-05T04:54:11.155 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.000353333 2024-02-05T04:54:11.155 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.155 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.155 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:11.155 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:29.198561+0000", 2024-02-05T04:54:11.155 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.3182e-05 2024-02-05T04:54:11.155 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:11.155 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:11.155 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:11.155 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.156 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.156 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:262 3.6s0 3:62c1182c:::smithi19639362-26:head [delete] snapc 3=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e38)", 2024-02-05T04:54:11.156 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:30.132144+0000", 2024-02-05T04:54:11.156 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 41.015131947999997, 2024-02-05T04:54:11.156 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.014886422999999999, 2024-02-05T04:54:11.156 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:11.156 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:11.156 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:54:11.156 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:54:11.156 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:11.156 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 262 2024-02-05T04:54:11.157 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.157 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:11.157 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.157 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:11.157 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.132144+0000", 2024-02-05T04:54:11.157 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:11.157 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.157 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.157 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:11.157 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.132144+0000", 2024-02-05T04:54:11.157 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:11.157 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.157 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.157 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:11.157 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.132146+0000", 2024-02-05T04:54:11.158 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.3990000000000001e-06 2024-02-05T04:54:11.158 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.158 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.158 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:11.158 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.132152+0000", 2024-02-05T04:54:11.158 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.2600000000000002e-06 2024-02-05T04:54:11.158 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.158 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.158 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:11.158 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.132166+0000", 2024-02-05T04:54:11.158 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.3526999999999999e-05 2024-02-05T04:54:11.158 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.158 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.158 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:11.158 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.132175+0000", 2024-02-05T04:54:11.158 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.8020000000000003e-06 2024-02-05T04:54:11.159 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.159 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.159 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:11.159 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.132249+0000", 2024-02-05T04:54:11.159 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.3250000000000005e-05 2024-02-05T04:54:11.159 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.159 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.159 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "op must wait for map", 2024-02-05T04:54:11.159 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.132256+0000", 2024-02-05T04:54:11.159 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.3749999999999997e-06 2024-02-05T04:54:11.159 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.159 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.159 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:11.159 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.139131+0000", 2024-02-05T04:54:11.159 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.006875038 2024-02-05T04:54:11.160 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.160 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.160 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:54:11.160 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.139221+0000", 2024-02-05T04:54:11.160 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.9709000000000005e-05 2024-02-05T04:54:11.160 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.160 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.160 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:11.160 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.140034+0000", 2024-02-05T04:54:11.160 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00081289899999999996 2024-02-05T04:54:11.160 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.160 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.160 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:11.161 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.142644+0000", 2024-02-05T04:54:11.161 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.002610008 2024-02-05T04:54:11.161 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.161 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.161 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "commit_sent", 2024-02-05T04:54:11.161 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.146991+0000", 2024-02-05T04:54:11.161 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0043471509999999996 2024-02-05T04:54:11.161 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.161 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.161 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:11.161 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.147031+0000", 2024-02-05T04:54:11.161 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.0005000000000001e-05 2024-02-05T04:54:11.161 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:11.161 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:11.161 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:11.161 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.162 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.162 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:379 3.1s0 3:86872077:::smithi19639362-21:head [copy-from ver 6] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-02-05T04:54:11.162 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:34.158922+0000", 2024-02-05T04:54:11.162 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 36.988353912000001, 2024-02-05T04:54:11.162 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.012016882, 2024-02-05T04:54:11.162 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:11.162 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:11.162 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:54:11.162 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:54:11.162 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:11.162 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 379 2024-02-05T04:54:11.162 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.162 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:11.162 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.162 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:11.163 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.158922+0000", 2024-02-05T04:54:11.163 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:11.163 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.163 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.163 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:11.163 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.158922+0000", 2024-02-05T04:54:11.163 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:11.163 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.163 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.163 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:11.163 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.158925+0000", 2024-02-05T04:54:11.163 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.3329999999999999e-06 2024-02-05T04:54:11.163 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.163 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.163 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:11.164 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.158938+0000", 2024-02-05T04:54:11.164 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.3373000000000001e-05 2024-02-05T04:54:11.164 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.164 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.164 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:11.164 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.158970+0000", 2024-02-05T04:54:11.164 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.1449000000000003e-05 2024-02-05T04:54:11.164 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.164 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.164 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:11.164 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.158994+0000", 2024-02-05T04:54:11.164 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.4539999999999999e-05 2024-02-05T04:54:11.164 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.164 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.165 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:11.165 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.159134+0000", 2024-02-05T04:54:11.165 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00013956 2024-02-05T04:54:11.165 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.165 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.165 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "op must wait for map", 2024-02-05T04:54:11.165 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.159147+0000", 2024-02-05T04:54:11.165 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.3206e-05 2024-02-05T04:54:11.165 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.165 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.165 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:11.165 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.161069+0000", 2024-02-05T04:54:11.165 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.001921835 2024-02-05T04:54:11.165 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.166 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.166 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:54:11.166 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.161158+0000", 2024-02-05T04:54:11.166 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.9604000000000005e-05 2024-02-05T04:54:11.166 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.166 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.166 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:11.166 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.167690+0000", 2024-02-05T04:54:11.166 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0065311830000000003 2024-02-05T04:54:11.166 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.166 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.166 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:11.166 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.169420+0000", 2024-02-05T04:54:11.166 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.001730325 2024-02-05T04:54:11.166 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.166 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.167 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "commit_sent", 2024-02-05T04:54:11.167 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.170848+0000", 2024-02-05T04:54:11.167 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.001428393 2024-02-05T04:54:11.167 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.167 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.167 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:11.167 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:34.170939+0000", 2024-02-05T04:54:11.167 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.1081000000000004e-05 2024-02-05T04:54:11.167 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:11.167 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:11.167 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:11.167 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.167 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.167 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:398 3.1s0 3:8451ec0c:::smithi19639362-8:head [read 0~737280,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-02-05T04:54:11.168 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:35.183088+0000", 2024-02-05T04:54:11.168 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 35.964188298000003, 2024-02-05T04:54:11.168 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.011852491, 2024-02-05T04:54:11.168 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:11.168 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "started", 2024-02-05T04:54:11.168 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:54:11.168 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:54:11.168 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:11.168 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 398 2024-02-05T04:54:11.168 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.168 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:11.168 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.168 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:11.168 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.183088+0000", 2024-02-05T04:54:11.168 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:11.169 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.169 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.169 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:11.169 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.183088+0000", 2024-02-05T04:54:11.169 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:11.169 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.169 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.169 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:11.169 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.183090+0000", 2024-02-05T04:54:11.169 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.511e-06 2024-02-05T04:54:11.169 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.169 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.169 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:11.169 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.183096+0000", 2024-02-05T04:54:11.169 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.3500000000000002e-06 2024-02-05T04:54:11.170 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.170 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.170 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:11.170 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.183108+0000", 2024-02-05T04:54:11.170 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.1722999999999999e-05 2024-02-05T04:54:11.170 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.170 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.170 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:11.170 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.183116+0000", 2024-02-05T04:54:11.170 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.1939999999999998e-06 2024-02-05T04:54:11.170 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.170 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.170 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:11.170 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.189528+0000", 2024-02-05T04:54:11.170 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0064124450000000001 2024-02-05T04:54:11.171 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.171 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.171 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "waiting for rw locks", 2024-02-05T04:54:11.171 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.189627+0000", 2024-02-05T04:54:11.171 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.8451999999999999e-05 2024-02-05T04:54:11.171 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.171 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.171 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:11.171 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.190600+0000", 2024-02-05T04:54:11.171 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00097288099999999998 2024-02-05T04:54:11.171 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.171 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.171 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:54:11.171 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.190679+0000", 2024-02-05T04:54:11.171 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.9692999999999997e-05 2024-02-05T04:54:11.172 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.172 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.172 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:11.172 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.194941+0000", 2024-02-05T04:54:11.172 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0042612420000000002 2024-02-05T04:54:11.172 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:11.172 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:11.172 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:11.172 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.172 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.172 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:399 3.1s0 3:8451ec0c:::smithi19639362-8:head [read 0~737280,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-02-05T04:54:11.172 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:35.183130+0000", 2024-02-05T04:54:11.172 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 35.964146601000003, 2024-02-05T04:54:11.172 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.013111441, 2024-02-05T04:54:11.172 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:11.173 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "started", 2024-02-05T04:54:11.173 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:54:11.173 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:54:11.173 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:11.173 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 399 2024-02-05T04:54:11.173 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.173 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:11.173 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.173 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:11.173 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.183130+0000", 2024-02-05T04:54:11.173 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:11.173 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.173 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.173 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:11.173 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.183130+0000", 2024-02-05T04:54:11.173 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:11.174 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.174 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.174 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:11.174 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.183130+0000", 2024-02-05T04:54:11.174 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.37e-07 2024-02-05T04:54:11.174 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.174 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.174 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:11.174 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.183134+0000", 2024-02-05T04:54:11.174 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.189e-06 2024-02-05T04:54:11.174 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.174 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.174 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:11.174 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.183141+0000", 2024-02-05T04:54:11.174 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.8000000000000001e-06 2024-02-05T04:54:11.175 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.175 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.175 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:11.175 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.183181+0000", 2024-02-05T04:54:11.175 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.9864000000000003e-05 2024-02-05T04:54:11.175 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.175 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.175 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:11.175 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.190123+0000", 2024-02-05T04:54:11.175 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0069416510000000001 2024-02-05T04:54:11.175 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.175 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.175 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "waiting for rw locks", 2024-02-05T04:54:11.175 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.190209+0000", 2024-02-05T04:54:11.175 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.6447999999999996e-05 2024-02-05T04:54:11.176 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.176 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.176 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:11.176 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.191024+0000", 2024-02-05T04:54:11.176 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00081457299999999997 2024-02-05T04:54:11.176 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.176 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.176 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:54:11.176 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.191074+0000", 2024-02-05T04:54:11.176 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.0464999999999999e-05 2024-02-05T04:54:11.176 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.176 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.176 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:11.176 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.196241+0000", 2024-02-05T04:54:11.176 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0051669139999999999 2024-02-05T04:54:11.177 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:11.177 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:11.177 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:11.177 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.177 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.177 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:400 3.1s0 3:8451ec0c:::smithi19639362-8:head [read 0~737280,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-02-05T04:54:11.177 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:35.183194+0000", 2024-02-05T04:54:11.177 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 35.964082368, 2024-02-05T04:54:11.177 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.013579028, 2024-02-05T04:54:11.177 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:11.177 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "started", 2024-02-05T04:54:11.177 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:54:11.177 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:54:11.177 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:11.177 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 400 2024-02-05T04:54:11.177 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.178 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:11.178 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.178 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:11.178 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.183194+0000", 2024-02-05T04:54:11.178 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:11.178 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.178 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.178 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:11.178 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.183194+0000", 2024-02-05T04:54:11.178 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:11.178 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.178 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.178 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:11.178 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.183195+0000", 2024-02-05T04:54:11.178 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.0299999999999999e-07 2024-02-05T04:54:11.179 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.179 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.179 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:11.179 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.183198+0000", 2024-02-05T04:54:11.179 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.1990000000000002e-06 2024-02-05T04:54:11.179 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.179 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.179 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:11.179 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.183205+0000", 2024-02-05T04:54:11.179 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.8789999999999997e-06 2024-02-05T04:54:11.179 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.179 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.179 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:11.179 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.183212+0000", 2024-02-05T04:54:11.179 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.2350000000000002e-06 2024-02-05T04:54:11.180 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.180 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.180 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:11.180 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.190255+0000", 2024-02-05T04:54:11.180 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.007042853 2024-02-05T04:54:11.180 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.180 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.180 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "waiting for rw locks", 2024-02-05T04:54:11.180 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.190345+0000", 2024-02-05T04:54:11.180 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.0160000000000004e-05 2024-02-05T04:54:11.180 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.180 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.180 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:11.180 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.193877+0000", 2024-02-05T04:54:11.181 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0035322410000000002 2024-02-05T04:54:11.181 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.181 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.181 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:54:11.181 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.193962+0000", 2024-02-05T04:54:11.181 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.4635000000000005e-05 2024-02-05T04:54:11.181 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.181 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.181 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:11.181 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:35.196773+0000", 2024-02-05T04:54:11.181 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0028112229999999998 2024-02-05T04:54:11.181 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:11.181 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:11.181 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:11.181 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.181 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.182 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:426 3.6s0 3:62c1182c:::smithi19639362-26:head [read 0~2572288,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-02-05T04:54:11.182 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:40.951069+0000", 2024-02-05T04:54:11.182 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 30.196207028, 2024-02-05T04:54:11.182 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.010366143, 2024-02-05T04:54:11.182 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:11.182 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "started", 2024-02-05T04:54:11.182 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:54:11.182 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:54:11.182 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:11.182 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 426 2024-02-05T04:54:11.182 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.182 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:11.182 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.182 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:11.182 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.951069+0000", 2024-02-05T04:54:11.182 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:11.183 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.183 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.183 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:11.183 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.951069+0000", 2024-02-05T04:54:11.183 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:11.183 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.183 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.183 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:11.183 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.951070+0000", 2024-02-05T04:54:11.183 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.7000000000000004e-07 2024-02-05T04:54:11.183 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.183 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.183 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:11.183 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.951073+0000", 2024-02-05T04:54:11.183 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.467e-06 2024-02-05T04:54:11.184 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.184 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.184 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:11.184 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.951086+0000", 2024-02-05T04:54:11.184 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.2999999999999999e-05 2024-02-05T04:54:11.184 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.184 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.184 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:11.184 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.951094+0000", 2024-02-05T04:54:11.184 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.8979999999999996e-06 2024-02-05T04:54:11.184 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.184 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.184 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:11.184 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.951184+0000", 2024-02-05T04:54:11.184 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.9809999999999996e-05 2024-02-05T04:54:11.185 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.185 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.185 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "waiting_for_map not empty", 2024-02-05T04:54:11.185 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.951188+0000", 2024-02-05T04:54:11.185 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.3189999999999996e-06 2024-02-05T04:54:11.185 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.185 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.185 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:11.185 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.953531+0000", 2024-02-05T04:54:11.185 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0023427140000000001 2024-02-05T04:54:11.185 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.185 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.185 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:54:11.185 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.953591+0000", 2024-02-05T04:54:11.185 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.9707000000000003e-05 2024-02-05T04:54:11.185 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.186 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.186 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:11.186 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.961435+0000", 2024-02-05T04:54:11.186 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0078445579999999997 2024-02-05T04:54:11.186 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:11.186 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:11.186 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:11.186 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.186 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.186 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:427 3.6s0 3:62c1182c:::smithi19639362-26:head [read 0~2572288,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-02-05T04:54:11.186 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:40.951117+0000", 2024-02-05T04:54:11.186 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 30.196159286, 2024-02-05T04:54:11.186 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.012236415, 2024-02-05T04:54:11.186 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:11.186 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "started", 2024-02-05T04:54:11.187 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:54:11.187 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:54:11.187 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:11.187 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 427 2024-02-05T04:54:11.187 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.187 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:11.187 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.187 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:11.187 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.951117+0000", 2024-02-05T04:54:11.187 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:11.187 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.187 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.187 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:11.187 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.951117+0000", 2024-02-05T04:54:11.187 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:11.187 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.188 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.188 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:11.188 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.951118+0000", 2024-02-05T04:54:11.188 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.7199999999999998e-07 2024-02-05T04:54:11.188 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.188 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.188 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:11.188 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.951122+0000", 2024-02-05T04:54:11.188 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.2540000000000003e-06 2024-02-05T04:54:11.188 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.188 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.188 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:11.188 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.951130+0000", 2024-02-05T04:54:11.188 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.3760000000000008e-06 2024-02-05T04:54:11.188 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.189 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.189 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:11.189 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.951137+0000", 2024-02-05T04:54:11.189 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.3430000000000001e-06 2024-02-05T04:54:11.189 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.189 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.189 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:11.189 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.951207+0000", 2024-02-05T04:54:11.189 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.9974999999999993e-05 2024-02-05T04:54:11.189 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.189 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.189 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "waiting_for_map not empty", 2024-02-05T04:54:11.189 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.951210+0000", 2024-02-05T04:54:11.189 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.4589999999999998e-06 2024-02-05T04:54:11.189 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.190 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.190 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:11.190 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.955115+0000", 2024-02-05T04:54:11.190 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.003904687 2024-02-05T04:54:11.190 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.190 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.190 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:54:11.190 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.955169+0000", 2024-02-05T04:54:11.190 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.3823999999999998e-05 2024-02-05T04:54:11.190 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.190 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.190 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:11.190 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.963353+0000", 2024-02-05T04:54:11.190 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0081848249999999997 2024-02-05T04:54:11.190 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:11.191 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:11.191 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:11.191 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.191 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.191 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:431 3.bs0 3:de65491d:::smithi19639362-2:head [setxattr _header (56) in=7b] snapc b=[b,a,9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-02-05T04:54:11.191 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:40.969591+0000", 2024-02-05T04:54:11.191 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 30.177684971000001, 2024-02-05T04:54:11.191 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.011310982000000001, 2024-02-05T04:54:11.191 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:11.191 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:11.191 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:54:11.191 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:54:11.191 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:11.191 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 431 2024-02-05T04:54:11.191 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.192 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:11.192 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.192 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:11.192 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.969591+0000", 2024-02-05T04:54:11.192 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:11.192 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.192 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.192 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:11.192 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.969591+0000", 2024-02-05T04:54:11.192 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:11.192 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.192 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.192 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:11.192 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.969592+0000", 2024-02-05T04:54:11.192 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.75e-07 2024-02-05T04:54:11.193 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.193 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.193 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:11.193 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.969596+0000", 2024-02-05T04:54:11.193 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.0550000000000001e-06 2024-02-05T04:54:11.193 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.193 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.193 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:11.193 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.969605+0000", 2024-02-05T04:54:11.193 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.0520000000000007e-06 2024-02-05T04:54:11.193 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.193 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.193 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:11.193 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.969611+0000", 2024-02-05T04:54:11.193 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.2990000000000002e-06 2024-02-05T04:54:11.194 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.194 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.194 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:11.194 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.972667+0000", 2024-02-05T04:54:11.194 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0030555030000000002 2024-02-05T04:54:11.194 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.194 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.194 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:54:11.194 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.972758+0000", 2024-02-05T04:54:11.194 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.0587999999999994e-05 2024-02-05T04:54:11.194 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.194 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.194 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:11.194 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.973132+0000", 2024-02-05T04:54:11.194 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00037420900000000002 2024-02-05T04:54:11.194 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.195 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.195 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:11.195 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.980593+0000", 2024-02-05T04:54:11.195 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0074614390000000003 2024-02-05T04:54:11.195 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.195 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.195 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "commit_sent", 2024-02-05T04:54:11.195 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.980636+0000", 2024-02-05T04:54:11.195 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.3025999999999998e-05 2024-02-05T04:54:11.195 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.195 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.195 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:11.195 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.980902+0000", 2024-02-05T04:54:11.195 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00026613599999999999 2024-02-05T04:54:11.195 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:11.196 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:11.196 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:11.196 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.196 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.196 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:432 3.bs0 3:de65491d:::smithi19639362-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-02-05T04:54:11.196 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:40.969620+0000", 2024-02-05T04:54:11.196 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 30.177656171999999, 2024-02-05T04:54:11.196 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.012177466, 2024-02-05T04:54:11.196 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:11.196 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "started", 2024-02-05T04:54:11.196 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:54:11.196 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:54:11.196 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:11.196 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 432 2024-02-05T04:54:11.197 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.197 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:11.197 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.197 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:11.197 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.969620+0000", 2024-02-05T04:54:11.197 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:11.197 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.197 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.197 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:11.197 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.969620+0000", 2024-02-05T04:54:11.197 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:11.197 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.197 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.197 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:11.197 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.969621+0000", 2024-02-05T04:54:11.198 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.75e-07 2024-02-05T04:54:11.198 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.198 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.198 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:11.198 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.969623+0000", 2024-02-05T04:54:11.198 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.7460000000000001e-06 2024-02-05T04:54:11.198 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.198 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.198 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:11.198 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.969633+0000", 2024-02-05T04:54:11.198 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.1209999999999996e-06 2024-02-05T04:54:11.198 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.198 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.198 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:11.198 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.969639+0000", 2024-02-05T04:54:11.199 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.0000000000000002e-06 2024-02-05T04:54:11.199 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.199 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.199 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:11.199 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.973551+0000", 2024-02-05T04:54:11.199 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0039128829999999998 2024-02-05T04:54:11.199 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.199 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.199 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "waiting for rw locks", 2024-02-05T04:54:11.199 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.973609+0000", 2024-02-05T04:54:11.199 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.7367000000000001e-05 2024-02-05T04:54:11.199 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.199 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.199 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:11.199 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.980931+0000", 2024-02-05T04:54:11.200 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.007322131 2024-02-05T04:54:11.200 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.200 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.200 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:54:11.200 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.980979+0000", 2024-02-05T04:54:11.200 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.8288000000000003e-05 2024-02-05T04:54:11.200 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.200 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.200 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:11.200 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:40.981798+0000", 2024-02-05T04:54:11.200 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00081845500000000005 2024-02-05T04:54:11.200 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:11.200 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:11.200 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:11.200 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.201 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.201 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:621 3.1s0 3:86872077:::smithi19639362-21:head [append 0~0] snapc 16=[16,15,13,12,11,f,b,8,7] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-02-05T04:54:11.201 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:59.950189+0000", 2024-02-05T04:54:11.201 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 11.197087752, 2024-02-05T04:54:11.201 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.013218306000000001, 2024-02-05T04:54:11.201 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:11.201 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:11.201 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:54:11.201 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:54:11.201 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:11.201 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 621 2024-02-05T04:54:11.201 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.201 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:11.201 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.201 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:11.202 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.950189+0000", 2024-02-05T04:54:11.202 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:11.202 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.202 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.202 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:11.202 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.950189+0000", 2024-02-05T04:54:11.202 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:11.202 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.202 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.202 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:11.202 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.950191+0000", 2024-02-05T04:54:11.202 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.497e-06 2024-02-05T04:54:11.202 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.202 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.202 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:11.203 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.950199+0000", 2024-02-05T04:54:11.203 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.977e-06 2024-02-05T04:54:11.203 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.203 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.203 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:11.203 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.950220+0000", 2024-02-05T04:54:11.203 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.128e-05 2024-02-05T04:54:11.203 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.203 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.203 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:11.203 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.950232+0000", 2024-02-05T04:54:11.203 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.1782e-05 2024-02-05T04:54:11.203 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.203 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.203 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:11.204 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.950314+0000", 2024-02-05T04:54:11.204 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.1625000000000005e-05 2024-02-05T04:54:11.204 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.204 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.204 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "op must wait for map", 2024-02-05T04:54:11.204 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.950324+0000", 2024-02-05T04:54:11.204 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.0138e-05 2024-02-05T04:54:11.204 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.204 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.204 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:11.204 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.954835+0000", 2024-02-05T04:54:11.204 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0045107890000000003 2024-02-05T04:54:11.204 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.204 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.204 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:54:11.205 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.954967+0000", 2024-02-05T04:54:11.205 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.000132621 2024-02-05T04:54:11.205 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.205 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.205 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:11.205 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.955919+0000", 2024-02-05T04:54:11.205 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00095165400000000002 2024-02-05T04:54:11.205 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.205 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.205 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:11.205 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.959653+0000", 2024-02-05T04:54:11.205 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0037343670000000002 2024-02-05T04:54:11.205 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.205 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.205 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "commit_sent", 2024-02-05T04:54:11.206 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.963323+0000", 2024-02-05T04:54:11.206 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0036693590000000001 2024-02-05T04:54:11.206 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.206 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.206 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:11.206 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.963407+0000", 2024-02-05T04:54:11.206 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.4216999999999997e-05 2024-02-05T04:54:11.206 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:11.206 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:11.206 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:11.206 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.206 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.206 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:622 3.1s0 3:86872077:::smithi19639362-21:head [rollback 11] snapc 16=[16,15,13,12,11,f,b,8,7] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-02-05T04:54:11.206 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:59.950265+0000", 2024-02-05T04:54:11.206 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 11.197011680999999, 2024-02-05T04:54:11.207 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.015715145999999999, 2024-02-05T04:54:11.207 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:11.207 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:11.207 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:54:11.207 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:54:11.207 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:11.207 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 622 2024-02-05T04:54:11.207 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.207 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:11.207 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.207 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:11.207 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.950265+0000", 2024-02-05T04:54:11.207 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:11.207 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.207 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.208 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:11.208 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.950265+0000", 2024-02-05T04:54:11.208 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:11.208 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.208 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.208 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:11.208 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.950265+0000", 2024-02-05T04:54:11.208 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.4300000000000002e-07 2024-02-05T04:54:11.208 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.208 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.208 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:11.208 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.950269+0000", 2024-02-05T04:54:11.208 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.9020000000000001e-06 2024-02-05T04:54:11.208 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.208 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.209 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:11.209 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.950280+0000", 2024-02-05T04:54:11.209 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.099e-05 2024-02-05T04:54:11.209 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.209 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.209 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:11.209 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.950288+0000", 2024-02-05T04:54:11.209 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.3579999999999997e-06 2024-02-05T04:54:11.209 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.209 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.209 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:11.209 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.950395+0000", 2024-02-05T04:54:11.209 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.000106946 2024-02-05T04:54:11.209 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.209 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.210 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "waiting_for_map not empty", 2024-02-05T04:54:11.210 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.950401+0000", 2024-02-05T04:54:11.210 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.7299999999999999e-06 2024-02-05T04:54:11.210 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.210 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.210 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:11.210 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.956948+0000", 2024-02-05T04:54:11.210 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0065471230000000002 2024-02-05T04:54:11.210 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.210 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.210 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:54:11.210 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.957068+0000", 2024-02-05T04:54:11.210 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.000119932 2024-02-05T04:54:11.210 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.210 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.211 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:11.211 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.957765+0000", 2024-02-05T04:54:11.211 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00069669000000000003 2024-02-05T04:54:11.211 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.211 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.211 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:11.211 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.959658+0000", 2024-02-05T04:54:11.211 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0018925350000000001 2024-02-05T04:54:11.211 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.211 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.211 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "commit_sent", 2024-02-05T04:54:11.211 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.965902+0000", 2024-02-05T04:54:11.211 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.006244734 2024-02-05T04:54:11.211 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.211 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.212 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:11.212 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.965980+0000", 2024-02-05T04:54:11.212 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.7463e-05 2024-02-05T04:54:11.212 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:11.212 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:11.212 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:11.212 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.212 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.212 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:623 3.1s0 3:86872077:::smithi19639362-21:head [append 0~0] snapc 16=[16,15,13,12,11,f,b,8,7] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-02-05T04:54:11.212 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:59.950299+0000", 2024-02-05T04:54:11.212 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 11.196977006000001, 2024-02-05T04:54:11.212 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.015901859000000001, 2024-02-05T04:54:11.212 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:11.212 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:11.213 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:54:11.213 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:54:11.213 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:11.213 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 623 2024-02-05T04:54:11.213 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.213 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:11.213 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.213 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:11.213 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.950299+0000", 2024-02-05T04:54:11.213 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:11.213 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.213 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.213 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:11.213 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.950299+0000", 2024-02-05T04:54:11.213 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:11.213 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.214 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.214 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:11.214 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.950300+0000", 2024-02-05T04:54:11.214 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.1600000000000001e-07 2024-02-05T04:54:11.214 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.214 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.214 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:11.214 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.950303+0000", 2024-02-05T04:54:11.214 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.0429999999999999e-06 2024-02-05T04:54:11.214 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.214 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.214 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:11.214 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.950311+0000", 2024-02-05T04:54:11.214 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.5310000000000001e-06 2024-02-05T04:54:11.214 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.214 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.215 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:11.215 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.950318+0000", 2024-02-05T04:54:11.215 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.4869999999999999e-06 2024-02-05T04:54:11.215 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.215 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.215 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:11.215 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.950433+0000", 2024-02-05T04:54:11.215 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.000115085 2024-02-05T04:54:11.215 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.215 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.215 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "waiting_for_map not empty", 2024-02-05T04:54:11.215 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.950440+0000", 2024-02-05T04:54:11.215 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.8120000000000002e-06 2024-02-05T04:54:11.215 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.215 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.216 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:11.216 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.958383+0000", 2024-02-05T04:54:11.216 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0079435619999999995 2024-02-05T04:54:11.216 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.216 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.216 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:54:11.216 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.958504+0000", 2024-02-05T04:54:11.216 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00012047700000000001 2024-02-05T04:54:11.216 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.216 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.216 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:11.216 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.959000+0000", 2024-02-05T04:54:11.216 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00049647500000000002 2024-02-05T04:54:11.216 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.216 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.217 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:11.217 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.963203+0000", 2024-02-05T04:54:11.217 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0042025439999999999 2024-02-05T04:54:11.217 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.217 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.217 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "commit_sent", 2024-02-05T04:54:11.217 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.966140+0000", 2024-02-05T04:54:11.217 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0029367159999999998 2024-02-05T04:54:11.217 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.217 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.217 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:11.217 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:59.966201+0000", 2024-02-05T04:54:11.217 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.1611000000000004e-05 2024-02-05T04:54:11.217 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:11.217 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:11.217 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:11.218 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.218 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.218 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:685 3.bs0 3:db6fe427:::smithi19639362-43:head [read 0~2064384,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e73)", 2024-02-05T04:54:11.218 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:54:05.031386+0000", 2024-02-05T04:54:11.218 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 6.1158901710000002, 2024-02-05T04:54:11.218 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.010866103, 2024-02-05T04:54:11.218 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:11.218 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "started", 2024-02-05T04:54:11.218 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:54:11.218 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:54:11.218 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:11.218 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 685 2024-02-05T04:54:11.218 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.218 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:11.219 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.219 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:11.219 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:05.031386+0000", 2024-02-05T04:54:11.219 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:11.219 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.219 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.219 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:11.219 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:05.031386+0000", 2024-02-05T04:54:11.219 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:11.219 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.219 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.219 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:11.219 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:05.031387+0000", 2024-02-05T04:54:11.219 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.1999999999999999e-07 2024-02-05T04:54:11.219 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.220 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.220 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:11.220 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:05.031392+0000", 2024-02-05T04:54:11.220 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.7829999999999998e-06 2024-02-05T04:54:11.220 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.220 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.220 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:11.220 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:05.031402+0000", 2024-02-05T04:54:11.220 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.0803e-05 2024-02-05T04:54:11.220 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.220 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.220 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:11.220 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:05.031410+0000", 2024-02-05T04:54:11.220 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.4900000000000003e-06 2024-02-05T04:54:11.221 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.221 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.221 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:11.221 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:05.031560+0000", 2024-02-05T04:54:11.221 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.000149827 2024-02-05T04:54:11.221 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.221 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.221 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "waiting_for_map not empty", 2024-02-05T04:54:11.221 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:05.031563+0000", 2024-02-05T04:54:11.221 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.1540000000000002e-06 2024-02-05T04:54:11.221 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.221 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.221 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:11.221 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:05.035201+0000", 2024-02-05T04:54:11.221 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0036383230000000002 2024-02-05T04:54:11.222 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.222 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.222 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:54:11.222 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:05.035290+0000", 2024-02-05T04:54:11.222 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.9145000000000001e-05 2024-02-05T04:54:11.222 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.222 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.222 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:11.222 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:05.042252+0000", 2024-02-05T04:54:11.222 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0069618579999999996 2024-02-05T04:54:11.222 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:11.222 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:11.222 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:11.222 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.222 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.223 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:687 3.bs0 3:db93bf34:::smithi19639362-24:head [append 0~524288] snapc 18=[18,17,16,15,12,8,7] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-02-05T04:54:11.223 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:54:06.052207+0000", 2024-02-05T04:54:11.223 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 5.0950692980000003, 2024-02-05T04:54:11.223 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.017565661, 2024-02-05T04:54:11.223 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:11.223 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:11.223 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:54:11.223 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:54:11.223 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:11.223 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 687 2024-02-05T04:54:11.223 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.223 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:11.223 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.223 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:11.223 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.052207+0000", 2024-02-05T04:54:11.224 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:11.224 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.224 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.224 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:11.224 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.052207+0000", 2024-02-05T04:54:11.224 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:11.224 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.224 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.224 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:11.224 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.052209+0000", 2024-02-05T04:54:11.224 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.1129999999999999e-06 2024-02-05T04:54:11.224 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.224 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.224 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:11.224 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.052966+0000", 2024-02-05T04:54:11.225 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00075682899999999999 2024-02-05T04:54:11.225 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.225 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.225 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:11.225 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.052979+0000", 2024-02-05T04:54:11.225 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.2860999999999999e-05 2024-02-05T04:54:11.225 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.225 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.225 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:11.225 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.052988+0000", 2024-02-05T04:54:11.225 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.7490000000000004e-06 2024-02-05T04:54:11.225 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.225 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.225 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:11.225 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.053045+0000", 2024-02-05T04:54:11.226 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.7701999999999997e-05 2024-02-05T04:54:11.226 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.226 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.226 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:54:11.226 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.053126+0000", 2024-02-05T04:54:11.226 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.0715000000000002e-05 2024-02-05T04:54:11.226 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.226 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.226 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:11.226 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.054380+0000", 2024-02-05T04:54:11.226 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0012539109999999999 2024-02-05T04:54:11.226 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.226 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.226 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:11.226 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.066987+0000", 2024-02-05T04:54:11.226 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.012607317999999999 2024-02-05T04:54:11.227 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.227 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.227 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "commit_sent", 2024-02-05T04:54:11.227 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.069654+0000", 2024-02-05T04:54:11.227 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0026664060000000001 2024-02-05T04:54:11.227 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.227 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.227 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:11.227 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.069773+0000", 2024-02-05T04:54:11.227 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.000119057 2024-02-05T04:54:11.227 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:11.227 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:11.227 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:11.227 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.227 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.228 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:688 3.bs0 3:db93bf34:::smithi19639362-24:head [append 0~573440] snapc 18=[18,17,16,15,12,8,7] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-02-05T04:54:11.228 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:54:06.055734+0000", 2024-02-05T04:54:11.228 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 5.0915426630000002, 2024-02-05T04:54:11.228 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.014228612999999999, 2024-02-05T04:54:11.228 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:11.228 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:11.228 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:54:11.228 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:54:11.228 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:11.228 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 688 2024-02-05T04:54:11.228 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.228 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:11.228 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.228 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:11.228 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.055734+0000", 2024-02-05T04:54:11.229 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:11.229 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.229 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.229 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:11.229 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.055734+0000", 2024-02-05T04:54:11.229 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:11.229 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.229 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.229 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:11.229 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.055735+0000", 2024-02-05T04:54:11.229 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.263e-06 2024-02-05T04:54:11.229 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.229 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.229 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:11.230 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.056321+0000", 2024-02-05T04:54:11.230 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00058652300000000004 2024-02-05T04:54:11.230 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.230 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.230 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:11.230 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.056334+0000", 2024-02-05T04:54:11.230 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.2668e-05 2024-02-05T04:54:11.230 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.230 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.230 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:11.230 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.056340+0000", 2024-02-05T04:54:11.230 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.5929999999999996e-06 2024-02-05T04:54:11.230 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.230 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.230 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:11.231 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.060150+0000", 2024-02-05T04:54:11.231 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0038099319999999998 2024-02-05T04:54:11.231 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.231 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.231 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:54:11.231 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.060267+0000", 2024-02-05T04:54:11.231 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.000117075 2024-02-05T04:54:11.231 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.231 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.231 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:11.231 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.061190+0000", 2024-02-05T04:54:11.231 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00092279500000000002 2024-02-05T04:54:11.231 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.231 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.231 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:11.232 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.069311+0000", 2024-02-05T04:54:11.232 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0081217560000000008 2024-02-05T04:54:11.232 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.232 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.232 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "commit_sent", 2024-02-05T04:54:11.232 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.069892+0000", 2024-02-05T04:54:11.232 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.000581198 2024-02-05T04:54:11.232 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.232 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.232 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:11.232 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.069962+0000", 2024-02-05T04:54:11.232 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.9809999999999997e-05 2024-02-05T04:54:11.232 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:11.232 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:11.232 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:11.233 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.233 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.233 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.5s1 74/62 ECSubWrite(tid=80, reqid=osd.15.0:80, at_version=74'61, trim_to=0'0, roll_forward_to=72'57))", 2024-02-05T04:54:11.233 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:54:06.058214+0000", 2024-02-05T04:54:11.233 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 5.0890624969999996, 2024-02-05T04:54:11.233 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.010344099000000001, 2024-02-05T04:54:11.233 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:11.233 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:54:11.233 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:11.233 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.233 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:11.233 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.058214+0000", 2024-02-05T04:54:11.233 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:11.233 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.233 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.234 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:11.234 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.058214+0000", 2024-02-05T04:54:11.234 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:11.234 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.234 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.234 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:11.234 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.058215+0000", 2024-02-05T04:54:11.234 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.9500000000000001e-07 2024-02-05T04:54:11.234 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.234 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.234 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:11.234 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.058234+0000", 2024-02-05T04:54:11.234 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.9653999999999999e-05 2024-02-05T04:54:11.234 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.234 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.234 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:11.235 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.058248+0000", 2024-02-05T04:54:11.235 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.4144999999999999e-05 2024-02-05T04:54:11.235 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.235 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.235 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:11.235 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.058257+0000", 2024-02-05T04:54:11.235 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.7579999999999998e-06 2024-02-05T04:54:11.235 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.235 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.235 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:11.235 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.058313+0000", 2024-02-05T04:54:11.235 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.5584000000000002e-05 2024-02-05T04:54:11.235 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.235 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.235 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:11.236 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.058323+0000", 2024-02-05T04:54:11.236 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.0249e-05 2024-02-05T04:54:11.236 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.236 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.236 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:11.236 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.068521+0000", 2024-02-05T04:54:11.236 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.01019804 2024-02-05T04:54:11.236 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.236 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.236 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:11.236 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.068558+0000", 2024-02-05T04:54:11.236 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.6874e-05 2024-02-05T04:54:11.236 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:11.236 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:11.237 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:11.237 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.237 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.237 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:689 3.bs0 3:db93bf34:::smithi19639362-24:head [append 0~688128] snapc 18=[18,17,16,15,12,8,7] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-02-05T04:54:11.237 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:54:06.060576+0000", 2024-02-05T04:54:11.237 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 5.0867007869999998, 2024-02-05T04:54:11.237 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.011205849, 2024-02-05T04:54:11.237 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:11.237 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:11.237 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:54:11.237 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:54:11.237 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:11.237 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 689 2024-02-05T04:54:11.237 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.237 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:11.238 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.238 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:11.238 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.060576+0000", 2024-02-05T04:54:11.238 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:11.238 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.238 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.238 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:11.238 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.060576+0000", 2024-02-05T04:54:11.238 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:11.238 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.238 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.238 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:11.238 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.060577+0000", 2024-02-05T04:54:11.238 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.3549999999999999e-06 2024-02-05T04:54:11.238 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.239 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.239 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:11.239 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.061414+0000", 2024-02-05T04:54:11.239 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00083751500000000005 2024-02-05T04:54:11.239 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.239 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.239 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:11.239 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.061426+0000", 2024-02-05T04:54:11.239 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.1737e-05 2024-02-05T04:54:11.239 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.239 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.239 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:11.239 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.061432+0000", 2024-02-05T04:54:11.239 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.8799999999999996e-06 2024-02-05T04:54:11.239 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.239 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.240 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:11.240 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.062764+0000", 2024-02-05T04:54:11.240 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0013320960000000001 2024-02-05T04:54:11.240 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.240 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.240 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:54:11.240 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.062889+0000", 2024-02-05T04:54:11.240 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00012482199999999999 2024-02-05T04:54:11.240 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.240 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.240 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:11.240 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.063953+0000", 2024-02-05T04:54:11.240 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0010644199999999999 2024-02-05T04:54:11.240 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.241 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.241 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:11.241 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.069313+0000", 2024-02-05T04:54:11.241 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0053592379999999997 2024-02-05T04:54:11.241 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.241 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.241 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "commit_sent", 2024-02-05T04:54:11.241 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.071691+0000", 2024-02-05T04:54:11.241 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00237864 2024-02-05T04:54:11.241 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.241 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.241 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:11.241 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.071781+0000", 2024-02-05T04:54:11.241 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.0146000000000001e-05 2024-02-05T04:54:11.242 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:11.242 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:11.242 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:11.242 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.242 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.242 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:692 3.bs0 3:db93bf34:::smithi19639362-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e74)", 2024-02-05T04:54:11.242 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:54:06.062454+0000", 2024-02-05T04:54:11.242 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 5.084821904, 2024-02-05T04:54:11.242 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.013153322, 2024-02-05T04:54:11.242 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:11.242 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "started", 2024-02-05T04:54:11.242 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:54:11.242 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:54:11.242 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:11.242 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 692 2024-02-05T04:54:11.242 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.243 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:11.243 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.243 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:11.243 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.062454+0000", 2024-02-05T04:54:11.243 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:11.243 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.243 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.243 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:11.243 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.062454+0000", 2024-02-05T04:54:11.243 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:11.243 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.243 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.243 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:11.243 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.062455+0000", 2024-02-05T04:54:11.243 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.61e-07 2024-02-05T04:54:11.244 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.244 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.244 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:11.244 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.062458+0000", 2024-02-05T04:54:11.244 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.7810000000000001e-06 2024-02-05T04:54:11.244 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.244 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.244 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:11.244 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.062466+0000", 2024-02-05T04:54:11.244 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.2430000000000005e-06 2024-02-05T04:54:11.244 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.244 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.244 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:11.244 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.062481+0000", 2024-02-05T04:54:11.245 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.5218999999999999e-05 2024-02-05T04:54:11.245 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.245 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.245 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:11.245 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.066745+0000", 2024-02-05T04:54:11.245 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0042640409999999997 2024-02-05T04:54:11.245 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.245 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.245 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "waiting for rw locks", 2024-02-05T04:54:11.245 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.066853+0000", 2024-02-05T04:54:11.245 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00010786299999999999 2024-02-05T04:54:11.245 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.245 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.245 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:11.245 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.072690+0000", 2024-02-05T04:54:11.246 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0058366490000000002 2024-02-05T04:54:11.246 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.246 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.246 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:54:11.246 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.072768+0000", 2024-02-05T04:54:11.246 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.8517999999999996e-05 2024-02-05T04:54:11.246 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.246 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.246 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:11.246 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.075608+0000", 2024-02-05T04:54:11.246 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0028393469999999999 2024-02-05T04:54:11.246 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:11.246 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:11.246 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:11.246 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.247 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.247 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:707 3.1s0 3:86872077:::smithi19639362-21:head [rmxattr NRZdh]WclahTe^_btdqe\\^mVYqQOaXaa\\aPVXWnk in=40b,rmxattr QnfshqY in=7b,rmxattr UkYrdYaZakpnReakOdc\\^bO[YPqfdkdtbNkghaSQ in=40b,rmxattr YOk\\Qsqnjk`Uh[icZasWhcZnad[h[k[Qmk]TihbR in=40b,rmxattr ]\\oXSo`PWafUO^dbhR^i`hifQgWjRhR[rRa^bnej in=40b,rmxattr _jkmO]cXrN_S^act\\QOOjpUZOj_Qf^otnZbpWZPU in=40b,rmxattr dRpscRhc^OtlbThWSmXVSNge]s in=26b,rmxattr l_Qeh\\RcafrRYN_YZf_iiaWagZfe[sop]^\\qTggq in=40b,rmxattr pZdlecZrV]qp]`sTkf_bSjZVZjhbScXfb[Sr_bsa in=40b] snapc 18=[18,17,16,12,8,7] ondisk+write+known_if_redirected+supports_pool_eio e75)", 2024-02-05T04:54:11.247 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:54:07.057099+0000", 2024-02-05T04:54:11.247 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 4.0901776070000002, 2024-02-05T04:54:11.247 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.019827641999999999, 2024-02-05T04:54:11.247 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:11.247 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:11.247 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:54:11.247 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:54:11.247 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:11.247 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 707 2024-02-05T04:54:11.247 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.247 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:11.247 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.247 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:11.248 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:07.057099+0000", 2024-02-05T04:54:11.248 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:11.248 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.248 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.248 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:11.248 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:07.057099+0000", 2024-02-05T04:54:11.248 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:11.248 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.248 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.248 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:11.248 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:07.057100+0000", 2024-02-05T04:54:11.248 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.6789999999999999e-06 2024-02-05T04:54:11.248 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.248 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.248 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:11.249 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:07.057106+0000", 2024-02-05T04:54:11.249 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.6099999999999997e-06 2024-02-05T04:54:11.249 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.249 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.249 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:11.249 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:07.057120+0000", 2024-02-05T04:54:11.249 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.3623e-05 2024-02-05T04:54:11.249 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.249 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.249 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:11.249 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:07.057129+0000", 2024-02-05T04:54:11.249 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.9590000000000001e-06 2024-02-05T04:54:11.249 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.249 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.249 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:11.249 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:07.057200+0000", 2024-02-05T04:54:11.250 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.1038000000000007e-05 2024-02-05T04:54:11.250 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.250 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.250 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:54:11.250 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:07.057356+0000", 2024-02-05T04:54:11.250 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00015600400000000001 2024-02-05T04:54:11.250 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.250 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.250 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:11.250 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:07.059400+0000", 2024-02-05T04:54:11.250 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.002044106 2024-02-05T04:54:11.250 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.250 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.250 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:11.250 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:07.065632+0000", 2024-02-05T04:54:11.251 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0062323880000000002 2024-02-05T04:54:11.251 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.251 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.251 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "commit_sent", 2024-02-05T04:54:11.251 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:07.076869+0000", 2024-02-05T04:54:11.251 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.011236658 2024-02-05T04:54:11.251 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:11.251 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:11.251 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:11.251 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:07.076926+0000", 2024-02-05T04:54:11.251 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.7577000000000001e-05 2024-02-05T04:54:11.251 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:11.251 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:11.251 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:11.251 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:11.251 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:11.252 INFO:teuthology.orchestra.run.smithi091.stdout:} 2024-02-05T04:54:11.253 INFO:tasks.rados.rados.0.smithi196.stdout:319: finishing write tid 3 to smithi19639362-24 2024-02-05T04:54:11.253 INFO:tasks.rados.rados.0.smithi196.stdout:319: finishing write tid 4 to smithi19639362-24 2024-02-05T04:54:11.253 INFO:tasks.rados.rados.0.smithi196.stdout:319: finishing write tid 5 to smithi19639362-24 2024-02-05T04:54:11.253 INFO:tasks.rados.rados.0.smithi196.stdout:319: finishing write tid 6 to smithi19639362-24 2024-02-05T04:54:11.253 INFO:tasks.rados.rados.0.smithi196.stdout:319: oid 24 updating version 0 to 128 2024-02-05T04:54:11.253 INFO:tasks.rados.rados.0.smithi196.stdout:319: oid 24 version 128 is already newer than 127 2024-02-05T04:54:11.253 INFO:tasks.rados.rados.0.smithi196.stdout:319: oid 24 updating version 128 to 129 2024-02-05T04:54:11.253 INFO:tasks.rados.rados.0.smithi196.stdout:319: oid 24 version 129 is already newer than 126 2024-02-05T04:54:11.253 INFO:tasks.rados.rados.0.smithi196.stdout:319: oid 24 updating version 129 to 130 2024-02-05T04:54:11.253 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 24 v 130 (ObjNum 137 snap 26 seq_num 137) dirty exists 2024-02-05T04:54:11.253 INFO:tasks.rados.rados.0.smithi196.stdout:319: left oid 24 (ObjNum 137 snap 26 seq_num 137) 2024-02-05T04:54:11.253 INFO:tasks.rados.rados.0.smithi196.stdout:320: finishing write tid 1 to smithi19639362-27 2024-02-05T04:54:11.253 INFO:tasks.rados.rados.0.smithi196.stdout:320: finishing write tid 2 to smithi19639362-27 2024-02-05T04:54:11.254 INFO:tasks.rados.rados.0.smithi196.stdout:320: finishing write tid 3 to smithi19639362-27 2024-02-05T04:54:11.254 INFO:tasks.rados.rados.0.smithi196.stdout:320: finishing write tid 4 to smithi19639362-27 2024-02-05T04:54:11.254 INFO:tasks.rados.rados.0.smithi196.stdout:320: finishing write tid 5 to smithi19639362-27 2024-02-05T04:54:11.254 INFO:tasks.rados.rados.0.smithi196.stdout:320: finishing write tid 6 to smithi19639362-27 2024-02-05T04:54:11.254 INFO:tasks.rados.rados.0.smithi196.stdout:320: oid 27 updating version 0 to 38 2024-02-05T04:54:11.254 INFO:tasks.rados.rados.0.smithi196.stdout:320: oid 27 version 38 is already newer than 35 2024-02-05T04:54:11.254 INFO:tasks.rados.rados.0.smithi196.stdout:320: oid 27 version 38 is already newer than 34 2024-02-05T04:54:11.254 INFO:tasks.rados.rados.0.smithi196.stdout:320: oid 27 version 38 is already newer than 36 2024-02-05T04:54:11.254 INFO:tasks.rados.rados.0.smithi196.stdout:320: oid 27 version 38 is already newer than 37 2024-02-05T04:54:11.254 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 27 v 38 (ObjNum 138 snap 26 seq_num 138) dirty exists 2024-02-05T04:54:11.254 INFO:tasks.rados.rados.0.smithi196.stdout:320: left oid 27 (ObjNum 138 snap 26 seq_num 138) 2024-02-05T04:54:11.255 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-02-05T04:54:11.256 INFO:tasks.ceph.osd.10.smithi150.stderr:2024-02-05T04:54:11.252+0000 7f3075a4d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 34018) UID: 0 2024-02-05T04:54:11.354 INFO:tasks.ceph.osd.9.smithi091.stderr:2024-02-05T04:54:11.354+0000 7efc18827640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 34279) UID: 0 2024-02-05T04:54:11.366 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-02-05T04:54:11.366 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [], 2024-02-05T04:54:11.366 INFO:teuthology.orchestra.run.smithi150.stdout: "num_ops": 0 2024-02-05T04:54:11.366 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-02-05T04:54:11.375 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-02-05T04:54:11.455 INFO:tasks.ceph.osd.2.smithi150.stderr:2024-02-05T04:54:11.454+0000 7f7337bf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 34014) UID: 0 2024-02-05T04:54:11.480 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-02-05T04:54:11.480 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [], 2024-02-05T04:54:11.480 INFO:teuthology.orchestra.run.smithi150.stdout: "complaint_time": 30, 2024-02-05T04:54:11.481 INFO:teuthology.orchestra.run.smithi150.stdout: "num_blocked_ops": 0 2024-02-05T04:54:11.481 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-02-05T04:54:11.489 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-02-05T04:54:11.556 INFO:tasks.ceph.osd.15.smithi196.stderr:2024-02-05T04:54:11.555+0000 7f1facc45640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 34611) UID: 0 2024-02-05T04:54:11.637 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-02-05T04:54:11.637 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 20, 2024-02-05T04:54:11.637 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 600, 2024-02-05T04:54:11.637 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [ 2024-02-05T04:54:11.637 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.637 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.4209.0:85 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'55, mlcod=24'50)", 2024-02-05T04:54:11.637 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:09.697543+0000", 2024-02-05T04:54:11.637 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 61.93750052, 2024-02-05T04:54:11.637 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0081615030000000005, 2024-02-05T04:54:11.637 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:11.637 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:11.638 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:11.638 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.638 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:11.638 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.697543+0000", 2024-02-05T04:54:11.638 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:11.638 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.638 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.638 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:11.638 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.697543+0000", 2024-02-05T04:54:11.638 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:11.638 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.638 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.638 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:11.638 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.697544+0000", 2024-02-05T04:54:11.639 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.7899999999999999e-07 2024-02-05T04:54:11.639 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.639 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.639 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:11.639 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.697546+0000", 2024-02-05T04:54:11.639 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.7970000000000001e-06 2024-02-05T04:54:11.639 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.639 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.639 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:11.639 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.697553+0000", 2024-02-05T04:54:11.639 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.2929999999999997e-06 2024-02-05T04:54:11.639 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.639 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.639 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:11.639 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.697556+0000", 2024-02-05T04:54:11.640 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.4309999999999998e-06 2024-02-05T04:54:11.640 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.640 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.640 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:11.640 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.697596+0000", 2024-02-05T04:54:11.640 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.977e-05 2024-02-05T04:54:11.640 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.640 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.640 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:54:11.640 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.697607+0000", 2024-02-05T04:54:11.640 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1459e-05 2024-02-05T04:54:11.640 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.640 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.641 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-02-05T04:54:11.641 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.705669+0000", 2024-02-05T04:54:11.641 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0080620009999999992 2024-02-05T04:54:11.641 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.641 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.641 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:11.641 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.705705+0000", 2024-02-05T04:54:11.641 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.5373000000000002e-05 2024-02-05T04:54:11.641 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:11.641 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:11.641 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:11.641 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.641 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.641 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.4209.0:86 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'56, mlcod=24'52)", 2024-02-05T04:54:11.642 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:09.698014+0000", 2024-02-05T04:54:11.642 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 61.937029420000002, 2024-02-05T04:54:11.642 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0077071750000000001, 2024-02-05T04:54:11.642 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:11.642 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:11.642 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:11.642 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.642 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:11.642 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698014+0000", 2024-02-05T04:54:11.642 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:11.642 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.642 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.642 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:11.642 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698014+0000", 2024-02-05T04:54:11.642 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:11.643 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.643 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.643 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:11.643 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698015+0000", 2024-02-05T04:54:11.643 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.9900000000000002e-07 2024-02-05T04:54:11.643 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.643 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.643 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:11.643 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698018+0000", 2024-02-05T04:54:11.643 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.0019999999999998e-06 2024-02-05T04:54:11.643 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.643 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.643 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:11.643 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698024+0000", 2024-02-05T04:54:11.643 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.251e-06 2024-02-05T04:54:11.644 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.644 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.644 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:11.644 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698028+0000", 2024-02-05T04:54:11.644 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.76e-06 2024-02-05T04:54:11.644 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.644 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.644 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:11.644 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698066+0000", 2024-02-05T04:54:11.644 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.8556999999999997e-05 2024-02-05T04:54:11.644 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.644 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.644 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:54:11.645 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.698078+0000", 2024-02-05T04:54:11.645 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1620999999999999e-05 2024-02-05T04:54:11.645 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.645 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.645 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-02-05T04:54:11.645 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.705706+0000", 2024-02-05T04:54:11.645 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0076284400000000002 2024-02-05T04:54:11.645 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.645 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.645 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:11.645 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.705721+0000", 2024-02-05T04:54:11.645 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.5245000000000001e-05 2024-02-05T04:54:11.645 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:11.645 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:11.645 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:11.646 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.646 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.646 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(client.4209.0:117 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'79, mlcod=24'69)", 2024-02-05T04:54:11.646 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:09.857803+0000", 2024-02-05T04:54:11.646 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 61.777240280000001, 2024-02-05T04:54:11.646 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0076301729999999996, 2024-02-05T04:54:11.646 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:11.646 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:11.646 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:11.646 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.646 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:11.646 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.857803+0000", 2024-02-05T04:54:11.646 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:11.647 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.647 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.647 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:11.647 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.857803+0000", 2024-02-05T04:54:11.647 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:11.647 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.647 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.647 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:11.647 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.857804+0000", 2024-02-05T04:54:11.647 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.1999999999999999e-07 2024-02-05T04:54:11.647 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.647 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.647 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:11.647 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.857810+0000", 2024-02-05T04:54:11.648 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.8690000000000001e-06 2024-02-05T04:54:11.648 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.648 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.648 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:11.648 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.857821+0000", 2024-02-05T04:54:11.648 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1059e-05 2024-02-05T04:54:11.648 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.648 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.648 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:11.648 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.857827+0000", 2024-02-05T04:54:11.648 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.1430000000000003e-06 2024-02-05T04:54:11.648 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.648 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.649 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:11.649 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.860138+0000", 2024-02-05T04:54:11.649 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0023111160000000002 2024-02-05T04:54:11.649 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.649 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.649 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:54:11.649 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.860152+0000", 2024-02-05T04:54:11.649 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.3896999999999999e-05 2024-02-05T04:54:11.649 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.649 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.649 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-02-05T04:54:11.649 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.865400+0000", 2024-02-05T04:54:11.649 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0052475040000000001 2024-02-05T04:54:11.650 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.650 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.650 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:11.650 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:09.865434+0000", 2024-02-05T04:54:11.650 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.3865000000000002e-05 2024-02-05T04:54:11.650 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:11.650 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:11.650 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:11.650 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.650 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.650 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.4634.0:246 3.es0 3:796382fe:::smithi19639362-29:head [append 0~0] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:54:11.650 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:29.186925+0000", 2024-02-05T04:54:11.650 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 42.448118960000002, 2024-02-05T04:54:11.650 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0088703129999999995, 2024-02-05T04:54:11.650 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:11.651 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:11.651 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-02-05T04:54:11.651 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.4634", 2024-02-05T04:54:11.651 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:11.651 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 246 2024-02-05T04:54:11.651 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.651 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:11.651 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.651 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:11.651 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186925+0000", 2024-02-05T04:54:11.651 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:11.651 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.651 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.651 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:11.651 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186925+0000", 2024-02-05T04:54:11.652 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:11.652 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.652 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.652 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:11.652 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186925+0000", 2024-02-05T04:54:11.652 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.53e-07 2024-02-05T04:54:11.652 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.652 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.652 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:11.652 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186928+0000", 2024-02-05T04:54:11.652 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.6900000000000001e-06 2024-02-05T04:54:11.652 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.652 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.652 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:11.652 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186933+0000", 2024-02-05T04:54:11.653 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.502e-06 2024-02-05T04:54:11.653 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.653 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.653 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:11.653 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186937+0000", 2024-02-05T04:54:11.653 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.6200000000000001e-06 2024-02-05T04:54:11.653 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.653 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.653 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:11.653 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.190223+0000", 2024-02-05T04:54:11.653 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0032863050000000002 2024-02-05T04:54:11.653 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.653 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.653 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:54:11.653 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.190312+0000", 2024-02-05T04:54:11.654 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.8821999999999998e-05 2024-02-05T04:54:11.654 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.654 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.654 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:11.654 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.190713+0000", 2024-02-05T04:54:11.654 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00040077400000000001 2024-02-05T04:54:11.654 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.654 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.654 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:11.654 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.193148+0000", 2024-02-05T04:54:11.654 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.002435544 2024-02-05T04:54:11.654 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.654 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.654 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-02-05T04:54:11.654 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.195751+0000", 2024-02-05T04:54:11.655 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0026025000000000002 2024-02-05T04:54:11.655 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.655 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.655 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:11.655 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.195795+0000", 2024-02-05T04:54:11.655 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.4202999999999997e-05 2024-02-05T04:54:11.655 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:11.655 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:11.655 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:11.655 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.655 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.655 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.4634.0:249 3.es0 3:70a5b9dd:::smithi19639362-46:head [delete] snapc 3=[3] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:54:11.655 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:29.186943+0000", 2024-02-05T04:54:11.656 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 42.448101000000001, 2024-02-05T04:54:11.656 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.011398484, 2024-02-05T04:54:11.656 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:11.656 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:11.656 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-02-05T04:54:11.656 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.4634", 2024-02-05T04:54:11.656 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:11.656 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 249 2024-02-05T04:54:11.656 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.656 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:11.656 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.656 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:11.656 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186943+0000", 2024-02-05T04:54:11.656 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:11.656 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.657 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.657 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:11.657 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186943+0000", 2024-02-05T04:54:11.657 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:11.657 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.657 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.657 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:11.657 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186943+0000", 2024-02-05T04:54:11.657 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.1899999999999999e-07 2024-02-05T04:54:11.657 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.657 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.657 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:11.657 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186944+0000", 2024-02-05T04:54:11.657 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.207e-06 2024-02-05T04:54:11.657 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.658 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.658 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:11.658 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186948+0000", 2024-02-05T04:54:11.658 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.7129999999999999e-06 2024-02-05T04:54:11.658 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.658 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.658 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:11.658 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.186951+0000", 2024-02-05T04:54:11.658 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.9979999999999999e-06 2024-02-05T04:54:11.658 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.658 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.658 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:11.658 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.191572+0000", 2024-02-05T04:54:11.658 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0046207669999999996 2024-02-05T04:54:11.658 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.658 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.659 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:54:11.659 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.191630+0000", 2024-02-05T04:54:11.659 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.8774999999999999e-05 2024-02-05T04:54:11.659 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.659 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.659 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:11.659 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.192049+0000", 2024-02-05T04:54:11.659 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00041888 2024-02-05T04:54:11.659 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.660 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.660 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:11.660 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.194188+0000", 2024-02-05T04:54:11.661 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0021391359999999998 2024-02-05T04:54:11.661 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.661 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.661 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-02-05T04:54:11.661 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.198284+0000", 2024-02-05T04:54:11.661 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0040960509999999999 2024-02-05T04:54:11.661 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.661 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.661 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:11.661 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:29.198341+0000", 2024-02-05T04:54:11.661 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.6737999999999998e-05 2024-02-05T04:54:11.661 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:11.661 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:11.661 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:11.662 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.662 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.662 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.4634.0:382 3.es0 3:7f0ef6bd:::smithi19639362-19:head [append 0~655360] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-02-05T04:54:11.662 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:34.167434+0000", 2024-02-05T04:54:11.662 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 37.467609711000001, 2024-02-05T04:54:11.662 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0077709989999999998, 2024-02-05T04:54:11.662 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:11.662 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:11.662 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-02-05T04:54:11.662 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.4634", 2024-02-05T04:54:11.662 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:11.662 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 382 2024-02-05T04:54:11.662 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.663 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:11.663 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.663 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:11.663 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.167434+0000", 2024-02-05T04:54:11.663 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:11.663 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.663 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.663 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:11.663 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.167434+0000", 2024-02-05T04:54:11.663 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:11.663 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.663 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.663 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:11.663 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.167436+0000", 2024-02-05T04:54:11.663 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.885e-06 2024-02-05T04:54:11.663 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.664 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.664 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:11.664 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.168595+0000", 2024-02-05T04:54:11.664 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0011590350000000001 2024-02-05T04:54:11.664 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.664 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.664 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:11.664 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.168610+0000", 2024-02-05T04:54:11.664 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.4778e-05 2024-02-05T04:54:11.664 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.664 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.664 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:11.665 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.168700+0000", 2024-02-05T04:54:11.665 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.0462000000000002e-05 2024-02-05T04:54:11.665 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.665 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.665 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:11.665 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.168791+0000", 2024-02-05T04:54:11.665 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.1015999999999997e-05 2024-02-05T04:54:11.665 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.665 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.665 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:54:11.665 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.168911+0000", 2024-02-05T04:54:11.665 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000119864 2024-02-05T04:54:11.665 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.665 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.666 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:11.666 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.170527+0000", 2024-02-05T04:54:11.666 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0016163589999999999 2024-02-05T04:54:11.666 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.666 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.666 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:11.666 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.174753+0000", 2024-02-05T04:54:11.666 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0042254429999999997 2024-02-05T04:54:11.666 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.666 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.666 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-02-05T04:54:11.666 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.175115+0000", 2024-02-05T04:54:11.666 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00036262499999999999 2024-02-05T04:54:11.666 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.666 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.667 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:11.667 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:34.175205+0000", 2024-02-05T04:54:11.667 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.9531999999999996e-05 2024-02-05T04:54:11.667 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:11.667 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:11.667 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:11.667 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.667 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.667 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.3s4 57/31 ECSubWrite(tid=76, reqid=osd.4.0:76, at_version=57'35, trim_to=0'0, roll_forward_to=47'33))", 2024-02-05T04:54:11.667 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:48.952725+0000", 2024-02-05T04:54:11.667 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 22.682318928000001, 2024-02-05T04:54:11.667 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.007485312, 2024-02-05T04:54:11.667 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:11.667 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:11.668 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:11.668 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.668 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:11.668 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.952725+0000", 2024-02-05T04:54:11.668 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:11.668 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.668 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.668 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:11.668 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.952725+0000", 2024-02-05T04:54:11.668 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:11.668 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.668 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.668 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:11.668 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.952726+0000", 2024-02-05T04:54:11.669 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.7100000000000011e-07 2024-02-05T04:54:11.669 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.669 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.669 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:11.669 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.952751+0000", 2024-02-05T04:54:11.669 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.5477e-05 2024-02-05T04:54:11.669 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.669 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.669 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:11.669 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.952765+0000", 2024-02-05T04:54:11.669 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.3502e-05 2024-02-05T04:54:11.669 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.669 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.669 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:11.669 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.952773+0000", 2024-02-05T04:54:11.670 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.1499999999999999e-06 2024-02-05T04:54:11.670 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.670 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.670 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:11.670 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.952832+0000", 2024-02-05T04:54:11.670 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.9095000000000003e-05 2024-02-05T04:54:11.670 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.670 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.670 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:11.670 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.952839+0000", 2024-02-05T04:54:11.670 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.764e-06 2024-02-05T04:54:11.670 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.670 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.670 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:11.670 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.960160+0000", 2024-02-05T04:54:11.671 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0073216820000000004 2024-02-05T04:54:11.671 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.671 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.671 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:11.671 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:48.960210+0000", 2024-02-05T04:54:11.671 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.9670999999999999e-05 2024-02-05T04:54:11.671 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:11.671 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:11.671 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:11.671 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.671 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.671 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.fs1 58/31 ECSubWrite(tid=104, reqid=client.4634.0:518, at_version=58'51, trim_to=0'0, roll_forward_to=57'49))", 2024-02-05T04:54:11.671 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:49.974036+0000", 2024-02-05T04:54:11.671 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 21.661007768000001, 2024-02-05T04:54:11.672 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0089249829999999992, 2024-02-05T04:54:11.672 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:11.672 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:11.672 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:11.672 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.672 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:11.672 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.974036+0000", 2024-02-05T04:54:11.672 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:11.672 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.672 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.672 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:11.672 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.974036+0000", 2024-02-05T04:54:11.672 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:11.672 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.672 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.673 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:11.673 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.974037+0000", 2024-02-05T04:54:11.673 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.0999999999999997e-07 2024-02-05T04:54:11.673 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.673 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.673 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:11.673 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.976765+0000", 2024-02-05T04:54:11.673 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0027280059999999998 2024-02-05T04:54:11.673 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.673 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.673 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:11.673 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.976780+0000", 2024-02-05T04:54:11.673 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.5238999999999999e-05 2024-02-05T04:54:11.673 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.673 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.674 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:11.674 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.976789+0000", 2024-02-05T04:54:11.674 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.8759999999999994e-06 2024-02-05T04:54:11.674 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.674 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.674 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:11.674 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.976857+0000", 2024-02-05T04:54:11.674 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.7984999999999994e-05 2024-02-05T04:54:11.674 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.674 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.674 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:11.674 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.976869+0000", 2024-02-05T04:54:11.674 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.1579999999999999e-05 2024-02-05T04:54:11.674 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.674 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.674 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:11.675 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.982917+0000", 2024-02-05T04:54:11.675 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0060485069999999998 2024-02-05T04:54:11.675 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.675 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.675 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:11.675 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:49.982961+0000", 2024-02-05T04:54:11.675 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.388e-05 2024-02-05T04:54:11.675 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:11.675 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:11.675 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:11.675 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.675 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.675 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.fs1 63/31 ECSubWrite(tid=122, reqid=osd.11.0:122, at_version=63'56, trim_to=0'0, roll_forward_to=58'51))", 2024-02-05T04:54:11.675 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:54.990151+0000", 2024-02-05T04:54:11.675 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 16.644892222999999, 2024-02-05T04:54:11.676 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.011036684, 2024-02-05T04:54:11.676 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:11.676 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:11.676 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:11.676 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.676 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:11.676 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.990151+0000", 2024-02-05T04:54:11.676 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:11.676 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.676 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.676 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:11.676 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.990151+0000", 2024-02-05T04:54:11.676 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:11.676 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.676 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.677 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:11.677 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.990152+0000", 2024-02-05T04:54:11.677 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.9999999999999997e-07 2024-02-05T04:54:11.677 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.677 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.677 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:11.677 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.990172+0000", 2024-02-05T04:54:11.677 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.0000000000000002e-05 2024-02-05T04:54:11.677 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.677 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.677 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:11.677 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.990183+0000", 2024-02-05T04:54:11.677 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0845999999999999e-05 2024-02-05T04:54:11.677 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.678 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.678 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:11.678 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.990196+0000", 2024-02-05T04:54:11.678 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.258e-05 2024-02-05T04:54:11.678 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.678 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.678 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:11.678 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.990255+0000", 2024-02-05T04:54:11.678 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.9175000000000002e-05 2024-02-05T04:54:11.678 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.678 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.678 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:11.678 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.990384+0000", 2024-02-05T04:54:11.678 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000128942 2024-02-05T04:54:11.678 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.679 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.679 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:11.679 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.001103+0000", 2024-02-05T04:54:11.679 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.010718811999999999 2024-02-05T04:54:11.679 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.679 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.679 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:11.679 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.001188+0000", 2024-02-05T04:54:11.679 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.5629000000000003e-05 2024-02-05T04:54:11.679 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:11.679 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:11.679 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:11.679 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.679 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.680 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.13s4 63/62 ECSubWrite(tid=83, reqid=osd.4.0:83, at_version=63'39, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:11.680 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:54.996293+0000", 2024-02-05T04:54:11.680 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 16.638750735999999, 2024-02-05T04:54:11.680 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.010849175000000001, 2024-02-05T04:54:11.680 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:11.680 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:11.680 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:11.680 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.680 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:11.680 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.996293+0000", 2024-02-05T04:54:11.680 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:11.680 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.680 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.680 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:11.680 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.996293+0000", 2024-02-05T04:54:11.681 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:11.681 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.681 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.681 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:11.681 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.996293+0000", 2024-02-05T04:54:11.681 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.2799999999999996e-07 2024-02-05T04:54:11.681 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.681 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.681 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:11.681 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.996317+0000", 2024-02-05T04:54:11.681 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.3402000000000001e-05 2024-02-05T04:54:11.681 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.681 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.681 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:11.681 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.996333+0000", 2024-02-05T04:54:11.682 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.6438000000000001e-05 2024-02-05T04:54:11.682 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.682 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.682 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:11.682 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.996342+0000", 2024-02-05T04:54:11.682 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.6899999999999998e-06 2024-02-05T04:54:11.682 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.682 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.682 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:11.682 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.996393+0000", 2024-02-05T04:54:11.682 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.0819999999999998e-05 2024-02-05T04:54:11.682 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.682 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.682 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:11.682 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.996402+0000", 2024-02-05T04:54:11.683 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.3000000000000007e-06 2024-02-05T04:54:11.683 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.683 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.683 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:11.683 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.007100+0000", 2024-02-05T04:54:11.683 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.010698193 2024-02-05T04:54:11.683 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.683 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.683 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:11.683 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.007142+0000", 2024-02-05T04:54:11.683 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.1804000000000002e-05 2024-02-05T04:54:11.683 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:11.683 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:11.683 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:11.684 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.684 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.684 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.12s1 63/62 ECSubWrite(tid=73, reqid=osd.12.0:73, at_version=63'55, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:11.684 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:54.998972+0000", 2024-02-05T04:54:11.684 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 16.63607193, 2024-02-05T04:54:11.684 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.011914983000000001, 2024-02-05T04:54:11.684 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:11.684 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:11.684 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:11.684 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.684 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:11.684 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.998972+0000", 2024-02-05T04:54:11.684 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:11.685 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.685 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.685 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:11.685 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.998972+0000", 2024-02-05T04:54:11.685 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:11.685 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.685 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.685 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:11.685 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.998972+0000", 2024-02-05T04:54:11.685 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.5199999999999996e-07 2024-02-05T04:54:11.685 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.685 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.685 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:11.685 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.998992+0000", 2024-02-05T04:54:11.686 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.9911999999999998e-05 2024-02-05T04:54:11.686 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.686 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.686 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:11.686 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.999013+0000", 2024-02-05T04:54:11.686 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.107e-05 2024-02-05T04:54:11.686 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.686 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.686 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:11.686 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.999022+0000", 2024-02-05T04:54:11.686 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.1310000000000006e-06 2024-02-05T04:54:11.686 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.686 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.686 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:11.687 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.999126+0000", 2024-02-05T04:54:11.687 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000104842 2024-02-05T04:54:11.687 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.687 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.687 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:11.687 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.999136+0000", 2024-02-05T04:54:11.687 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.8810000000000008e-06 2024-02-05T04:54:11.687 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.687 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.687 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:11.687 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.010847+0000", 2024-02-05T04:54:11.687 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.011711222 2024-02-05T04:54:11.687 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.687 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.687 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:11.688 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.010887+0000", 2024-02-05T04:54:11.688 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.9273000000000002e-05 2024-02-05T04:54:11.688 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:11.688 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:11.688 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:11.688 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.688 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.688 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.0s1 63/62 ECSubWrite(tid=52, reqid=client.4634.0:576, at_version=63'6, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:11.688 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:54.999558+0000", 2024-02-05T04:54:11.688 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 16.635485435, 2024-02-05T04:54:11.688 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0077513749999999996, 2024-02-05T04:54:11.688 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:11.688 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:11.689 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:11.689 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.689 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:11.689 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.999558+0000", 2024-02-05T04:54:11.689 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:11.689 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.689 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.689 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:11.689 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.999558+0000", 2024-02-05T04:54:11.689 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:11.689 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.689 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.689 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:11.689 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:54.999559+0000", 2024-02-05T04:54:11.689 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.6400000000000001e-07 2024-02-05T04:54:11.690 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.690 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.690 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:11.690 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.000299+0000", 2024-02-05T04:54:11.690 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00074006600000000005 2024-02-05T04:54:11.690 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.690 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.690 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:11.690 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.000312+0000", 2024-02-05T04:54:11.690 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2996000000000001e-05 2024-02-05T04:54:11.690 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.690 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.690 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:11.690 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.000321+0000", 2024-02-05T04:54:11.691 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.9029999999999993e-06 2024-02-05T04:54:11.691 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.691 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.691 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:11.691 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.000393+0000", 2024-02-05T04:54:11.691 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.1783e-05 2024-02-05T04:54:11.691 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.691 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.691 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:11.691 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.000403+0000", 2024-02-05T04:54:11.691 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.9690000000000006e-06 2024-02-05T04:54:11.691 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.691 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.691 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:11.692 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.007278+0000", 2024-02-05T04:54:11.692 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0068756479999999998 2024-02-05T04:54:11.692 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.692 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.692 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:11.692 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.007310+0000", 2024-02-05T04:54:11.692 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.1245999999999999e-05 2024-02-05T04:54:11.692 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:11.692 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:11.692 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:11.692 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.692 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.692 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.fs1 63/31 ECSubWrite(tid=124, reqid=osd.11.0:122, at_version=0'0, trim_to=0'0, roll_forward_to=63'56))", 2024-02-05T04:54:11.693 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:55.001791+0000", 2024-02-05T04:54:11.693 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 16.633252473999999, 2024-02-05T04:54:11.693 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.009426182, 2024-02-05T04:54:11.693 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:11.693 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:11.693 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:11.693 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.693 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:11.693 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.001791+0000", 2024-02-05T04:54:11.693 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:11.693 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.693 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.693 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:11.693 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.001791+0000", 2024-02-05T04:54:11.693 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:11.694 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.694 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.694 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:11.694 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.001792+0000", 2024-02-05T04:54:11.694 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.6899999999999996e-07 2024-02-05T04:54:11.694 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.694 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.694 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:11.694 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.001807+0000", 2024-02-05T04:54:11.694 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.4671999999999999e-05 2024-02-05T04:54:11.694 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.694 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.694 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:11.694 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.001822+0000", 2024-02-05T04:54:11.694 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.5071e-05 2024-02-05T04:54:11.695 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.695 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.695 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:11.695 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.001831+0000", 2024-02-05T04:54:11.695 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.1419999999999999e-06 2024-02-05T04:54:11.695 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.695 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.695 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:11.695 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.001987+0000", 2024-02-05T04:54:11.695 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00015554300000000001 2024-02-05T04:54:11.695 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.695 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.695 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:11.695 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.002123+0000", 2024-02-05T04:54:11.696 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000136231 2024-02-05T04:54:11.696 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.696 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.696 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:11.696 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.011175+0000", 2024-02-05T04:54:11.696 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0090525250000000005 2024-02-05T04:54:11.696 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.696 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.696 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:11.696 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.011217+0000", 2024-02-05T04:54:11.696 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.2029e-05 2024-02-05T04:54:11.696 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:11.696 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:11.696 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:11.697 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.697 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.697 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.4s1 63/62 ECSubWrite(tid=122, reqid=osd.1.0:122, at_version=63'73, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:11.697 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:55.002586+0000", 2024-02-05T04:54:11.697 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 16.632458133, 2024-02-05T04:54:11.697 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.011429455999999999, 2024-02-05T04:54:11.697 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:11.697 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:11.697 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:11.697 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.697 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:11.697 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.002586+0000", 2024-02-05T04:54:11.697 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:11.697 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.697 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.698 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:11.698 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.002586+0000", 2024-02-05T04:54:11.698 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:11.698 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.698 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.698 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:11.698 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.002586+0000", 2024-02-05T04:54:11.698 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.8599999999999998e-07 2024-02-05T04:54:11.698 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.698 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.698 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:11.698 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.002607+0000", 2024-02-05T04:54:11.698 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.0480999999999999e-05 2024-02-05T04:54:11.698 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.698 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.699 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:11.699 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.002619+0000", 2024-02-05T04:54:11.699 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.2099999999999999e-05 2024-02-05T04:54:11.699 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.699 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.699 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:11.699 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.002627+0000", 2024-02-05T04:54:11.699 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.9719999999999992e-06 2024-02-05T04:54:11.699 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.699 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.699 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:11.699 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.002691+0000", 2024-02-05T04:54:11.699 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.4784999999999997e-05 2024-02-05T04:54:11.699 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.700 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.700 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:11.700 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.002702+0000", 2024-02-05T04:54:11.700 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0138e-05 2024-02-05T04:54:11.700 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.700 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.700 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:11.700 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.013981+0000", 2024-02-05T04:54:11.700 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.011279632 2024-02-05T04:54:11.700 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.700 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.700 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:11.700 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.014015+0000", 2024-02-05T04:54:11.700 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.3862000000000002e-05 2024-02-05T04:54:11.701 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:11.701 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:11.701 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:11.701 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.701 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.701 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.8s2 63/62 ECSubWrite(tid=84, reqid=client.4634.0:593, at_version=0'0, trim_to=0'0, roll_forward_to=63'27))", 2024-02-05T04:54:11.701 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:55.002745+0000", 2024-02-05T04:54:11.701 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 16.632298902999999, 2024-02-05T04:54:11.701 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0076694420000000003, 2024-02-05T04:54:11.703 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:11.703 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:11.703 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:11.703 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.703 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:11.703 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.002745+0000", 2024-02-05T04:54:11.703 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:11.703 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.703 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.703 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:11.703 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.002745+0000", 2024-02-05T04:54:11.703 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:11.703 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.703 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.704 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:11.704 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.002746+0000", 2024-02-05T04:54:11.704 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.2099999999999995e-07 2024-02-05T04:54:11.704 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.704 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.704 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:11.704 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.002756+0000", 2024-02-05T04:54:11.704 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0509999999999999e-05 2024-02-05T04:54:11.704 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.704 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.704 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:11.704 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.002773+0000", 2024-02-05T04:54:11.704 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.6682000000000001e-05 2024-02-05T04:54:11.704 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.705 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.705 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:11.705 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.002782+0000", 2024-02-05T04:54:11.705 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.6910000000000003e-06 2024-02-05T04:54:11.705 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.705 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.705 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:11.705 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.002858+0000", 2024-02-05T04:54:11.705 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.6508999999999995e-05 2024-02-05T04:54:11.705 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.705 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.705 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:11.705 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.002869+0000", 2024-02-05T04:54:11.705 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0796e-05 2024-02-05T04:54:11.706 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.706 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.706 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:11.706 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.010372+0000", 2024-02-05T04:54:11.706 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0075036180000000001 2024-02-05T04:54:11.706 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.706 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.706 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:11.706 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.010414+0000", 2024-02-05T04:54:11.706 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.1715000000000003e-05 2024-02-05T04:54:11.706 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:11.706 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:11.706 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:11.706 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.706 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.707 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.0s1 63/62 ECSubWrite(tid=53, reqid=client.4634.0:577, at_version=63'7, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:11.707 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:55.002908+0000", 2024-02-05T04:54:11.707 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 16.632135412, 2024-02-05T04:54:11.707 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0075376250000000001, 2024-02-05T04:54:11.707 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:11.707 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:11.707 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:11.708 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.708 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:11.708 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.002908+0000", 2024-02-05T04:54:11.708 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:11.708 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.708 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.708 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:11.708 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.002908+0000", 2024-02-05T04:54:11.708 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:11.708 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.708 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.708 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:11.708 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.002909+0000", 2024-02-05T04:54:11.708 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0499999999999999e-06 2024-02-05T04:54:11.709 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.709 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.709 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:11.709 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.002993+0000", 2024-02-05T04:54:11.709 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.4120000000000001e-05 2024-02-05T04:54:11.709 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.709 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.709 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:11.709 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.003017+0000", 2024-02-05T04:54:11.709 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.3777999999999998e-05 2024-02-05T04:54:11.709 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.709 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.709 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:11.709 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.003027+0000", 2024-02-05T04:54:11.709 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0208e-05 2024-02-05T04:54:11.710 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.710 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.710 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:11.710 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.003094+0000", 2024-02-05T04:54:11.710 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 6.6623000000000002e-05 2024-02-05T04:54:11.710 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.710 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.710 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:11.710 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.003107+0000", 2024-02-05T04:54:11.710 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.3329999999999999e-05 2024-02-05T04:54:11.710 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.710 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.710 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:11.710 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.010417+0000", 2024-02-05T04:54:11.710 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.007309945 2024-02-05T04:54:11.710 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.711 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.711 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:11.711 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.010446+0000", 2024-02-05T04:54:11.711 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.8571000000000001e-05 2024-02-05T04:54:11.711 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:11.711 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:11.711 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:11.711 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.711 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.711 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.1s5 63/62 ECSubWrite(tid=265, reqid=client.4634.0:594, at_version=63'53, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:11.711 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:55.004101+0000", 2024-02-05T04:54:11.711 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 16.630942701999999, 2024-02-05T04:54:11.711 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0098795819999999996, 2024-02-05T04:54:11.711 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:11.711 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:11.712 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:11.712 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.712 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:11.712 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.004101+0000", 2024-02-05T04:54:11.712 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:11.712 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.712 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.712 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:11.712 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.004101+0000", 2024-02-05T04:54:11.712 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:11.712 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.712 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.712 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:11.712 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.004102+0000", 2024-02-05T04:54:11.713 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.5700000000000001e-07 2024-02-05T04:54:11.713 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.713 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.713 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:11.713 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.005989+0000", 2024-02-05T04:54:11.713 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.001887644 2024-02-05T04:54:11.713 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.713 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.713 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:11.713 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.006010+0000", 2024-02-05T04:54:11.713 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.0429e-05 2024-02-05T04:54:11.713 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.713 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.713 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:11.713 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.006021+0000", 2024-02-05T04:54:11.714 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0658e-05 2024-02-05T04:54:11.714 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.714 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.714 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:11.714 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.006094+0000", 2024-02-05T04:54:11.714 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.3268000000000004e-05 2024-02-05T04:54:11.714 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.714 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.714 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:11.714 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.006104+0000", 2024-02-05T04:54:11.714 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0331e-05 2024-02-05T04:54:11.714 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.714 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.714 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:11.714 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.013933+0000", 2024-02-05T04:54:11.714 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0078288880000000009 2024-02-05T04:54:11.715 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.715 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.715 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:11.715 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.013981+0000", 2024-02-05T04:54:11.715 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.7506999999999998e-05 2024-02-05T04:54:11.715 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:11.715 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:11.715 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:11.715 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.715 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.715 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDECSubOpWrite(3.0s1 63/62 ECSubWrite(tid=54, reqid=client.4634.0:578, at_version=63'8, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:11.715 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:53:55.005643+0000", 2024-02-05T04:54:11.715 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 16.629400209, 2024-02-05T04:54:11.715 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0084155380000000002, 2024-02-05T04:54:11.716 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:11.716 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", 2024-02-05T04:54:11.716 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:11.716 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.716 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:11.716 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.005643+0000", 2024-02-05T04:54:11.716 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:11.716 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.716 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.716 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:11.716 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.005643+0000", 2024-02-05T04:54:11.716 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:11.716 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.716 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.716 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:11.717 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.005644+0000", 2024-02-05T04:54:11.717 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.6300000000000004e-07 2024-02-05T04:54:11.717 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.717 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.717 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:11.717 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.005730+0000", 2024-02-05T04:54:11.717 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.6080000000000003e-05 2024-02-05T04:54:11.717 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.717 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.717 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:11.717 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.005748+0000", 2024-02-05T04:54:11.717 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.7277000000000001e-05 2024-02-05T04:54:11.717 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.717 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.717 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:11.718 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.005758+0000", 2024-02-05T04:54:11.718 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0030000000000001e-05 2024-02-05T04:54:11.718 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.718 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.718 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:11.718 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.005849+0000", 2024-02-05T04:54:11.718 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.1558000000000007e-05 2024-02-05T04:54:11.718 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.718 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.718 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:11.718 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.005860+0000", 2024-02-05T04:54:11.718 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0671e-05 2024-02-05T04:54:11.718 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.718 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.718 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:11.718 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.014029+0000", 2024-02-05T04:54:11.719 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0081686699999999994 2024-02-05T04:54:11.719 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.719 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.719 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:11.719 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:53:55.014059+0000", 2024-02-05T04:54:11.719 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.0389e-05 2024-02-05T04:54:11.719 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:11.719 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:11.719 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:11.719 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.719 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.719 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.4634.0:625 3.es0 3:796382fe:::smithi19639362-29:head [copy-from ver 25] snapc 16=[16,15,12,11,f,b,8,7] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-02-05T04:54:11.719 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:54:00.952558+0000", 2024-02-05T04:54:11.719 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 10.682486006, 2024-02-05T04:54:11.719 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.032663225999999997, 2024-02-05T04:54:11.720 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:11.720 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:11.720 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-02-05T04:54:11.720 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.4634", 2024-02-05T04:54:11.720 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:11.720 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 625 2024-02-05T04:54:11.720 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.720 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:11.720 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.720 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:11.720 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.952558+0000", 2024-02-05T04:54:11.720 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:11.720 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.720 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.721 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:11.721 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.952558+0000", 2024-02-05T04:54:11.721 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:11.721 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.721 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.721 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:11.721 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.952560+0000", 2024-02-05T04:54:11.721 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 2.0329999999999998e-06 2024-02-05T04:54:11.721 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.721 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.721 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:11.721 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.952568+0000", 2024-02-05T04:54:11.721 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.5629999999999992e-06 2024-02-05T04:54:11.721 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.721 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.722 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:11.722 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.952582+0000", 2024-02-05T04:54:11.722 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.3784999999999999e-05 2024-02-05T04:54:11.722 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.722 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.722 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:11.722 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.952592+0000", 2024-02-05T04:54:11.722 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0132999999999999e-05 2024-02-05T04:54:11.722 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.722 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.722 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:11.722 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.952723+0000", 2024-02-05T04:54:11.722 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000130968 2024-02-05T04:54:11.722 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.722 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.722 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:54:11.723 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.952851+0000", 2024-02-05T04:54:11.723 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000127849 2024-02-05T04:54:11.723 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.723 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.723 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_started", 2024-02-05T04:54:11.723 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.976458+0000", 2024-02-05T04:54:11.723 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.023607337999999999 2024-02-05T04:54:11.723 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.723 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.723 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_committed", 2024-02-05T04:54:11.723 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.981678+0000", 2024-02-05T04:54:11.723 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.005219942 2024-02-05T04:54:11.723 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.723 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.723 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", 2024-02-05T04:54:11.724 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.985162+0000", 2024-02-05T04:54:11.724 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0034837090000000002 2024-02-05T04:54:11.724 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.724 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.724 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:11.724 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.985221+0000", 2024-02-05T04:54:11.724 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.8906000000000001e-05 2024-02-05T04:54:11.724 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:11.724 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:11.724 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:11.724 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.724 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.724 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.4634.0:626 3.es0 3:796382fe:::smithi19639362-29:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e69)", 2024-02-05T04:54:11.724 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2024-02-05T04:54:00.952658+0000", 2024-02-05T04:54:11.724 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 10.682386013, 2024-02-05T04:54:11.724 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.027128118, 2024-02-05T04:54:11.725 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { 2024-02-05T04:54:11.725 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "started", 2024-02-05T04:54:11.725 INFO:teuthology.orchestra.run.smithi150.stdout: "client_info": { 2024-02-05T04:54:11.725 INFO:teuthology.orchestra.run.smithi150.stdout: "client": "client.4634", 2024-02-05T04:54:11.725 INFO:teuthology.orchestra.run.smithi150.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:11.725 INFO:teuthology.orchestra.run.smithi150.stdout: "tid": 626 2024-02-05T04:54:11.725 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.725 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2024-02-05T04:54:11.725 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.725 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2024-02-05T04:54:11.725 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.952658+0000", 2024-02-05T04:54:11.725 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:11.725 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.725 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.725 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2024-02-05T04:54:11.726 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.952658+0000", 2024-02-05T04:54:11.726 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2024-02-05T04:54:11.726 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.726 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.726 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2024-02-05T04:54:11.726 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.952659+0000", 2024-02-05T04:54:11.726 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.8000000000000004e-07 2024-02-05T04:54:11.726 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.726 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.726 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2024-02-05T04:54:11.726 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.952663+0000", 2024-02-05T04:54:11.726 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.7219999999999999e-06 2024-02-05T04:54:11.726 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.726 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.726 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "dispatched", 2024-02-05T04:54:11.726 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.952674+0000", 2024-02-05T04:54:11.727 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 1.0886e-05 2024-02-05T04:54:11.727 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.727 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.727 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2024-02-05T04:54:11.727 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.952681+0000", 2024-02-05T04:54:11.727 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.2169999999999997e-06 2024-02-05T04:54:11.727 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.727 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.727 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:11.727 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.953055+0000", 2024-02-05T04:54:11.727 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000373254 2024-02-05T04:54:11.727 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.727 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.727 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "waiting for blocked object", 2024-02-05T04:54:11.727 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.953099+0000", 2024-02-05T04:54:11.727 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.4583999999999999e-05 2024-02-05T04:54:11.728 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.728 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.728 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2024-02-05T04:54:11.728 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.979623+0000", 2024-02-05T04:54:11.728 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.026523794999999999 2024-02-05T04:54:11.728 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.728 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.728 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2024-02-05T04:54:11.728 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.979709+0000", 2024-02-05T04:54:11.728 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 8.6291000000000004e-05 2024-02-05T04:54:11.728 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-02-05T04:54:11.728 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-02-05T04:54:11.728 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2024-02-05T04:54:11.728 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2024-02-05T04:54:00.979786+0000", 2024-02-05T04:54:11.728 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.6489000000000006e-05 2024-02-05T04:54:11.729 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:11.729 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:11.729 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:11.729 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-02-05T04:54:11.729 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-05T04:54:11.729 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-02-05T04:54:11.730 INFO:tasks.ceph.osd.11.smithi196.stderr:2024-02-05T04:54:11.730+0000 7f85a8d86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 34609) UID: 0 2024-02-05T04:54:11.731 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-02-05T04:54:11.830 INFO:tasks.ceph.osd.2.smithi150.stderr:2024-02-05T04:54:11.829+0000 7f7337bf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 34014) UID: 0 2024-02-05T04:54:11.837 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-02-05T04:54:11.837 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [], 2024-02-05T04:54:11.837 INFO:teuthology.orchestra.run.smithi150.stdout: "num_ops": 0 2024-02-05T04:54:11.838 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-02-05T04:54:11.846 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-02-05T04:54:11.860 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T04:54:11.932 INFO:tasks.ceph.osd.4.smithi062.stderr:2024-02-05T04:54:11.930+0000 7f708c308640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34689) UID: 0 2024-02-05T04:54:11.991 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0: {} 2024-02-05T04:54:11.991 INFO:teuthology.orchestra.run.smithi062.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T04:54:11.993 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-02-05T04:54:11.993 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [], 2024-02-05T04:54:11.993 INFO:teuthology.orchestra.run.smithi150.stdout: "complaint_time": 30, 2024-02-05T04:54:11.993 INFO:teuthology.orchestra.run.smithi150.stdout: "num_blocked_ops": 0 2024-02-05T04:54:11.994 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-02-05T04:54:12.001 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-02-05T04:54:12.010 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1: {} 2024-02-05T04:54:12.010 INFO:teuthology.orchestra.run.smithi062.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T04:54:12.028 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2: {} 2024-02-05T04:54:12.028 INFO:teuthology.orchestra.run.smithi062.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T04:54:12.046 INFO:tasks.ceph.osd.10.smithi150.stderr:2024-02-05T04:54:12.045+0000 7f3075a4d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 34018) UID: 0 2024-02-05T04:54:12.046 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3: {} 2024-02-05T04:54:12.047 INFO:teuthology.orchestra.run.smithi062.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T04:54:12.063 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4: {} 2024-02-05T04:54:12.063 INFO:teuthology.orchestra.run.smithi062.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T04:54:12.064 INFO:teuthology.orchestra.run.smithi062.stderr:nodeep-scrub is set 2024-02-05T04:54:12.064 INFO:teuthology.orchestra.run.smithi062.stderr:reweighted osd.15 to 1 (10000) 2024-02-05T04:54:12.072 INFO:tasks.rados.rados.0.smithi196.stdout:318: done (3 left) 2024-02-05T04:54:12.072 INFO:tasks.rados.rados.0.smithi196.stdout:319: done (2 left) 2024-02-05T04:54:12.073 INFO:tasks.rados.rados.0.smithi196.stdout:320: done (1 left) 2024-02-05T04:54:12.073 INFO:tasks.rados.rados.0.smithi196.stdout:321: done (0 left) 2024-02-05T04:54:12.073 INFO:tasks.rados.rados.0.smithi196.stdout:322: rollback oid 16 current snap is 27 2024-02-05T04:54:12.073 INFO:tasks.rados.rados.0.smithi196.stdout:rollback oid 16 to 16 2024-02-05T04:54:12.073 INFO:tasks.rados.rados.0.smithi196.stdout:323: rmattr oid 49 current snap is 27 2024-02-05T04:54:12.073 INFO:tasks.rados.rados.0.smithi196.stdout:323: done (1 left) 2024-02-05T04:54:12.073 INFO:tasks.rados.rados.0.smithi196.stdout:324: delete oid 12 current snap is 27 2024-02-05T04:54:12.077 INFO:tasks.rados.rados.0.smithi196.stdout:324: done (1 left) 2024-02-05T04:54:12.077 INFO:tasks.rados.rados.0.smithi196.stdout:325: rmattr oid 8 current snap is 27 2024-02-05T04:54:12.077 INFO:tasks.rados.rados.0.smithi196.stdout:326: read oid 35 snap -1 2024-02-05T04:54:12.077 INFO:tasks.rados.rados.0.smithi196.stdout:326: expect (ObjNum 109 snap 17 seq_num 109) 2024-02-05T04:54:12.077 INFO:tasks.rados.rados.0.smithi196.stdout:327: rollback oid 29 current snap is 27 2024-02-05T04:54:12.077 INFO:tasks.rados.rados.0.smithi196.stdout:rollback oid 29 to 16 2024-02-05T04:54:12.078 INFO:tasks.rados.rados.0.smithi196.stdout:328: setattr oid 19 current snap is 27 2024-02-05T04:54:12.079 INFO:tasks.rados.rados.0.smithi196.stdout:322: finishing rollback tid 1 to smithi19639362-16 2024-02-05T04:54:12.079 INFO:tasks.rados.rados.0.smithi196.stdout:322: finishing rollback tid 2 to smithi19639362-16 2024-02-05T04:54:12.080 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 16 v 50 (ObjNum 60 snap 2 seq_num 60) dirty exists 2024-02-05T04:54:12.081 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-02-05T04:54:12.081 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-05T04:54:12.082 INFO:tasks.rados.rados.0.smithi196.stdout:322: done (4 left) 2024-02-05T04:54:12.082 INFO:tasks.rados.rados.0.smithi196.stdout:329: snap_remove snap 5 2024-02-05T04:54:12.090 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5: {} 2024-02-05T04:54:12.090 INFO:teuthology.orchestra.run.smithi062.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T04:54:12.108 INFO:teuthology.orchestra.run.smithi062.stdout:osd.6: {} 2024-02-05T04:54:12.108 INFO:teuthology.orchestra.run.smithi062.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T04:54:12.116 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:12.116 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:12.125 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-02-05T04:54:12.126 INFO:teuthology.orchestra.run.smithi062.stdout:osd.7: {} 2024-02-05T04:54:12.127 INFO:teuthology.orchestra.run.smithi062.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T04:54:12.134 INFO:tasks.ceph.osd.0.smithi062.stderr:2024-02-05T04:54:12.132+0000 7fece8e03640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34688) UID: 0 2024-02-05T04:54:12.144 INFO:teuthology.orchestra.run.smithi062.stdout:osd.8: {} 2024-02-05T04:54:12.144 INFO:teuthology.orchestra.run.smithi062.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-05T04:54:12.162 INFO:teuthology.orchestra.run.smithi062.stdout:osd.9: {} 2024-02-05T04:54:12.162 INFO:teuthology.orchestra.run.smithi062.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-05T04:54:12.180 INFO:teuthology.orchestra.run.smithi062.stdout:osd.10: {} 2024-02-05T04:54:12.180 INFO:teuthology.orchestra.run.smithi062.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-05T04:54:12.199 INFO:teuthology.orchestra.run.smithi062.stdout:osd.11: {} 2024-02-05T04:54:12.199 INFO:teuthology.orchestra.run.smithi062.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-05T04:54:12.216 INFO:teuthology.orchestra.run.smithi062.stdout:osd.12: {} 2024-02-05T04:54:12.216 INFO:teuthology.orchestra.run.smithi062.stderr:osd.12: osd_enable_op_tracker = '' 2024-02-05T04:54:12.234 INFO:teuthology.orchestra.run.smithi062.stdout:osd.13: {} 2024-02-05T04:54:12.234 INFO:teuthology.orchestra.run.smithi062.stderr:osd.13: osd_enable_op_tracker = '' 2024-02-05T04:54:12.234 INFO:tasks.ceph.osd.13.smithi091.stderr:2024-02-05T04:54:12.233+0000 7f526fd4d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 34281) UID: 0 2024-02-05T04:54:12.251 INFO:teuthology.orchestra.run.smithi062.stdout:osd.14: {} 2024-02-05T04:54:12.251 INFO:teuthology.orchestra.run.smithi062.stderr:osd.14: osd_enable_op_tracker = '' 2024-02-05T04:54:12.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-02-05T04:54:12.250+0000 7fab81477640 0 -- 172.21.15.62:0/306861739 >> [v2:172.21.15.196:6826/3187639683,v1:172.21.15.196:6827/3187639683] conn(0x7fab7c222050 msgr2=0x7fab7c224440 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-02-05T04:54:12.270 INFO:teuthology.orchestra.run.smithi062.stdout:osd.15: {} 2024-02-05T04:54:12.270 INFO:teuthology.orchestra.run.smithi062.stderr:osd.15: osd_enable_op_tracker = '' 2024-02-05T04:54:12.272 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:12.272 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:12.282 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-02-05T04:54:12.335 INFO:tasks.ceph.osd.0.smithi062.stderr:2024-02-05T04:54:12.333+0000 7fece8e03640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34688) UID: 0 2024-02-05T04:54:12.389 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:54:12.390 INFO:teuthology.orchestra.run.smithi062.stderr:dumped all 2024-02-05T04:54:12.391 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:12.391 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:12.396 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-02-05T04:54:12.400 INFO:teuthology.orchestra.run.smithi062.stdout:{"pg_ready":true,"pg_map":{"version":100,"stamp":"2024-02-05T04:54:12.064816+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":118784544,"num_objects":110,"num_object_clones":60,"num_object_copies":760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":110,"num_whiteouts":7,"num_read":320,"num_read_kb":66385,"num_write":453,"num_write_kb":73652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":15421984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1441,"ondisk_log_size":1441,"up":200,"acting":200,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":200,"num_osds":15,"num_per_pool_osds":15,"num_per_pool_omap_osds":15,"kb":1415577600,"kb_used":676940,"kb_used_data":230000,"kb_used_omap":23,"kb_used_meta":446696,"kb_avail":1414900660,"statfs":{"total":1449551462400,"available":1448858275840,"internally_reserved":0,"allocated":235520000,"data_stored":319330734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23925,"internal_metadata":457417355},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5439488,"num_objects":5,"num_object_clones":6,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":11,"num_read_kb":17720,"num_write":-5,"num_write_kb":5720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-4,"num_bytes_recovered":-3727360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.013319"},"pg_stats":[{"pgid":"3.19","version":"75'31","reported_seq":113,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-05T04:54:07.044424+0000","last_change":"2024-02-05T04:54:07.045572+0000","last_active":"2024-02-05T04:54:07.044424+0000","last_peered":"2024-02-05T04:54:07.044424+0000","last_clean":"2024-02-05T04:54:07.044424+0000","last_became_active":"2024-02-05T04:53:59.958048+0000","last_became_peered":"2024-02-05T04:53:59.958048+0000","last_unstale":"2024-02-05T04:54:07.044424+0000","last_undegraded":"2024-02-05T04:54:07.044424+0000","last_fullsized":"2024-02-05T04:54:07.044424+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":5,"last_scrub":"57'21","last_scrub_stamp":"2024-02-05T04:54:03.585473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:54:03.585473+0000","objects_scrubbed":3,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:55:06.674676+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.005206535,"stat_sum":{"num_bytes":1900544,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,11,12,14,7,6],"acting":[9,1,11,12,14,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"9","length":"9"},{"start":"13","length":"3"}]},{"pgid":"3.18","version":"45'23","reported_seq":80,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-05T04:54:08.058674+0000","last_change":"2024-02-05T04:54:07.043765+0000","last_active":"2024-02-05T04:54:08.058674+0000","last_peered":"2024-02-05T04:54:08.058674+0000","last_clean":"2024-02-05T04:54:08.058674+0000","last_became_active":"2024-02-05T04:53:59.958280+0000","last_became_peered":"2024-02-05T04:53:59.958280+0000","last_unstale":"2024-02-05T04:54:08.058674+0000","last_undegraded":"2024-02-05T04:54:08.058674+0000","last_fullsized":"2024-02-05T04:54:08.058674+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":5,"last_scrub":"45'23","last_scrub_stamp":"2024-02-05T04:54:03.694275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:54:03.694275+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:55:30.533656+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00075505899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,6,1,13,15,7,5],"acting":[12,6,1,13,15,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"9","length":"9"},{"start":"13","length":"3"}]},{"pgid":"3.17","version":"75'43","reported_seq":118,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-05T04:54:08.062637+0000","last_change":"2024-02-05T04:54:07.081867+0000","last_active":"2024-02-05T04:54:08.062637+0000","last_peered":"2024-02-05T04:54:08.062637+0000","last_clean":"2024-02-05T04:54:08.062637+0000","last_became_active":"2024-02-05T04:53:59.955376+0000","last_became_peered":"2024-02-05T04:53:59.955376+0000","last_unstale":"2024-02-05T04:54:08.062637+0000","last_undegraded":"2024-02-05T04:54:08.062637+0000","last_fullsized":"2024-02-05T04:54:08.062637+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":5,"last_scrub":"58'30","last_scrub_stamp":"2024-02-05T04:54:03.827890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:54:03.827890+0000","objects_scrubbed":2,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:55:04.966456+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.023257488999999999,"stat_sum":{"num_bytes":1736704,"num_objects":3,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":2048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,10,4,7,14,11],"acting":[0,3,10,4,7,14,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"9","length":"9"},{"start":"13","length":"3"}]},{"pgid":"3.16","version":"75'36","reported_seq":116,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-05T04:54:08.058666+0000","last_change":"2024-02-05T04:54:07.055115+0000","last_active":"2024-02-05T04:54:08.058666+0000","last_peered":"2024-02-05T04:54:08.058666+0000","last_clean":"2024-02-05T04:54:08.058666+0000","last_became_active":"2024-02-05T04:53:59.956531+0000","last_became_peered":"2024-02-05T04:53:59.956531+0000","last_unstale":"2024-02-05T04:54:08.058666+0000","last_undegraded":"2024-02-05T04:54:08.058666+0000","last_fullsized":"2024-02-05T04:54:08.058666+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":5,"last_scrub":"71'30","last_scrub_stamp":"2024-02-05T04:54:03.634933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:54:03.634933+0000","objects_scrubbed":2,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:55:32.808902+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.010565771,"stat_sum":{"num_bytes":622592,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,15,9,5,10,12,1],"acting":[11,15,9,5,10,12,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"9","length":"9"},{"start":"13","length":"3"}]},{"pgid":"3.15","version":"58'40","reported_seq":80,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-05T04:54:08.058968+0000","last_change":"2024-02-05T04:54:07.044227+0000","last_active":"2024-02-05T04:54:08.058968+0000","last_peered":"2024-02-05T04:54:08.058968+0000","last_clean":"2024-02-05T04:54:08.058968+0000","last_became_active":"2024-02-05T04:53:59.955382+0000","last_became_peered":"2024-02-05T04:53:59.955382+0000","last_unstale":"2024-02-05T04:54:08.058968+0000","last_undegraded":"2024-02-05T04:54:08.058968+0000","last_fullsized":"2024-02-05T04:54:08.058968+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":5,"last_scrub":"58'40","last_scrub_stamp":"2024-02-05T04:54:04.666054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:54:04.666054+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:55:20.921718+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0010679039999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,6,3,4,14,2,15],"acting":[12,6,3,4,14,2,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"9","length":"9"},{"start":"13","length":"3"}]},{"pgid":"3.14","version":"79'87","reported_seq":141,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-05T04:54:11.062977+0000","last_change":"2024-02-05T04:54:10.062997+0000","last_active":"2024-02-05T04:54:11.062977+0000","last_peered":"2024-02-05T04:54:11.062977+0000","last_clean":"2024-02-05T04:54:11.062977+0000","last_became_active":"2024-02-05T04:53:59.957250+0000","last_became_peered":"2024-02-05T04:53:59.957250+0000","last_unstale":"2024-02-05T04:54:11.062977+0000","last_undegraded":"2024-02-05T04:54:11.062977+0000","last_fullsized":"2024-02-05T04:54:11.062977+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":5,"last_scrub":"71'79","last_scrub_stamp":"2024-02-05T04:54:03.639971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:54:03.639971+0000","objects_scrubbed":5,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:55:09.216854+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00061923700000000002,"stat_sum":{"num_bytes":7454720,"num_objects":6,"num_object_clones":4,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,14,15,0,12,9,13],"acting":[3,14,15,0,12,9,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"b","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"15","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":55,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-05T04:54:06.048457+0000","last_change":"2024-02-05T04:53:12.781269+0000","last_active":"2024-02-05T04:54:06.048457+0000","last_peered":"2024-02-05T04:54:06.048457+0000","last_clean":"2024-02-05T04:54:06.048457+0000","last_became_active":"2024-02-05T04:53:12.780970+0000","last_became_peered":"2024-02-05T04:53:12.780970+0000","last_unstale":"2024-02-05T04:54:06.048457+0000","last_undegraded":"2024-02-05T04:54:06.048457+0000","last_fullsized":"2024-02-05T04:54:06.048457+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_clean_scrub_stamp":"2024-02-05T04:53:11.017155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:36.119048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,0],"acting":[15,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.6","version":"75'44","reported_seq":270,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-05T04:54:07.088213+0000","last_change":"2024-02-05T04:54:07.072361+0000","last_active":"2024-02-05T04:54:07.088213+0000","last_peered":"2024-02-05T04:54:07.088213+0000","last_clean":"2024-02-05T04:54:07.088213+0000","last_became_active":"2024-02-05T04:53:54.992358+0000","last_became_peered":"2024-02-05T04:53:54.992358+0000","last_unstale":"2024-02-05T04:54:07.088213+0000","last_undegraded":"2024-02-05T04:54:07.088213+0000","last_fullsized":"2024-02-05T04:54:07.088213+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'26","last_scrub_stamp":"2024-02-05T04:53:55.597534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:55.597534+0000","objects_scrubbed":1,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:55:11.475373+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.017878090999999999,"stat_sum":{"num_bytes":4571136,"num_objects":3,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,12,4,15,1,7,10],"acting":[13,12,4,15,1,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"9","length":"9"},{"start":"13","length":"3"}]},{"pgid":"2.6","version":"0'0","reported_seq":57,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-05T04:54:06.042328+0000","last_change":"2024-02-05T04:53:12.037950+0000","last_active":"2024-02-05T04:54:06.042328+0000","last_peered":"2024-02-05T04:54:06.042328+0000","last_clean":"2024-02-05T04:54:06.042328+0000","last_became_active":"2024-02-05T04:53:12.037750+0000","last_became_peered":"2024-02-05T04:53:12.037750+0000","last_unstale":"2024-02-05T04:54:06.042328+0000","last_undegraded":"2024-02-05T04:54:06.042328+0000","last_fullsized":"2024-02-05T04:54:06.042328+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_clean_scrub_stamp":"2024-02-05T04:53:11.017155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:26.129079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"74'32","reported_seq":220,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-05T04:54:07.054778+0000","last_change":"2024-02-05T04:54:07.055408+0000","last_active":"2024-02-05T04:54:07.054778+0000","last_peered":"2024-02-05T04:54:07.054778+0000","last_clean":"2024-02-05T04:54:07.054778+0000","last_became_active":"2024-02-05T04:53:54.997596+0000","last_became_peered":"2024-02-05T04:53:54.997596+0000","last_unstale":"2024-02-05T04:54:07.054778+0000","last_undegraded":"2024-02-05T04:54:07.054778+0000","last_fullsized":"2024-02-05T04:54:07.054778+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'30","last_scrub_stamp":"2024-02-05T04:53:56.628250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:56.628250+0000","objects_scrubbed":2,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:55:11.669159+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00099417400000000001,"stat_sum":{"num_bytes":3670016,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,6,12,1,4,5,11],"acting":[13,6,12,1,4,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"9","length":"9"},{"start":"13","length":"3"}]},{"pgid":"2.5","version":"0'0","reported_seq":61,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-05T04:54:10.060951+0000","last_change":"2024-02-05T04:53:12.753233+0000","last_active":"2024-02-05T04:54:10.060951+0000","last_peered":"2024-02-05T04:54:10.060951+0000","last_clean":"2024-02-05T04:54:10.060951+0000","last_became_active":"2024-02-05T04:53:12.752915+0000","last_became_peered":"2024-02-05T04:53:12.752915+0000","last_unstale":"2024-02-05T04:54:10.060951+0000","last_undegraded":"2024-02-05T04:54:10.060951+0000","last_fullsized":"2024-02-05T04:54:10.060951+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_clean_scrub_stamp":"2024-02-05T04:53:11.017155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:18.391150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"74'81","reported_seq":319,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-05T04:54:07.043596+0000","last_change":"2024-02-05T04:54:07.044364+0000","last_active":"2024-02-05T04:54:07.043596+0000","last_peered":"2024-02-05T04:54:07.043596+0000","last_clean":"2024-02-05T04:54:07.043596+0000","last_became_active":"2024-02-05T04:53:54.996849+0000","last_became_peered":"2024-02-05T04:53:54.996849+0000","last_unstale":"2024-02-05T04:54:07.043596+0000","last_undegraded":"2024-02-05T04:54:07.043596+0000","last_fullsized":"2024-02-05T04:54:07.043596+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'73","last_scrub_stamp":"2024-02-05T04:53:55.059635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:55.059635+0000","objects_scrubbed":6,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:55:01.074914+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00090901299999999997,"stat_sum":{"num_bytes":9338880,"num_objects":6,"num_object_clones":3,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5,0,9,4,13],"acting":[1,2,5,0,9,4,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"9","length":"9"},{"start":"13","length":"3"}]},{"pgid":"2.4","version":"0'0","reported_seq":57,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-05T04:54:06.049591+0000","last_change":"2024-02-05T04:53:12.032853+0000","last_active":"2024-02-05T04:54:06.049591+0000","last_peered":"2024-02-05T04:54:06.049591+0000","last_clean":"2024-02-05T04:54:06.049591+0000","last_became_active":"2024-02-05T04:53:12.032664+0000","last_became_peered":"2024-02-05T04:53:12.032664+0000","last_unstale":"2024-02-05T04:54:06.049591+0000","last_undegraded":"2024-02-05T04:54:06.049591+0000","last_fullsized":"2024-02-05T04:54:06.049591+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_clean_scrub_stamp":"2024-02-05T04:53:11.017155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:14.638056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"75'65","reported_seq":285,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-05T04:54:07.067192+0000","last_change":"2024-02-05T04:54:07.067892+0000","last_active":"2024-02-05T04:54:07.067192+0000","last_peered":"2024-02-05T04:54:07.067192+0000","last_clean":"2024-02-05T04:54:07.067192+0000","last_became_active":"2024-02-05T04:53:54.992715+0000","last_became_peered":"2024-02-05T04:53:54.992715+0000","last_unstale":"2024-02-05T04:54:07.067192+0000","last_undegraded":"2024-02-05T04:54:07.067192+0000","last_fullsized":"2024-02-05T04:54:07.067192+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'42","last_scrub_stamp":"2024-02-05T04:53:55.638186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:55.638186+0000","objects_scrubbed":6,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:55:14.892149+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.017413310000000001,"stat_sum":{"num_bytes":7716864,"num_objects":7,"num_object_clones":4,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":2,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":1904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13,5,3,7,1,6],"acting":[15,13,5,3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"9","length":"9"},{"start":"13","length":"3"}]},{"pgid":"3.3","version":"79'55","reported_seq":257,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-05T04:54:11.098038+0000","last_change":"2024-02-05T04:54:10.071055+0000","last_active":"2024-02-05T04:54:11.098038+0000","last_peered":"2024-02-05T04:54:11.098038+0000","last_clean":"2024-02-05T04:54:11.098038+0000","last_became_active":"2024-02-05T04:53:54.990470+0000","last_became_peered":"2024-02-05T04:53:54.990470+0000","last_unstale":"2024-02-05T04:54:11.098038+0000","last_undegraded":"2024-02-05T04:54:11.098038+0000","last_fullsized":"2024-02-05T04:54:11.098038+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'37","last_scrub_stamp":"2024-02-05T04:53:56.868534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:56.868534+0000","objects_scrubbed":1,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:55:24.972294+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00055099800000000003,"stat_sum":{"num_bytes":6193152,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":11,"num_write_kb":4112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1,15,2,5,3],"acting":[4,7,1,15,2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"b","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"15","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":60,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-05T04:54:10.062964+0000","last_change":"2024-02-05T04:53:12.035901+0000","last_active":"2024-02-05T04:54:10.062964+0000","last_peered":"2024-02-05T04:54:10.062964+0000","last_clean":"2024-02-05T04:54:10.062964+0000","last_became_active":"2024-02-05T04:53:12.035739+0000","last_became_peered":"2024-02-05T04:53:12.035739+0000","last_unstale":"2024-02-05T04:54:10.062964+0000","last_undegraded":"2024-02-05T04:54:10.062964+0000","last_fullsized":"2024-02-05T04:54:10.062964+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_clean_scrub_stamp":"2024-02-05T04:53:11.017155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:14.062467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9],"acting":[3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"63'9","reported_seq":222,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-05T04:54:07.047531+0000","last_change":"2024-02-05T04:54:07.048116+0000","last_active":"2024-02-05T04:54:07.047531+0000","last_peered":"2024-02-05T04:54:07.047531+0000","last_clean":"2024-02-05T04:54:07.047531+0000","last_became_active":"2024-02-05T04:53:54.996542+0000","last_became_peered":"2024-02-05T04:53:54.996542+0000","last_unstale":"2024-02-05T04:54:07.047531+0000","last_undegraded":"2024-02-05T04:54:07.047531+0000","last_fullsized":"2024-02-05T04:54:07.047531+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-05T04:53:57.002316+0000","last_deep_scrub":"63'9","last_deep_scrub_stamp":"2024-02-05T04:53:57.002316+0000","last_clean_scrub_stamp":"2024-02-05T04:53:57.002316+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:55:04.761610+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0.00078250200000000002,"stat_sum":{"num_bytes":1884160,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,2,15,0,12,6,9],"acting":[14,2,15,0,12,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"9","length":"9"},{"start":"13","length":"3"}]},{"pgid":"2.1","version":"0'0","reported_seq":35,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-05T04:54:10.070570+0000","last_change":"2024-02-05T04:53:46.969248+0000","last_active":"2024-02-05T04:54:10.070570+0000","last_peered":"2024-02-05T04:54:10.070570+0000","last_clean":"2024-02-05T04:54:10.070570+0000","last_became_active":"2024-02-05T04:53:46.969047+0000","last_became_peered":"2024-02-05T04:53:46.969047+0000","last_unstale":"2024-02-05T04:54:10.070570+0000","last_undegraded":"2024-02-05T04:54:10.070570+0000","last_fullsized":"2024-02-05T04:54:10.070570+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_clean_scrub_stamp":"2024-02-05T04:53:11.017155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:15.558091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1","version":"75'84","reported_seq":320,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-05T04:54:07.081659+0000","last_change":"2024-02-05T04:54:07.082414+0000","last_active":"2024-02-05T04:54:07.081659+0000","last_peered":"2024-02-05T04:54:07.081659+0000","last_clean":"2024-02-05T04:54:07.081659+0000","last_became_active":"2024-02-05T04:53:54.998667+0000","last_became_peered":"2024-02-05T04:53:54.998667+0000","last_unstale":"2024-02-05T04:54:07.081659+0000","last_undegraded":"2024-02-05T04:54:07.081659+0000","last_fullsized":"2024-02-05T04:54:07.081659+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'72","last_scrub_stamp":"2024-02-05T04:54:03.657329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:54:03.657329+0000","objects_scrubbed":9,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:55:22.365283+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.027694667999999999,"stat_sum":{"num_bytes":6586368,"num_objects":9,"num_object_clones":6,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,11,4,10,15,2,14],"acting":[13,11,4,10,15,2,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"9","length":"9"},{"start":"13","length":"3"}]},{"pgid":"2.0","version":"0'0","reported_seq":60,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-05T04:54:10.062319+0000","last_change":"2024-02-05T04:53:12.040500+0000","last_active":"2024-02-05T04:54:10.062319+0000","last_peered":"2024-02-05T04:54:10.062319+0000","last_clean":"2024-02-05T04:54:10.062319+0000","last_became_active":"2024-02-05T04:53:12.040297+0000","last_became_peered":"2024-02-05T04:53:12.040297+0000","last_unstale":"2024-02-05T04:54:10.062319+0000","last_undegraded":"2024-02-05T04:54:10.062319+0000","last_fullsized":"2024-02-05T04:54:10.062319+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_clean_scrub_stamp":"2024-02-05T04:53:11.017155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:20.053075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"75'75","reported_seq":306,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-05T04:54:08.064539+0000","last_change":"2024-02-05T04:54:07.053983+0000","last_active":"2024-02-05T04:54:08.064539+0000","last_peered":"2024-02-05T04:54:08.064539+0000","last_clean":"2024-02-05T04:54:08.064539+0000","last_became_active":"2024-02-05T04:53:54.991703+0000","last_became_peered":"2024-02-05T04:53:54.991703+0000","last_unstale":"2024-02-05T04:54:08.064539+0000","last_undegraded":"2024-02-05T04:54:08.064539+0000","last_fullsized":"2024-02-05T04:54:08.064539+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'53","last_scrub_stamp":"2024-02-05T04:53:55.645439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:55.645439+0000","objects_scrubbed":2,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:55:16.568184+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.010768246,"stat_sum":{"num_bytes":8863744,"num_objects":5,"num_object_clones":3,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":7,"num_read_kb":3746,"num_write":7,"num_write_kb":5840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,2,3,6,15,10,11],"acting":[12,2,3,6,15,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"9","length":"9"},{"start":"13","length":"3"}]},{"pgid":"2.3","version":"0'0","reported_seq":60,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-05T04:54:10.070424+0000","last_change":"2024-02-05T04:53:12.028466+0000","last_active":"2024-02-05T04:54:10.070424+0000","last_peered":"2024-02-05T04:54:10.070424+0000","last_clean":"2024-02-05T04:54:10.070424+0000","last_became_active":"2024-02-05T04:53:12.028138+0000","last_became_peered":"2024-02-05T04:53:12.028138+0000","last_unstale":"2024-02-05T04:54:10.070424+0000","last_undegraded":"2024-02-05T04:54:10.070424+0000","last_fullsized":"2024-02-05T04:54:10.070424+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_clean_scrub_stamp":"2024-02-05T04:53:11.017155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:38.427375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,15],"acting":[4,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"26'157","reported_seq":249,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-05T04:54:07.047797+0000","last_change":"2024-02-05T04:53:49.948646+0000","last_active":"2024-02-05T04:54:07.047797+0000","last_peered":"2024-02-05T04:54:07.047797+0000","last_clean":"2024-02-05T04:54:07.047797+0000","last_became_active":"2024-02-05T04:53:49.948231+0000","last_became_peered":"2024-02-05T04:53:49.948231+0000","last_unstale":"2024-02-05T04:54:07.047797+0000","last_undegraded":"2024-02-05T04:54:07.047797+0000","last_fullsized":"2024-02-05T04:54:07.047797+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:07.900741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:07.900741+0000","last_clean_scrub_stamp":"2024-02-05T04:53:07.900741+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:29.642907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"74'37","reported_seq":240,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-05T04:54:11.085221+0000","last_change":"2024-02-05T04:54:10.070981+0000","last_active":"2024-02-05T04:54:11.085221+0000","last_peered":"2024-02-05T04:54:11.085221+0000","last_clean":"2024-02-05T04:54:11.085221+0000","last_became_active":"2024-02-05T04:53:54.985712+0000","last_became_peered":"2024-02-05T04:53:54.985712+0000","last_unstale":"2024-02-05T04:54:11.085221+0000","last_undegraded":"2024-02-05T04:54:11.085221+0000","last_fullsized":"2024-02-05T04:54:11.085221+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'27","last_scrub_stamp":"2024-02-05T04:53:55.900246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:55.900246+0000","objects_scrubbed":4,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:55:14.593511+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00054530900000000001,"stat_sum":{"num_bytes":3211264,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,0,14,15,6],"acting":[4,1,2,0,14,15,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"b","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"15","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"3.9","version":"75'25","reported_seq":225,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-05T04:54:07.075123+0000","last_change":"2024-02-05T04:54:07.050391+0000","last_active":"2024-02-05T04:54:07.075123+0000","last_peered":"2024-02-05T04:54:07.075123+0000","last_clean":"2024-02-05T04:54:07.075123+0000","last_became_active":"2024-02-05T04:53:54.994390+0000","last_became_peered":"2024-02-05T04:53:54.994390+0000","last_unstale":"2024-02-05T04:54:07.075123+0000","last_undegraded":"2024-02-05T04:54:07.075123+0000","last_fullsized":"2024-02-05T04:54:07.075123+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'21","last_scrub_stamp":"2024-02-05T04:53:55.970267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:55.970267+0000","objects_scrubbed":1,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:55:15.810158+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.002425511,"stat_sum":{"num_bytes":524288,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0,13,3,15,9,10],"acting":[14,0,13,3,15,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"9","length":"9"},{"start":"13","length":"3"}]},{"pgid":"3.a","version":"0'0","reported_seq":190,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-05T04:54:11.062958+0000","last_change":"2024-02-05T04:54:10.063292+0000","last_active":"2024-02-05T04:54:11.062958+0000","last_peered":"2024-02-05T04:54:11.062958+0000","last_clean":"2024-02-05T04:54:11.062958+0000","last_became_active":"2024-02-05T04:53:46.968151+0000","last_became_peered":"2024-02-05T04:53:46.968151+0000","last_unstale":"2024-02-05T04:54:11.062958+0000","last_undegraded":"2024-02-05T04:54:11.062958+0000","last_fullsized":"2024-02-05T04:54:11.062958+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:23.066897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:34.300771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080204099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,10,9,14,6,11],"acting":[3,7,10,9,14,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"b","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"15","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"3.b","version":"75'105","reported_seq":344,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-05T04:54:07.071586+0000","last_change":"2024-02-05T04:54:07.072264+0000","last_active":"2024-02-05T04:54:07.071586+0000","last_peered":"2024-02-05T04:54:07.071586+0000","last_clean":"2024-02-05T04:54:07.071586+0000","last_became_active":"2024-02-05T04:53:49.953645+0000","last_became_peered":"2024-02-05T04:53:49.953645+0000","last_unstale":"2024-02-05T04:54:07.071586+0000","last_undegraded":"2024-02-05T04:54:07.071586+0000","last_fullsized":"2024-02-05T04:54:07.071586+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:23.066897+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:44.803323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.017960402,"stat_sum":{"num_bytes":12271616,"num_objects":11,"num_object_clones":6,"num_object_copies":77,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":58,"num_read_kb":18358,"num_write":46,"num_write_kb":14368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":8683520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,1,10,0,6,7,12],"acting":[13,1,10,0,6,7,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"9","length":"9"},{"start":"13","length":"3"}]},{"pgid":"3.c","version":"74'35","reported_seq":209,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-05T04:54:08.058636+0000","last_change":"2024-02-05T04:54:07.043984+0000","last_active":"2024-02-05T04:54:08.058636+0000","last_peered":"2024-02-05T04:54:08.058636+0000","last_clean":"2024-02-05T04:54:08.058636+0000","last_became_active":"2024-02-05T04:53:24.723258+0000","last_became_peered":"2024-02-05T04:53:24.723258+0000","last_unstale":"2024-02-05T04:54:08.058636+0000","last_undegraded":"2024-02-05T04:54:08.058636+0000","last_fullsized":"2024-02-05T04:54:08.058636+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:23.066897+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:34.536310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048089999999999998,"stat_sum":{"num_bytes":5406720,"num_objects":3,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":10,"num_read_kb":197,"num_write":14,"num_write_kb":5280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,13,5,10,6,1,9],"acting":[11,13,5,10,6,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"9","length":"9"},{"start":"13","length":"3"}]},{"pgid":"3.d","version":"74'54","reported_seq":239,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-05T04:54:08.066261+0000","last_change":"2024-02-05T04:54:07.044209+0000","last_active":"2024-02-05T04:54:08.066261+0000","last_peered":"2024-02-05T04:54:08.066261+0000","last_clean":"2024-02-05T04:54:08.066261+0000","last_became_active":"2024-02-05T04:53:24.723128+0000","last_became_peered":"2024-02-05T04:53:24.723128+0000","last_unstale":"2024-02-05T04:54:08.066261+0000","last_undegraded":"2024-02-05T04:54:08.066261+0000","last_fullsized":"2024-02-05T04:54:08.066261+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:23.066897+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:46.155842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051377900000000002,"stat_sum":{"num_bytes":10616832,"num_objects":7,"num_object_clones":3,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":29,"num_read_kb":17099,"num_write":43,"num_write_kb":8928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,14,10,9,13,5],"acting":[7,3,14,10,9,13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"9","length":"9"},{"start":"13","length":"3"}]},{"pgid":"3.e","version":"75'70","reported_seq":279,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-05T04:54:08.054599+0000","last_change":"2024-02-05T04:54:07.063657+0000","last_active":"2024-02-05T04:54:08.054599+0000","last_peered":"2024-02-05T04:54:08.054599+0000","last_clean":"2024-02-05T04:54:08.054599+0000","last_became_active":"2024-02-05T04:53:49.949549+0000","last_became_peered":"2024-02-05T04:53:49.949549+0000","last_unstale":"2024-02-05T04:54:08.054599+0000","last_undegraded":"2024-02-05T04:54:08.054599+0000","last_fullsized":"2024-02-05T04:54:08.054599+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:23.066897+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:52.595324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015855462000000001,"stat_sum":{"num_bytes":8945664,"num_objects":6,"num_object_clones":4,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":8155,"num_write":52,"num_write_kb":10880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":6209536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7,11,13,6,5],"acting":[2,4,7,11,13,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"9","length":"9"},{"start":"13","length":"3"}]},{"pgid":"3.f","version":"75'82","reported_seq":273,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-05T04:54:08.062235+0000","last_change":"2024-02-05T04:54:07.044349+0000","last_active":"2024-02-05T04:54:08.062235+0000","last_peered":"2024-02-05T04:54:08.062235+0000","last_clean":"2024-02-05T04:54:08.062235+0000","last_became_active":"2024-02-05T04:53:24.723119+0000","last_became_peered":"2024-02-05T04:53:24.723119+0000","last_unstale":"2024-02-05T04:54:08.062235+0000","last_undegraded":"2024-02-05T04:54:08.062235+0000","last_fullsized":"2024-02-05T04:54:08.062235+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:23.066897+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:39.467476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073674999999999999,"stat_sum":{"num_bytes":12599296,"num_objects":14,"num_object_clones":7,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":1,"num_read":51,"num_read_kb":18626,"num_write":39,"num_write_kb":12800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,14,4,0,10,1],"acting":[11,2,14,4,0,10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"9","length":"9"},{"start":"13","length":"3"}]},{"pgid":"3.10","version":"75'11","reported_seq":76,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-05T04:54:08.058681+0000","last_change":"2024-02-05T04:54:07.043783+0000","last_active":"2024-02-05T04:54:08.058681+0000","last_peered":"2024-02-05T04:54:08.058681+0000","last_clean":"2024-02-05T04:54:08.058681+0000","last_became_active":"2024-02-05T04:53:59.961184+0000","last_became_peered":"2024-02-05T04:53:59.961184+0000","last_unstale":"2024-02-05T04:54:08.058681+0000","last_undegraded":"2024-02-05T04:54:08.058681+0000","last_fullsized":"2024-02-05T04:54:08.058681+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":5,"last_scrub":"34'5","last_scrub_stamp":"2024-02-05T04:54:05.647545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:54:05.647545+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:55:20.904873+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00077870700000000001,"stat_sum":{"num_bytes":2244608,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,6,7,1,5,9,11],"acting":[12,6,7,1,5,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"9","length":"9"},{"start":"13","length":"3"}]},{"pgid":"3.11","version":"61'51","reported_seq":80,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-05T04:54:08.051697+0000","last_change":"2024-02-05T04:54:07.040108+0000","last_active":"2024-02-05T04:54:08.051697+0000","last_peered":"2024-02-05T04:54:08.051697+0000","last_clean":"2024-02-05T04:54:08.051697+0000","last_became_active":"2024-02-05T04:53:59.956213+0000","last_became_peered":"2024-02-05T04:53:59.956213+0000","last_unstale":"2024-02-05T04:54:08.051697+0000","last_undegraded":"2024-02-05T04:54:08.051697+0000","last_fullsized":"2024-02-05T04:54:08.051697+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":5,"last_scrub":"61'51","last_scrub_stamp":"2024-02-05T04:54:03.753770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:54:03.753770+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:55:28.707278+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00079129299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,15,0,5,4,12],"acting":[10,11,15,0,5,4,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"9","length":"9"},{"start":"13","length":"3"}]},{"pgid":"3.12","version":"72'61","reported_seq":135,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-05T04:54:08.054477+0000","last_change":"2024-02-05T04:54:07.048770+0000","last_active":"2024-02-05T04:54:08.054477+0000","last_peered":"2024-02-05T04:54:08.054477+0000","last_clean":"2024-02-05T04:54:08.054477+0000","last_became_active":"2024-02-05T04:53:59.956704+0000","last_became_peered":"2024-02-05T04:53:59.956704+0000","last_unstale":"2024-02-05T04:54:08.054477+0000","last_undegraded":"2024-02-05T04:54:08.054477+0000","last_fullsized":"2024-02-05T04:54:08.054477+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":5,"last_scrub":"71'59","last_scrub_stamp":"2024-02-05T04:54:03.684079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:54:03.684079+0000","objects_scrubbed":4,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:55:03.871448+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.00089997500000000002,"stat_sum":{"num_bytes":557056,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":80,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5,6,11,3,15],"acting":[2,7,5,6,11,3,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"9","length":"9"},{"start":"13","length":"3"}]},{"pgid":"3.13","version":"74'48","reported_seq":245,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-05T04:54:08.058599+0000","last_change":"2024-02-05T04:54:07.044366+0000","last_active":"2024-02-05T04:54:08.058599+0000","last_peered":"2024-02-05T04:54:08.058599+0000","last_clean":"2024-02-05T04:54:08.058599+0000","last_became_active":"2024-02-05T04:53:59.956458+0000","last_became_peered":"2024-02-05T04:53:59.956458+0000","last_unstale":"2024-02-05T04:54:08.058599+0000","last_undegraded":"2024-02-05T04:54:08.058599+0000","last_fullsized":"2024-02-05T04:54:08.058599+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":5,"last_scrub":"72'46","last_scrub_stamp":"2024-02-05T04:54:04.649735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:54:04.649735+0000","objects_scrubbed":3,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:55:11.798362+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.001021164,"stat_sum":{"num_bytes":1277952,"num_objects":3,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,15,1,5,0,10],"acting":[11,7,15,1,5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"9","length":"9"},{"start":"13","length":"3"}]}],"pool_stats":[{"poolid":3,"num_pg":26,"stat_sum":{"num_bytes":118194176,"num_objects":108,"num_object_clones":60,"num_object_copies":756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":108,"num_whiteouts":7,"num_read":184,"num_read_kb":66267,"num_write":250,"num_write_kb":70192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":14893056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":230416384,"data_stored":327794688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1284,"ondisk_log_size":1284,"up":182,"acting":182,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":12,"up_from":23,"seq":98784247823,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44828,"kb_used_data":13644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94327012,"statfs":{"total":96636764160,"available":96590860288,"internally_reserved":0,"allocated":13971456,"data_stored":18188583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32500000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55700000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38500000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28899999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57599999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45300000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41099999999999998}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35399999999999998}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69499999999999995}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59399999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70499999999999996}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54800000000000004}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]}]},{"osd":11,"up_from":23,"seq":98784247825,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46040,"kb_used_data":14856,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325800,"statfs":{"total":96636764160,"available":96589619200,"internally_reserved":0,"allocated":15212544,"data_stored":19589415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,4,5,6,7,8,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65100000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56499999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41599999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38900000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.496}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.505}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66200000000000003}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.755}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84799999999999998}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67900000000000005}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84499999999999997}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71899999999999997}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82599999999999996}]}]},{"osd":10,"up_from":23,"seq":98784247825,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48956,"kb_used_data":17772,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322884,"statfs":{"total":96636764160,"available":96586633216,"internally_reserved":0,"allocated":18198528,"data_stored":24549671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,4,5,8,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.312}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.435}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39900000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46300000000000002}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45000000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59499999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38600000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88500000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]}]},{"osd":3,"up_from":21,"seq":90194313233,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44664,"kb_used_data":13476,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94327176,"statfs":{"total":96636764160,"available":96591028224,"internally_reserved":0,"allocated":13799424,"data_stored":16700426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,2,4,6,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34399999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55600000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.629}]},{"osd":4,"last update":"Mon Feb 5 04:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.36599999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]},{"osd":6,"last update":"Mon Feb 5 04:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.56799999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.314}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48899999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48299999999999998}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65800000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.439}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55500000000000005}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68999999999999995}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]}]},{"osd":4,"up_from":21,"seq":90194313232,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42512,"kb_used_data":15492,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94329328,"statfs":{"total":96636764160,"available":96593231872,"internally_reserved":0,"allocated":15863808,"data_stored":23304681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,9,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.22700000000000001}]},{"osd":1,"last update":"Mon Feb 5 04:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.53200000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41899999999999998}]},{"osd":3,"last update":"Mon Feb 5 04:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.59199999999999997}]},{"osd":5,"last update":"Mon Feb 5 04:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":2.9079999999999999,"5min":2.9079999999999999,"15min":2.9079999999999999},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.171,"5min":1.171,"15min":1.171},"last":0.51300000000000001}]},{"osd":6,"last update":"Mon Feb 5 04:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.379},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":3.0179999999999998,"5min":3.0179999999999998,"15min":3.0179999999999998},"last":0.502}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48399999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51000000000000001}]},{"osd":9,"last update":"Mon Feb 5 04:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":2.9849999999999999,"5min":2.9849999999999999,"15min":2.9849999999999999},"last":0.52200000000000002}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":14,"last update":"Mon Feb 5 04:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.073,"5min":1.073,"15min":1.073},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":3.0470000000000002,"5min":3.0470000000000002,"15min":3.0470000000000002},"last":0.44500000000000001}]},{"osd":15,"last update":"Mon Feb 5 04:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.903,"5min":1.903,"15min":1.903},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.66700000000000004}]}]},{"osd":5,"up_from":21,"seq":90194313232,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46528,"kb_used_data":15340,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325312,"statfs":{"total":96636764160,"available":96589119488,"internally_reserved":0,"allocated":15708160,"data_stored":22786538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,6,7,9,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 04:54:11 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"last":0.32700000000000001}]},{"osd":3,"last update":"Mon Feb 5 04:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.432},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.45800000000000002}]},{"osd":4,"last update":"Mon Feb 5 04:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.41099999999999998}]},{"osd":6,"last update":"Mon Feb 5 04:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.59399999999999997}]},{"osd":7,"last update":"Mon Feb 5 04:54:11 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.498}]},{"osd":9,"last update":"Mon Feb 5 04:54:11 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.54200000000000004}]},{"osd":12,"last update":"Mon Feb 5 04:54:11 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.55700000000000005}]},{"osd":13,"last update":"Mon Feb 5 04:54:11 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.57099999999999995}]},{"osd":14,"last update":"Mon Feb 5 04:54:11 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.069,"5min":1.069,"15min":1.069},"last":0.67500000000000004}]},{"osd":15,"last update":"Mon Feb 5 04:54:11 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.58199999999999996}]}]},{"osd":6,"up_from":21,"seq":90194313233,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48636,"kb_used_data":17448,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94323204,"statfs":{"total":96636764160,"available":96586960896,"internally_reserved":0,"allocated":17866752,"data_stored":24715754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,7,8,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52100000000000002}]},{"osd":1,"last update":"Mon Feb 5 04:54:08 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.379},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.28199999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.308}]},{"osd":3,"last update":"Mon Feb 5 04:54:08 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.68700000000000006}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41699999999999998}]},{"osd":5,"last update":"Mon Feb 5 04:54:08 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.36499999999999999}]},{"osd":7,"last update":"Mon Feb 5 04:54:11 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.63900000000000001}]},{"osd":8,"last update":"Mon Feb 5 04:54:11 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.50800000000000001}]},{"osd":9,"last update":"Mon Feb 5 04:54:08 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.496},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.48199999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.625}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61299999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59499999999999997}]},{"osd":14,"last update":"Mon Feb 5 04:54:08 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.56200000000000006}]},{"osd":15,"last update":"Mon Feb 5 04:54:08 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.097,"5min":1.097,"15min":1.097},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.64900000000000002}]}]},{"osd":1,"up_from":22,"seq":94489280528,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44836,"kb_used_data":17816,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94327004,"statfs":{"total":96636764160,"available":96590852096,"internally_reserved":0,"allocated":18243584,"data_stored":25792854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,4,5,8,9,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.5}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63200000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.312}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.626}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.436}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49099999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52400000000000002}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59299999999999997}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66500000000000004}]}]},{"osd":14,"up_from":22,"seq":94489280527,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40196,"kb_used_data":13176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94331644,"statfs":{"total":96636764160,"available":96595603456,"internally_reserved":0,"allocated":13492224,"data_stored":15266134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,6,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48899999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63300000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40500000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59499999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39000000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68200000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47199999999999998}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72299999999999998}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34599999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70499999999999996}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77000000000000002}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68899999999999995}]}]},{"osd":9,"up_from":22,"seq":94489280527,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38808,"kb_used_data":11788,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94333032,"statfs":{"total":96636764160,"available":96597024768,"internally_reserved":0,"allocated":12070912,"data_stored":14942550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,4,6,7,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44700000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48899999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30599999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56699999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.628}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66700000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68200000000000005}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71099999999999997}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77600000000000002}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81200000000000006}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82799999999999996}]}]},{"osd":13,"up_from":22,"seq":94489280526,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50584,"kb_used_data":19404,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94321256,"statfs":{"total":96636764160,"available":96584966144,"internally_reserved":0,"allocated":19869696,"data_stored":30208342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,4,5,6,7,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40999999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34100000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59699999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64200000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39400000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.5}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54900000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69499999999999995}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38200000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.505}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45700000000000002}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71899999999999997}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65800000000000003}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68400000000000005}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70699999999999996}]}]},{"osd":0,"up_from":23,"seq":98784247824,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46568,"kb_used_data":15384,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325272,"statfs":{"total":96636764160,"available":96589078528,"internally_reserved":0,"allocated":15753216,"data_stored":20683047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 04:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":1.4259999999999999,"5min":1.4259999999999999,"15min":1.4259999999999999},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.33300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47299999999999998}]},{"osd":3,"last update":"Mon Feb 5 04:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.6759999999999999,"5min":1.6759999999999999,"15min":1.6759999999999999},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.442}]},{"osd":4,"last update":"Mon Feb 5 04:54:08 2024","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.30099999999999999}]},{"osd":5,"last update":"Mon Feb 5 04:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.374},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":1.5660000000000001,"5min":1.5660000000000001,"15min":1.5660000000000001},"last":0.36099999999999999}]},{"osd":6,"last update":"Mon Feb 5 04:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":2.4929999999999999,"5min":2.4929999999999999,"15min":2.4929999999999999},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.284,"5min":1.284,"15min":1.284},"last":0.64900000000000002}]},{"osd":7,"last update":"Mon Feb 5 04:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.7549999999999999,"5min":1.7549999999999999,"15min":1.7549999999999999},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":2.7629999999999999,"5min":2.7629999999999999,"15min":2.7629999999999999},"last":0.46700000000000003}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.376}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38600000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45400000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60199999999999998}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77700000000000002}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]},{"osd":15,"last update":"Mon Feb 5 04:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.7150000000000001,"5min":1.7150000000000001,"15min":1.7150000000000001},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.351,"5min":1.351,"15min":1.351},"last":0.48899999999999999}]}]},{"osd":15,"up_from":22,"seq":94489280528,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41132,"kb_used_data":14112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94330708,"statfs":{"total":96636764160,"available":96594644992,"internally_reserved":0,"allocated":14450688,"data_stored":19296598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7,8,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54100000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83699999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69399999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.625}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76900000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64600000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.872}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79200000000000004}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82199999999999995}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71499999999999997}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91900000000000004}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98299999999999998}]}]},{"osd":2,"up_from":23,"seq":98784247825,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46404,"kb_used_data":15224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325436,"statfs":{"total":96636764160,"available":96589246464,"internally_reserved":0,"allocated":15589376,"data_stored":21408583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57699999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41699999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59399999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.372}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48099999999999998}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65100000000000002}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78800000000000003}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38200000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63800000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60599999999999998}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.747}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73199999999999998}]}]},{"osd":7,"up_from":23,"seq":98784247825,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46248,"kb_used_data":15068,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325592,"statfs":{"total":96636764160,"available":96589406208,"internally_reserved":0,"allocated":15429632,"data_stored":21897558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41899999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75700000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66900000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96199999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72299999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0009999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89400000000000002}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79100000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98699999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67600000000000005}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91200000000000003}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85699999999999998}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97599999999999998}]}]},{"osd":8,"up_from":0,"seq":98784247822,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":14671872,"data_stored":20070400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":17715200,"data_stored":26550272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":13975552,"data_stored":20267008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12136448,"data_stored":15527936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":15900672,"data_stored":25223168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":14577664,"data_stored":22142976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16736256,"data_stored":24072192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":14905344,"data_stored":22351872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":9826304,"data_stored":12816384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":11005952,"data_stored":14340096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":17117184,"data_stored":23937024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":14131200,"data_stored":18976768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":12890112,"data_stored":17575936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":18804736,"data_stored":29605888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":12636160,"data_stored":15642624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":13385728,"data_stored":18694144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T04:54:12.404 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-05T04:54:12.439 INFO:tasks.ceph.osd.10.smithi150.stderr:2024-02-05T04:54:12.438+0000 7f3075a4d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 34018) UID: 0 2024-02-05T04:54:12.501 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:12.501 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:12.509 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-02-05T04:54:12.537 INFO:tasks.ceph.osd.8.smithi062.stderr:2024-02-05T04:54:12.535+0000 7f9385bf2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 34693) UID: 0 2024-02-05T04:54:12.638 INFO:tasks.ceph.osd.0.smithi062.stderr:2024-02-05T04:54:12.636+0000 7fece8e03640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34688) UID: 0 2024-02-05T04:54:12.655 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:12.655 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:12.664 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-02-05T04:54:12.734 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:54:12.735 INFO:teuthology.orchestra.run.smithi062.stderr:dumped all 2024-02-05T04:54:12.739 INFO:tasks.ceph.osd.10.smithi150.stderr:2024-02-05T04:54:12.738+0000 7f3075a4d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 34018) UID: 0 2024-02-05T04:54:12.745 INFO:teuthology.orchestra.run.smithi062.stdout:{"pg_ready":true,"pg_map":{"version":100,"stamp":"2024-02-05T04:54:12.064816+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":118784544,"num_objects":110,"num_object_clones":60,"num_object_copies":760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":110,"num_whiteouts":7,"num_read":320,"num_read_kb":66385,"num_write":453,"num_write_kb":73652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":15421984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1441,"ondisk_log_size":1441,"up":200,"acting":200,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":200,"num_osds":15,"num_per_pool_osds":15,"num_per_pool_omap_osds":15,"kb":1415577600,"kb_used":676940,"kb_used_data":230000,"kb_used_omap":23,"kb_used_meta":446696,"kb_avail":1414900660,"statfs":{"total":1449551462400,"available":1448858275840,"internally_reserved":0,"allocated":235520000,"data_stored":319330734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23925,"internal_metadata":457417355},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5439488,"num_objects":5,"num_object_clones":6,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":11,"num_read_kb":17720,"num_write":-5,"num_write_kb":5720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-4,"num_bytes_recovered":-3727360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.013319"},"pg_stats":[{"pgid":"3.19","version":"75'31","reported_seq":113,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-05T04:54:07.044424+0000","last_change":"2024-02-05T04:54:07.045572+0000","last_active":"2024-02-05T04:54:07.044424+0000","last_peered":"2024-02-05T04:54:07.044424+0000","last_clean":"2024-02-05T04:54:07.044424+0000","last_became_active":"2024-02-05T04:53:59.958048+0000","last_became_peered":"2024-02-05T04:53:59.958048+0000","last_unstale":"2024-02-05T04:54:07.044424+0000","last_undegraded":"2024-02-05T04:54:07.044424+0000","last_fullsized":"2024-02-05T04:54:07.044424+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":5,"last_scrub":"57'21","last_scrub_stamp":"2024-02-05T04:54:03.585473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:54:03.585473+0000","objects_scrubbed":3,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:55:06.674676+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.005206535,"stat_sum":{"num_bytes":1900544,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,11,12,14,7,6],"acting":[9,1,11,12,14,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"9","length":"9"},{"start":"13","length":"3"}]},{"pgid":"3.18","version":"45'23","reported_seq":80,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-05T04:54:08.058674+0000","last_change":"2024-02-05T04:54:07.043765+0000","last_active":"2024-02-05T04:54:08.058674+0000","last_peered":"2024-02-05T04:54:08.058674+0000","last_clean":"2024-02-05T04:54:08.058674+0000","last_became_active":"2024-02-05T04:53:59.958280+0000","last_became_peered":"2024-02-05T04:53:59.958280+0000","last_unstale":"2024-02-05T04:54:08.058674+0000","last_undegraded":"2024-02-05T04:54:08.058674+0000","last_fullsized":"2024-02-05T04:54:08.058674+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":5,"last_scrub":"45'23","last_scrub_stamp":"2024-02-05T04:54:03.694275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:54:03.694275+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:55:30.533656+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00075505899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,6,1,13,15,7,5],"acting":[12,6,1,13,15,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"9","length":"9"},{"start":"13","length":"3"}]},{"pgid":"3.17","version":"75'43","reported_seq":118,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-05T04:54:08.062637+0000","last_change":"2024-02-05T04:54:07.081867+0000","last_active":"2024-02-05T04:54:08.062637+0000","last_peered":"2024-02-05T04:54:08.062637+0000","last_clean":"2024-02-05T04:54:08.062637+0000","last_became_active":"2024-02-05T04:53:59.955376+0000","last_became_peered":"2024-02-05T04:53:59.955376+0000","last_unstale":"2024-02-05T04:54:08.062637+0000","last_undegraded":"2024-02-05T04:54:08.062637+0000","last_fullsized":"2024-02-05T04:54:08.062637+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":5,"last_scrub":"58'30","last_scrub_stamp":"2024-02-05T04:54:03.827890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:54:03.827890+0000","objects_scrubbed":2,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:55:04.966456+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.023257488999999999,"stat_sum":{"num_bytes":1736704,"num_objects":3,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":2048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,10,4,7,14,11],"acting":[0,3,10,4,7,14,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"9","length":"9"},{"start":"13","length":"3"}]},{"pgid":"3.16","version":"75'36","reported_seq":116,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-05T04:54:08.058666+0000","last_change":"2024-02-05T04:54:07.055115+0000","last_active":"2024-02-05T04:54:08.058666+0000","last_peered":"2024-02-05T04:54:08.058666+0000","last_clean":"2024-02-05T04:54:08.058666+0000","last_became_active":"2024-02-05T04:53:59.956531+0000","last_became_peered":"2024-02-05T04:53:59.956531+0000","last_unstale":"2024-02-05T04:54:08.058666+0000","last_undegraded":"2024-02-05T04:54:08.058666+0000","last_fullsized":"2024-02-05T04:54:08.058666+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":5,"last_scrub":"71'30","last_scrub_stamp":"2024-02-05T04:54:03.634933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:54:03.634933+0000","objects_scrubbed":2,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:55:32.808902+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.010565771,"stat_sum":{"num_bytes":622592,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,15,9,5,10,12,1],"acting":[11,15,9,5,10,12,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"9","length":"9"},{"start":"13","length":"3"}]},{"pgid":"3.15","version":"58'40","reported_seq":80,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-05T04:54:08.058968+0000","last_change":"2024-02-05T04:54:07.044227+0000","last_active":"2024-02-05T04:54:08.058968+0000","last_peered":"2024-02-05T04:54:08.058968+0000","last_clean":"2024-02-05T04:54:08.058968+0000","last_became_active":"2024-02-05T04:53:59.955382+0000","last_became_peered":"2024-02-05T04:53:59.955382+0000","last_unstale":"2024-02-05T04:54:08.058968+0000","last_undegraded":"2024-02-05T04:54:08.058968+0000","last_fullsized":"2024-02-05T04:54:08.058968+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":5,"last_scrub":"58'40","last_scrub_stamp":"2024-02-05T04:54:04.666054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:54:04.666054+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:55:20.921718+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0010679039999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,6,3,4,14,2,15],"acting":[12,6,3,4,14,2,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"9","length":"9"},{"start":"13","length":"3"}]},{"pgid":"3.14","version":"79'87","reported_seq":141,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-05T04:54:11.062977+0000","last_change":"2024-02-05T04:54:10.062997+0000","last_active":"2024-02-05T04:54:11.062977+0000","last_peered":"2024-02-05T04:54:11.062977+0000","last_clean":"2024-02-05T04:54:11.062977+0000","last_became_active":"2024-02-05T04:53:59.957250+0000","last_became_peered":"2024-02-05T04:53:59.957250+0000","last_unstale":"2024-02-05T04:54:11.062977+0000","last_undegraded":"2024-02-05T04:54:11.062977+0000","last_fullsized":"2024-02-05T04:54:11.062977+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":5,"last_scrub":"71'79","last_scrub_stamp":"2024-02-05T04:54:03.639971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:54:03.639971+0000","objects_scrubbed":5,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:55:09.216854+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00061923700000000002,"stat_sum":{"num_bytes":7454720,"num_objects":6,"num_object_clones":4,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,14,15,0,12,9,13],"acting":[3,14,15,0,12,9,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"b","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"15","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":55,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-05T04:54:06.048457+0000","last_change":"2024-02-05T04:53:12.781269+0000","last_active":"2024-02-05T04:54:06.048457+0000","last_peered":"2024-02-05T04:54:06.048457+0000","last_clean":"2024-02-05T04:54:06.048457+0000","last_became_active":"2024-02-05T04:53:12.780970+0000","last_became_peered":"2024-02-05T04:53:12.780970+0000","last_unstale":"2024-02-05T04:54:06.048457+0000","last_undegraded":"2024-02-05T04:54:06.048457+0000","last_fullsized":"2024-02-05T04:54:06.048457+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_clean_scrub_stamp":"2024-02-05T04:53:11.017155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:36.119048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,0],"acting":[15,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.6","version":"75'44","reported_seq":270,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-05T04:54:07.088213+0000","last_change":"2024-02-05T04:54:07.072361+0000","last_active":"2024-02-05T04:54:07.088213+0000","last_peered":"2024-02-05T04:54:07.088213+0000","last_clean":"2024-02-05T04:54:07.088213+0000","last_became_active":"2024-02-05T04:53:54.992358+0000","last_became_peered":"2024-02-05T04:53:54.992358+0000","last_unstale":"2024-02-05T04:54:07.088213+0000","last_undegraded":"2024-02-05T04:54:07.088213+0000","last_fullsized":"2024-02-05T04:54:07.088213+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'26","last_scrub_stamp":"2024-02-05T04:53:55.597534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:55.597534+0000","objects_scrubbed":1,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:55:11.475373+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.017878090999999999,"stat_sum":{"num_bytes":4571136,"num_objects":3,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,12,4,15,1,7,10],"acting":[13,12,4,15,1,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"9","length":"9"},{"start":"13","length":"3"}]},{"pgid":"2.6","version":"0'0","reported_seq":57,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-05T04:54:06.042328+0000","last_change":"2024-02-05T04:53:12.037950+0000","last_active":"2024-02-05T04:54:06.042328+0000","last_peered":"2024-02-05T04:54:06.042328+0000","last_clean":"2024-02-05T04:54:06.042328+0000","last_became_active":"2024-02-05T04:53:12.037750+0000","last_became_peered":"2024-02-05T04:53:12.037750+0000","last_unstale":"2024-02-05T04:54:06.042328+0000","last_undegraded":"2024-02-05T04:54:06.042328+0000","last_fullsized":"2024-02-05T04:54:06.042328+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_clean_scrub_stamp":"2024-02-05T04:53:11.017155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:26.129079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"74'32","reported_seq":220,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-05T04:54:07.054778+0000","last_change":"2024-02-05T04:54:07.055408+0000","last_active":"2024-02-05T04:54:07.054778+0000","last_peered":"2024-02-05T04:54:07.054778+0000","last_clean":"2024-02-05T04:54:07.054778+0000","last_became_active":"2024-02-05T04:53:54.997596+0000","last_became_peered":"2024-02-05T04:53:54.997596+0000","last_unstale":"2024-02-05T04:54:07.054778+0000","last_undegraded":"2024-02-05T04:54:07.054778+0000","last_fullsized":"2024-02-05T04:54:07.054778+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'30","last_scrub_stamp":"2024-02-05T04:53:56.628250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:56.628250+0000","objects_scrubbed":2,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:55:11.669159+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00099417400000000001,"stat_sum":{"num_bytes":3670016,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,6,12,1,4,5,11],"acting":[13,6,12,1,4,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"9","length":"9"},{"start":"13","length":"3"}]},{"pgid":"2.5","version":"0'0","reported_seq":61,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-05T04:54:10.060951+0000","last_change":"2024-02-05T04:53:12.753233+0000","last_active":"2024-02-05T04:54:10.060951+0000","last_peered":"2024-02-05T04:54:10.060951+0000","last_clean":"2024-02-05T04:54:10.060951+0000","last_became_active":"2024-02-05T04:53:12.752915+0000","last_became_peered":"2024-02-05T04:53:12.752915+0000","last_unstale":"2024-02-05T04:54:10.060951+0000","last_undegraded":"2024-02-05T04:54:10.060951+0000","last_fullsized":"2024-02-05T04:54:10.060951+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_clean_scrub_stamp":"2024-02-05T04:53:11.017155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:18.391150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"74'81","reported_seq":319,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-05T04:54:07.043596+0000","last_change":"2024-02-05T04:54:07.044364+0000","last_active":"2024-02-05T04:54:07.043596+0000","last_peered":"2024-02-05T04:54:07.043596+0000","last_clean":"2024-02-05T04:54:07.043596+0000","last_became_active":"2024-02-05T04:53:54.996849+0000","last_became_peered":"2024-02-05T04:53:54.996849+0000","last_unstale":"2024-02-05T04:54:07.043596+0000","last_undegraded":"2024-02-05T04:54:07.043596+0000","last_fullsized":"2024-02-05T04:54:07.043596+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'73","last_scrub_stamp":"2024-02-05T04:53:55.059635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:55.059635+0000","objects_scrubbed":6,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:55:01.074914+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00090901299999999997,"stat_sum":{"num_bytes":9338880,"num_objects":6,"num_object_clones":3,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5,0,9,4,13],"acting":[1,2,5,0,9,4,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"9","length":"9"},{"start":"13","length":"3"}]},{"pgid":"2.4","version":"0'0","reported_seq":57,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-05T04:54:06.049591+0000","last_change":"2024-02-05T04:53:12.032853+0000","last_active":"2024-02-05T04:54:06.049591+0000","last_peered":"2024-02-05T04:54:06.049591+0000","last_clean":"2024-02-05T04:54:06.049591+0000","last_became_active":"2024-02-05T04:53:12.032664+0000","last_became_peered":"2024-02-05T04:53:12.032664+0000","last_unstale":"2024-02-05T04:54:06.049591+0000","last_undegraded":"2024-02-05T04:54:06.049591+0000","last_fullsized":"2024-02-05T04:54:06.049591+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_clean_scrub_stamp":"2024-02-05T04:53:11.017155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:14.638056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"75'65","reported_seq":285,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-05T04:54:07.067192+0000","last_change":"2024-02-05T04:54:07.067892+0000","last_active":"2024-02-05T04:54:07.067192+0000","last_peered":"2024-02-05T04:54:07.067192+0000","last_clean":"2024-02-05T04:54:07.067192+0000","last_became_active":"2024-02-05T04:53:54.992715+0000","last_became_peered":"2024-02-05T04:53:54.992715+0000","last_unstale":"2024-02-05T04:54:07.067192+0000","last_undegraded":"2024-02-05T04:54:07.067192+0000","last_fullsized":"2024-02-05T04:54:07.067192+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'42","last_scrub_stamp":"2024-02-05T04:53:55.638186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:55.638186+0000","objects_scrubbed":6,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:55:14.892149+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.017413310000000001,"stat_sum":{"num_bytes":7716864,"num_objects":7,"num_object_clones":4,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":2,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":1904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13,5,3,7,1,6],"acting":[15,13,5,3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"9","length":"9"},{"start":"13","length":"3"}]},{"pgid":"3.3","version":"79'55","reported_seq":257,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-05T04:54:11.098038+0000","last_change":"2024-02-05T04:54:10.071055+0000","last_active":"2024-02-05T04:54:11.098038+0000","last_peered":"2024-02-05T04:54:11.098038+0000","last_clean":"2024-02-05T04:54:11.098038+0000","last_became_active":"2024-02-05T04:53:54.990470+0000","last_became_peered":"2024-02-05T04:53:54.990470+0000","last_unstale":"2024-02-05T04:54:11.098038+0000","last_undegraded":"2024-02-05T04:54:11.098038+0000","last_fullsized":"2024-02-05T04:54:11.098038+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'37","last_scrub_stamp":"2024-02-05T04:53:56.868534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:56.868534+0000","objects_scrubbed":1,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:55:24.972294+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00055099800000000003,"stat_sum":{"num_bytes":6193152,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":11,"num_write_kb":4112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1,15,2,5,3],"acting":[4,7,1,15,2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"b","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"15","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":60,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-05T04:54:10.062964+0000","last_change":"2024-02-05T04:53:12.035901+0000","last_active":"2024-02-05T04:54:10.062964+0000","last_peered":"2024-02-05T04:54:10.062964+0000","last_clean":"2024-02-05T04:54:10.062964+0000","last_became_active":"2024-02-05T04:53:12.035739+0000","last_became_peered":"2024-02-05T04:53:12.035739+0000","last_unstale":"2024-02-05T04:54:10.062964+0000","last_undegraded":"2024-02-05T04:54:10.062964+0000","last_fullsized":"2024-02-05T04:54:10.062964+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_clean_scrub_stamp":"2024-02-05T04:53:11.017155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:14.062467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9],"acting":[3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"63'9","reported_seq":222,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-05T04:54:07.047531+0000","last_change":"2024-02-05T04:54:07.048116+0000","last_active":"2024-02-05T04:54:07.047531+0000","last_peered":"2024-02-05T04:54:07.047531+0000","last_clean":"2024-02-05T04:54:07.047531+0000","last_became_active":"2024-02-05T04:53:54.996542+0000","last_became_peered":"2024-02-05T04:53:54.996542+0000","last_unstale":"2024-02-05T04:54:07.047531+0000","last_undegraded":"2024-02-05T04:54:07.047531+0000","last_fullsized":"2024-02-05T04:54:07.047531+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-05T04:53:57.002316+0000","last_deep_scrub":"63'9","last_deep_scrub_stamp":"2024-02-05T04:53:57.002316+0000","last_clean_scrub_stamp":"2024-02-05T04:53:57.002316+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:55:04.761610+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0.00078250200000000002,"stat_sum":{"num_bytes":1884160,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,2,15,0,12,6,9],"acting":[14,2,15,0,12,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"9","length":"9"},{"start":"13","length":"3"}]},{"pgid":"2.1","version":"0'0","reported_seq":35,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-05T04:54:10.070570+0000","last_change":"2024-02-05T04:53:46.969248+0000","last_active":"2024-02-05T04:54:10.070570+0000","last_peered":"2024-02-05T04:54:10.070570+0000","last_clean":"2024-02-05T04:54:10.070570+0000","last_became_active":"2024-02-05T04:53:46.969047+0000","last_became_peered":"2024-02-05T04:53:46.969047+0000","last_unstale":"2024-02-05T04:54:10.070570+0000","last_undegraded":"2024-02-05T04:54:10.070570+0000","last_fullsized":"2024-02-05T04:54:10.070570+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_clean_scrub_stamp":"2024-02-05T04:53:11.017155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:15.558091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1","version":"75'84","reported_seq":320,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-05T04:54:07.081659+0000","last_change":"2024-02-05T04:54:07.082414+0000","last_active":"2024-02-05T04:54:07.081659+0000","last_peered":"2024-02-05T04:54:07.081659+0000","last_clean":"2024-02-05T04:54:07.081659+0000","last_became_active":"2024-02-05T04:53:54.998667+0000","last_became_peered":"2024-02-05T04:53:54.998667+0000","last_unstale":"2024-02-05T04:54:07.081659+0000","last_undegraded":"2024-02-05T04:54:07.081659+0000","last_fullsized":"2024-02-05T04:54:07.081659+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'72","last_scrub_stamp":"2024-02-05T04:54:03.657329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:54:03.657329+0000","objects_scrubbed":9,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:55:22.365283+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.027694667999999999,"stat_sum":{"num_bytes":6586368,"num_objects":9,"num_object_clones":6,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,11,4,10,15,2,14],"acting":[13,11,4,10,15,2,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"9","length":"9"},{"start":"13","length":"3"}]},{"pgid":"2.0","version":"0'0","reported_seq":60,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-05T04:54:10.062319+0000","last_change":"2024-02-05T04:53:12.040500+0000","last_active":"2024-02-05T04:54:10.062319+0000","last_peered":"2024-02-05T04:54:10.062319+0000","last_clean":"2024-02-05T04:54:10.062319+0000","last_became_active":"2024-02-05T04:53:12.040297+0000","last_became_peered":"2024-02-05T04:53:12.040297+0000","last_unstale":"2024-02-05T04:54:10.062319+0000","last_undegraded":"2024-02-05T04:54:10.062319+0000","last_fullsized":"2024-02-05T04:54:10.062319+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_clean_scrub_stamp":"2024-02-05T04:53:11.017155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:20.053075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"75'75","reported_seq":306,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-05T04:54:08.064539+0000","last_change":"2024-02-05T04:54:07.053983+0000","last_active":"2024-02-05T04:54:08.064539+0000","last_peered":"2024-02-05T04:54:08.064539+0000","last_clean":"2024-02-05T04:54:08.064539+0000","last_became_active":"2024-02-05T04:53:54.991703+0000","last_became_peered":"2024-02-05T04:53:54.991703+0000","last_unstale":"2024-02-05T04:54:08.064539+0000","last_undegraded":"2024-02-05T04:54:08.064539+0000","last_fullsized":"2024-02-05T04:54:08.064539+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'53","last_scrub_stamp":"2024-02-05T04:53:55.645439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:55.645439+0000","objects_scrubbed":2,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:55:16.568184+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.010768246,"stat_sum":{"num_bytes":8863744,"num_objects":5,"num_object_clones":3,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":7,"num_read_kb":3746,"num_write":7,"num_write_kb":5840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,2,3,6,15,10,11],"acting":[12,2,3,6,15,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"9","length":"9"},{"start":"13","length":"3"}]},{"pgid":"2.3","version":"0'0","reported_seq":60,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-05T04:54:10.070424+0000","last_change":"2024-02-05T04:53:12.028466+0000","last_active":"2024-02-05T04:54:10.070424+0000","last_peered":"2024-02-05T04:54:10.070424+0000","last_clean":"2024-02-05T04:54:10.070424+0000","last_became_active":"2024-02-05T04:53:12.028138+0000","last_became_peered":"2024-02-05T04:53:12.028138+0000","last_unstale":"2024-02-05T04:54:10.070424+0000","last_undegraded":"2024-02-05T04:54:10.070424+0000","last_fullsized":"2024-02-05T04:54:10.070424+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_clean_scrub_stamp":"2024-02-05T04:53:11.017155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:38.427375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,15],"acting":[4,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"26'157","reported_seq":249,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-05T04:54:07.047797+0000","last_change":"2024-02-05T04:53:49.948646+0000","last_active":"2024-02-05T04:54:07.047797+0000","last_peered":"2024-02-05T04:54:07.047797+0000","last_clean":"2024-02-05T04:54:07.047797+0000","last_became_active":"2024-02-05T04:53:49.948231+0000","last_became_peered":"2024-02-05T04:53:49.948231+0000","last_unstale":"2024-02-05T04:54:07.047797+0000","last_undegraded":"2024-02-05T04:54:07.047797+0000","last_fullsized":"2024-02-05T04:54:07.047797+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:07.900741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:07.900741+0000","last_clean_scrub_stamp":"2024-02-05T04:53:07.900741+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:29.642907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"74'37","reported_seq":240,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-05T04:54:11.085221+0000","last_change":"2024-02-05T04:54:10.070981+0000","last_active":"2024-02-05T04:54:11.085221+0000","last_peered":"2024-02-05T04:54:11.085221+0000","last_clean":"2024-02-05T04:54:11.085221+0000","last_became_active":"2024-02-05T04:53:54.985712+0000","last_became_peered":"2024-02-05T04:53:54.985712+0000","last_unstale":"2024-02-05T04:54:11.085221+0000","last_undegraded":"2024-02-05T04:54:11.085221+0000","last_fullsized":"2024-02-05T04:54:11.085221+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'27","last_scrub_stamp":"2024-02-05T04:53:55.900246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:55.900246+0000","objects_scrubbed":4,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:55:14.593511+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00054530900000000001,"stat_sum":{"num_bytes":3211264,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,0,14,15,6],"acting":[4,1,2,0,14,15,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"b","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"15","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"3.9","version":"75'25","reported_seq":225,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-05T04:54:07.075123+0000","last_change":"2024-02-05T04:54:07.050391+0000","last_active":"2024-02-05T04:54:07.075123+0000","last_peered":"2024-02-05T04:54:07.075123+0000","last_clean":"2024-02-05T04:54:07.075123+0000","last_became_active":"2024-02-05T04:53:54.994390+0000","last_became_peered":"2024-02-05T04:53:54.994390+0000","last_unstale":"2024-02-05T04:54:07.075123+0000","last_undegraded":"2024-02-05T04:54:07.075123+0000","last_fullsized":"2024-02-05T04:54:07.075123+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'21","last_scrub_stamp":"2024-02-05T04:53:55.970267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:55.970267+0000","objects_scrubbed":1,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:55:15.810158+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.002425511,"stat_sum":{"num_bytes":524288,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0,13,3,15,9,10],"acting":[14,0,13,3,15,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"9","length":"9"},{"start":"13","length":"3"}]},{"pgid":"3.a","version":"0'0","reported_seq":190,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-05T04:54:11.062958+0000","last_change":"2024-02-05T04:54:10.063292+0000","last_active":"2024-02-05T04:54:11.062958+0000","last_peered":"2024-02-05T04:54:11.062958+0000","last_clean":"2024-02-05T04:54:11.062958+0000","last_became_active":"2024-02-05T04:53:46.968151+0000","last_became_peered":"2024-02-05T04:53:46.968151+0000","last_unstale":"2024-02-05T04:54:11.062958+0000","last_undegraded":"2024-02-05T04:54:11.062958+0000","last_fullsized":"2024-02-05T04:54:11.062958+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:23.066897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:34.300771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080204099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,10,9,14,6,11],"acting":[3,7,10,9,14,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"b","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"15","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"3.b","version":"75'105","reported_seq":344,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-05T04:54:07.071586+0000","last_change":"2024-02-05T04:54:07.072264+0000","last_active":"2024-02-05T04:54:07.071586+0000","last_peered":"2024-02-05T04:54:07.071586+0000","last_clean":"2024-02-05T04:54:07.071586+0000","last_became_active":"2024-02-05T04:53:49.953645+0000","last_became_peered":"2024-02-05T04:53:49.953645+0000","last_unstale":"2024-02-05T04:54:07.071586+0000","last_undegraded":"2024-02-05T04:54:07.071586+0000","last_fullsized":"2024-02-05T04:54:07.071586+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:23.066897+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:44.803323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.017960402,"stat_sum":{"num_bytes":12271616,"num_objects":11,"num_object_clones":6,"num_object_copies":77,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":58,"num_read_kb":18358,"num_write":46,"num_write_kb":14368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":8683520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,1,10,0,6,7,12],"acting":[13,1,10,0,6,7,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"9","length":"9"},{"start":"13","length":"3"}]},{"pgid":"3.c","version":"74'35","reported_seq":209,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-05T04:54:08.058636+0000","last_change":"2024-02-05T04:54:07.043984+0000","last_active":"2024-02-05T04:54:08.058636+0000","last_peered":"2024-02-05T04:54:08.058636+0000","last_clean":"2024-02-05T04:54:08.058636+0000","last_became_active":"2024-02-05T04:53:24.723258+0000","last_became_peered":"2024-02-05T04:53:24.723258+0000","last_unstale":"2024-02-05T04:54:08.058636+0000","last_undegraded":"2024-02-05T04:54:08.058636+0000","last_fullsized":"2024-02-05T04:54:08.058636+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:23.066897+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:34.536310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048089999999999998,"stat_sum":{"num_bytes":5406720,"num_objects":3,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":10,"num_read_kb":197,"num_write":14,"num_write_kb":5280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,13,5,10,6,1,9],"acting":[11,13,5,10,6,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"9","length":"9"},{"start":"13","length":"3"}]},{"pgid":"3.d","version":"74'54","reported_seq":239,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-05T04:54:08.066261+0000","last_change":"2024-02-05T04:54:07.044209+0000","last_active":"2024-02-05T04:54:08.066261+0000","last_peered":"2024-02-05T04:54:08.066261+0000","last_clean":"2024-02-05T04:54:08.066261+0000","last_became_active":"2024-02-05T04:53:24.723128+0000","last_became_peered":"2024-02-05T04:53:24.723128+0000","last_unstale":"2024-02-05T04:54:08.066261+0000","last_undegraded":"2024-02-05T04:54:08.066261+0000","last_fullsized":"2024-02-05T04:54:08.066261+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:23.066897+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:46.155842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051377900000000002,"stat_sum":{"num_bytes":10616832,"num_objects":7,"num_object_clones":3,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":29,"num_read_kb":17099,"num_write":43,"num_write_kb":8928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,14,10,9,13,5],"acting":[7,3,14,10,9,13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"9","length":"9"},{"start":"13","length":"3"}]},{"pgid":"3.e","version":"75'70","reported_seq":279,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-05T04:54:08.054599+0000","last_change":"2024-02-05T04:54:07.063657+0000","last_active":"2024-02-05T04:54:08.054599+0000","last_peered":"2024-02-05T04:54:08.054599+0000","last_clean":"2024-02-05T04:54:08.054599+0000","last_became_active":"2024-02-05T04:53:49.949549+0000","last_became_peered":"2024-02-05T04:53:49.949549+0000","last_unstale":"2024-02-05T04:54:08.054599+0000","last_undegraded":"2024-02-05T04:54:08.054599+0000","last_fullsized":"2024-02-05T04:54:08.054599+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:23.066897+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:52.595324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015855462000000001,"stat_sum":{"num_bytes":8945664,"num_objects":6,"num_object_clones":4,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":8155,"num_write":52,"num_write_kb":10880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":6209536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7,11,13,6,5],"acting":[2,4,7,11,13,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"9","length":"9"},{"start":"13","length":"3"}]},{"pgid":"3.f","version":"75'82","reported_seq":273,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-05T04:54:08.062235+0000","last_change":"2024-02-05T04:54:07.044349+0000","last_active":"2024-02-05T04:54:08.062235+0000","last_peered":"2024-02-05T04:54:08.062235+0000","last_clean":"2024-02-05T04:54:08.062235+0000","last_became_active":"2024-02-05T04:53:24.723119+0000","last_became_peered":"2024-02-05T04:53:24.723119+0000","last_unstale":"2024-02-05T04:54:08.062235+0000","last_undegraded":"2024-02-05T04:54:08.062235+0000","last_fullsized":"2024-02-05T04:54:08.062235+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:23.066897+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:39.467476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073674999999999999,"stat_sum":{"num_bytes":12599296,"num_objects":14,"num_object_clones":7,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":1,"num_read":51,"num_read_kb":18626,"num_write":39,"num_write_kb":12800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,14,4,0,10,1],"acting":[11,2,14,4,0,10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"9","length":"9"},{"start":"13","length":"3"}]},{"pgid":"3.10","version":"75'11","reported_seq":76,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-05T04:54:08.058681+0000","last_change":"2024-02-05T04:54:07.043783+0000","last_active":"2024-02-05T04:54:08.058681+0000","last_peered":"2024-02-05T04:54:08.058681+0000","last_clean":"2024-02-05T04:54:08.058681+0000","last_became_active":"2024-02-05T04:53:59.961184+0000","last_became_peered":"2024-02-05T04:53:59.961184+0000","last_unstale":"2024-02-05T04:54:08.058681+0000","last_undegraded":"2024-02-05T04:54:08.058681+0000","last_fullsized":"2024-02-05T04:54:08.058681+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":5,"last_scrub":"34'5","last_scrub_stamp":"2024-02-05T04:54:05.647545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:54:05.647545+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:55:20.904873+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00077870700000000001,"stat_sum":{"num_bytes":2244608,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,6,7,1,5,9,11],"acting":[12,6,7,1,5,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"9","length":"9"},{"start":"13","length":"3"}]},{"pgid":"3.11","version":"61'51","reported_seq":80,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-05T04:54:08.051697+0000","last_change":"2024-02-05T04:54:07.040108+0000","last_active":"2024-02-05T04:54:08.051697+0000","last_peered":"2024-02-05T04:54:08.051697+0000","last_clean":"2024-02-05T04:54:08.051697+0000","last_became_active":"2024-02-05T04:53:59.956213+0000","last_became_peered":"2024-02-05T04:53:59.956213+0000","last_unstale":"2024-02-05T04:54:08.051697+0000","last_undegraded":"2024-02-05T04:54:08.051697+0000","last_fullsized":"2024-02-05T04:54:08.051697+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":5,"last_scrub":"61'51","last_scrub_stamp":"2024-02-05T04:54:03.753770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:54:03.753770+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:55:28.707278+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00079129299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,15,0,5,4,12],"acting":[10,11,15,0,5,4,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"9","length":"9"},{"start":"13","length":"3"}]},{"pgid":"3.12","version":"72'61","reported_seq":135,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-05T04:54:08.054477+0000","last_change":"2024-02-05T04:54:07.048770+0000","last_active":"2024-02-05T04:54:08.054477+0000","last_peered":"2024-02-05T04:54:08.054477+0000","last_clean":"2024-02-05T04:54:08.054477+0000","last_became_active":"2024-02-05T04:53:59.956704+0000","last_became_peered":"2024-02-05T04:53:59.956704+0000","last_unstale":"2024-02-05T04:54:08.054477+0000","last_undegraded":"2024-02-05T04:54:08.054477+0000","last_fullsized":"2024-02-05T04:54:08.054477+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":5,"last_scrub":"71'59","last_scrub_stamp":"2024-02-05T04:54:03.684079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:54:03.684079+0000","objects_scrubbed":4,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:55:03.871448+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.00089997500000000002,"stat_sum":{"num_bytes":557056,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":80,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5,6,11,3,15],"acting":[2,7,5,6,11,3,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"9","length":"9"},{"start":"13","length":"3"}]},{"pgid":"3.13","version":"74'48","reported_seq":245,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-05T04:54:08.058599+0000","last_change":"2024-02-05T04:54:07.044366+0000","last_active":"2024-02-05T04:54:08.058599+0000","last_peered":"2024-02-05T04:54:08.058599+0000","last_clean":"2024-02-05T04:54:08.058599+0000","last_became_active":"2024-02-05T04:53:59.956458+0000","last_became_peered":"2024-02-05T04:53:59.956458+0000","last_unstale":"2024-02-05T04:54:08.058599+0000","last_undegraded":"2024-02-05T04:54:08.058599+0000","last_fullsized":"2024-02-05T04:54:08.058599+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":5,"last_scrub":"72'46","last_scrub_stamp":"2024-02-05T04:54:04.649735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:54:04.649735+0000","objects_scrubbed":3,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:55:11.798362+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.001021164,"stat_sum":{"num_bytes":1277952,"num_objects":3,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,15,1,5,0,10],"acting":[11,7,15,1,5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"9","length":"9"},{"start":"13","length":"3"}]}],"pool_stats":[{"poolid":3,"num_pg":26,"stat_sum":{"num_bytes":118194176,"num_objects":108,"num_object_clones":60,"num_object_copies":756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":108,"num_whiteouts":7,"num_read":184,"num_read_kb":66267,"num_write":250,"num_write_kb":70192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":14893056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":230416384,"data_stored":327794688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1284,"ondisk_log_size":1284,"up":182,"acting":182,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":12,"up_from":23,"seq":98784247823,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44828,"kb_used_data":13644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94327012,"statfs":{"total":96636764160,"available":96590860288,"internally_reserved":0,"allocated":13971456,"data_stored":18188583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32500000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55700000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38500000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28899999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57599999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45300000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41099999999999998}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35399999999999998}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69499999999999995}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59399999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70499999999999996}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54800000000000004}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]}]},{"osd":11,"up_from":23,"seq":98784247825,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46040,"kb_used_data":14856,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325800,"statfs":{"total":96636764160,"available":96589619200,"internally_reserved":0,"allocated":15212544,"data_stored":19589415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,4,5,6,7,8,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65100000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56499999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41599999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38900000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.496}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.505}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66200000000000003}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.755}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84799999999999998}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67900000000000005}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84499999999999997}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71899999999999997}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82599999999999996}]}]},{"osd":10,"up_from":23,"seq":98784247825,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48956,"kb_used_data":17772,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322884,"statfs":{"total":96636764160,"available":96586633216,"internally_reserved":0,"allocated":18198528,"data_stored":24549671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,4,5,8,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.312}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.435}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39900000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46300000000000002}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45000000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59499999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38600000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88500000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]}]},{"osd":3,"up_from":21,"seq":90194313233,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44664,"kb_used_data":13476,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94327176,"statfs":{"total":96636764160,"available":96591028224,"internally_reserved":0,"allocated":13799424,"data_stored":16700426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,2,4,6,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34399999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55600000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.629}]},{"osd":4,"last update":"Mon Feb 5 04:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.36599999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]},{"osd":6,"last update":"Mon Feb 5 04:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.56799999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.314}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48899999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48299999999999998}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65800000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.439}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55500000000000005}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68999999999999995}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]}]},{"osd":4,"up_from":21,"seq":90194313232,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42512,"kb_used_data":15492,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94329328,"statfs":{"total":96636764160,"available":96593231872,"internally_reserved":0,"allocated":15863808,"data_stored":23304681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,9,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.22700000000000001}]},{"osd":1,"last update":"Mon Feb 5 04:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.53200000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41899999999999998}]},{"osd":3,"last update":"Mon Feb 5 04:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.59199999999999997}]},{"osd":5,"last update":"Mon Feb 5 04:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":2.9079999999999999,"5min":2.9079999999999999,"15min":2.9079999999999999},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.171,"5min":1.171,"15min":1.171},"last":0.51300000000000001}]},{"osd":6,"last update":"Mon Feb 5 04:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.379},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":3.0179999999999998,"5min":3.0179999999999998,"15min":3.0179999999999998},"last":0.502}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48399999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51000000000000001}]},{"osd":9,"last update":"Mon Feb 5 04:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":2.9849999999999999,"5min":2.9849999999999999,"15min":2.9849999999999999},"last":0.52200000000000002}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":14,"last update":"Mon Feb 5 04:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.073,"5min":1.073,"15min":1.073},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":3.0470000000000002,"5min":3.0470000000000002,"15min":3.0470000000000002},"last":0.44500000000000001}]},{"osd":15,"last update":"Mon Feb 5 04:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.903,"5min":1.903,"15min":1.903},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.66700000000000004}]}]},{"osd":5,"up_from":21,"seq":90194313232,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46528,"kb_used_data":15340,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325312,"statfs":{"total":96636764160,"available":96589119488,"internally_reserved":0,"allocated":15708160,"data_stored":22786538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,6,7,9,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 04:54:11 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"last":0.32700000000000001}]},{"osd":3,"last update":"Mon Feb 5 04:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.432},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.45800000000000002}]},{"osd":4,"last update":"Mon Feb 5 04:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.41099999999999998}]},{"osd":6,"last update":"Mon Feb 5 04:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.59399999999999997}]},{"osd":7,"last update":"Mon Feb 5 04:54:11 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.498}]},{"osd":9,"last update":"Mon Feb 5 04:54:11 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.54200000000000004}]},{"osd":12,"last update":"Mon Feb 5 04:54:11 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.55700000000000005}]},{"osd":13,"last update":"Mon Feb 5 04:54:11 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.57099999999999995}]},{"osd":14,"last update":"Mon Feb 5 04:54:11 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.069,"5min":1.069,"15min":1.069},"last":0.67500000000000004}]},{"osd":15,"last update":"Mon Feb 5 04:54:11 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.58199999999999996}]}]},{"osd":6,"up_from":21,"seq":90194313233,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48636,"kb_used_data":17448,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94323204,"statfs":{"total":96636764160,"available":96586960896,"internally_reserved":0,"allocated":17866752,"data_stored":24715754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,7,8,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52100000000000002}]},{"osd":1,"last update":"Mon Feb 5 04:54:08 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.379},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.28199999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.308}]},{"osd":3,"last update":"Mon Feb 5 04:54:08 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.68700000000000006}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41699999999999998}]},{"osd":5,"last update":"Mon Feb 5 04:54:08 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.36499999999999999}]},{"osd":7,"last update":"Mon Feb 5 04:54:11 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.63900000000000001}]},{"osd":8,"last update":"Mon Feb 5 04:54:11 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.50800000000000001}]},{"osd":9,"last update":"Mon Feb 5 04:54:08 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.496},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.48199999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.625}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61299999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59499999999999997}]},{"osd":14,"last update":"Mon Feb 5 04:54:08 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.56200000000000006}]},{"osd":15,"last update":"Mon Feb 5 04:54:08 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.097,"5min":1.097,"15min":1.097},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.64900000000000002}]}]},{"osd":1,"up_from":22,"seq":94489280528,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44836,"kb_used_data":17816,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94327004,"statfs":{"total":96636764160,"available":96590852096,"internally_reserved":0,"allocated":18243584,"data_stored":25792854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,4,5,8,9,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.5}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63200000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.312}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.626}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.436}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49099999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52400000000000002}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59299999999999997}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66500000000000004}]}]},{"osd":14,"up_from":22,"seq":94489280527,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40196,"kb_used_data":13176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94331644,"statfs":{"total":96636764160,"available":96595603456,"internally_reserved":0,"allocated":13492224,"data_stored":15266134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,6,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48899999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63300000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40500000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59499999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39000000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68200000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47199999999999998}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72299999999999998}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34599999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70499999999999996}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77000000000000002}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68899999999999995}]}]},{"osd":9,"up_from":22,"seq":94489280527,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38808,"kb_used_data":11788,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94333032,"statfs":{"total":96636764160,"available":96597024768,"internally_reserved":0,"allocated":12070912,"data_stored":14942550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,4,6,7,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44700000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48899999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30599999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56699999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.628}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66700000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68200000000000005}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71099999999999997}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77600000000000002}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81200000000000006}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82799999999999996}]}]},{"osd":13,"up_from":22,"seq":94489280526,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50584,"kb_used_data":19404,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94321256,"statfs":{"total":96636764160,"available":96584966144,"internally_reserved":0,"allocated":19869696,"data_stored":30208342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,4,5,6,7,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40999999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34100000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59699999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64200000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39400000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.5}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54900000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69499999999999995}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38200000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.505}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45700000000000002}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71899999999999997}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65800000000000003}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68400000000000005}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70699999999999996}]}]},{"osd":0,"up_from":23,"seq":98784247824,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46568,"kb_used_data":15384,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325272,"statfs":{"total":96636764160,"available":96589078528,"internally_reserved":0,"allocated":15753216,"data_stored":20683047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 04:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":1.4259999999999999,"5min":1.4259999999999999,"15min":1.4259999999999999},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.33300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47299999999999998}]},{"osd":3,"last update":"Mon Feb 5 04:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.6759999999999999,"5min":1.6759999999999999,"15min":1.6759999999999999},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.442}]},{"osd":4,"last update":"Mon Feb 5 04:54:08 2024","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.30099999999999999}]},{"osd":5,"last update":"Mon Feb 5 04:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.374},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":1.5660000000000001,"5min":1.5660000000000001,"15min":1.5660000000000001},"last":0.36099999999999999}]},{"osd":6,"last update":"Mon Feb 5 04:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":2.4929999999999999,"5min":2.4929999999999999,"15min":2.4929999999999999},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.284,"5min":1.284,"15min":1.284},"last":0.64900000000000002}]},{"osd":7,"last update":"Mon Feb 5 04:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.7549999999999999,"5min":1.7549999999999999,"15min":1.7549999999999999},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":2.7629999999999999,"5min":2.7629999999999999,"15min":2.7629999999999999},"last":0.46700000000000003}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.376}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38600000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45400000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60199999999999998}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77700000000000002}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]},{"osd":15,"last update":"Mon Feb 5 04:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.7150000000000001,"5min":1.7150000000000001,"15min":1.7150000000000001},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.351,"5min":1.351,"15min":1.351},"last":0.48899999999999999}]}]},{"osd":15,"up_from":22,"seq":94489280528,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41132,"kb_used_data":14112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94330708,"statfs":{"total":96636764160,"available":96594644992,"internally_reserved":0,"allocated":14450688,"data_stored":19296598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7,8,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54100000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83699999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69399999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.625}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76900000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64600000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.872}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79200000000000004}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82199999999999995}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71499999999999997}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91900000000000004}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98299999999999998}]}]},{"osd":2,"up_from":23,"seq":98784247825,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46404,"kb_used_data":15224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325436,"statfs":{"total":96636764160,"available":96589246464,"internally_reserved":0,"allocated":15589376,"data_stored":21408583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57699999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41699999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59399999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.372}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48099999999999998}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65100000000000002}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78800000000000003}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38200000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63800000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60599999999999998}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.747}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73199999999999998}]}]},{"osd":7,"up_from":23,"seq":98784247825,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46248,"kb_used_data":15068,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325592,"statfs":{"total":96636764160,"available":96589406208,"internally_reserved":0,"allocated":15429632,"data_stored":21897558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41899999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75700000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66900000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96199999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72299999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0009999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89400000000000002}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79100000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98699999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67600000000000005}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91200000000000003}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85699999999999998}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97599999999999998}]}]},{"osd":8,"up_from":0,"seq":98784247822,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":14671872,"data_stored":20070400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":17715200,"data_stored":26550272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":13975552,"data_stored":20267008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12136448,"data_stored":15527936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":15900672,"data_stored":25223168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":14577664,"data_stored":22142976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16736256,"data_stored":24072192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":14905344,"data_stored":22351872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":9826304,"data_stored":12816384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":11005952,"data_stored":14340096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":17117184,"data_stored":23937024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":14131200,"data_stored":18976768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":12890112,"data_stored":17575936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":18804736,"data_stored":29605888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":12636160,"data_stored":15642624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":13385728,"data_stored":18694144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T04:54:12.749 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-02-05T04:54:12.749 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T04:54:12.749 INFO:tasks.thrashosds.thrasher:test_bluestore_reshard started 2024-02-05T04:54:12.749 INFO:tasks.thrashosds.thrasher:Killing osd 13, live_osds are [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-02-05T04:54:12.750 DEBUG:tasks.ceph.osd.13:waiting for process to exit 2024-02-05T04:54:12.750 INFO:teuthology.orchestra.run:waiting for 300 2024-02-05T04:54:12.769 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:12.770 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:12.778 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-02-05T04:54:12.788 INFO:tasks.ceph.osd.13:Stopped 2024-02-05T04:54:12.788 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd down 13 2024-02-05T04:54:12.840 INFO:tasks.ceph.osd.4.smithi062.stderr:2024-02-05T04:54:12.838+0000 7f708c308640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34689) UID: 0 2024-02-05T04:54:12.924 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:12.924 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:12.933 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-02-05T04:54:12.941 INFO:tasks.ceph.osd.4.smithi062.stderr:2024-02-05T04:54:12.939+0000 7f708c308640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34689) UID: 0 2024-02-05T04:54:13.042 INFO:tasks.ceph.osd.4.smithi062.stderr:2024-02-05T04:54:13.040+0000 7f708c308640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34689) UID: 0 2024-02-05T04:54:13.051 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:13.051 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:13.060 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-02-05T04:54:13.072 INFO:tasks.rados.rados.0.smithi196.stdout:329: done (4 left) 2024-02-05T04:54:13.073 INFO:tasks.rados.rados.0.smithi196.stdout:330: rollback oid 28 current snap is 27 2024-02-05T04:54:13.073 INFO:tasks.rados.rados.0.smithi196.stdout:rollback oid 28 to 6 2024-02-05T04:54:13.073 INFO:tasks.rados.rados.0.smithi196.stdout:331: rollback oid 17 current snap is 27 2024-02-05T04:54:13.073 INFO:tasks.rados.rados.0.smithi196.stdout:rollback oid 17 to 6 2024-02-05T04:54:13.073 INFO:tasks.rados.rados.0.smithi196.stdout:332: rollback oid 38 current snap is 27 2024-02-05T04:54:13.073 INFO:tasks.rados.rados.0.smithi196.stdout:rollback oid 38 to 23 2024-02-05T04:54:13.073 INFO:tasks.rados.rados.0.smithi196.stdout:333: snap_create 2024-02-05T04:54:13.077 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 8 v 86 (ObjNum 60 snap 2 seq_num 60) dirty exists 2024-02-05T04:54:13.077 INFO:tasks.rados.rados.0.smithi196.stdout:327: finishing rollback tid 0 to smithi19639362-29 2024-02-05T04:54:13.077 INFO:tasks.rados.rados.0.smithi196.stdout:327: finishing rollback tid 1 to smithi19639362-29 2024-02-05T04:54:13.077 INFO:tasks.rados.rados.0.smithi196.stdout:327: finishing rollback tid 2 to smithi19639362-29 2024-02-05T04:54:13.077 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 29 v 74 (ObjNum 78 snap 4 seq_num 78) dirty exists 2024-02-05T04:54:13.077 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 19 v 76 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-02-05T04:54:13.077 INFO:tasks.rados.rados.0.smithi196.stdout:326: expect (ObjNum 109 snap 17 seq_num 109) 2024-02-05T04:54:13.107 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-02-05T04:54:13.105+0000 7fe5ff27b640 -1 mon.a@0(leader).osd e81 definitely_dead 0 2024-02-05T04:54:13.108 INFO:teuthology.orchestra.run.smithi062.stderr:osd.13 is already down. 2024-02-05T04:54:13.120 INFO:tasks.thrashosds.thrasher:Removing osd 13, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 12, 13, 14, 15] 2024-02-05T04:54:13.121 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 13 2024-02-05T04:54:13.143 INFO:tasks.ceph.osd.5.smithi091.stderr:2024-02-05T04:54:13.142+0000 7f63983b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34277) UID: 0 2024-02-05T04:54:13.180 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:13.180 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:13.189 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-02-05T04:54:13.244 INFO:tasks.ceph.osd.1.smithi091.stderr:2024-02-05T04:54:13.243+0000 7ff9dc283640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34275) UID: 0 2024-02-05T04:54:13.336 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:13.336 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:13.345 INFO:tasks.ceph.osd.6.smithi150.stderr:2024-02-05T04:54:13.344+0000 7f6bea9f0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34015) UID: 0 2024-02-05T04:54:13.346 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-02-05T04:54:13.433 INFO:tasks.rados.rados.0.smithi196.stdout:331: finishing rollback tid 0 to smithi19639362-17 2024-02-05T04:54:13.433 INFO:tasks.rados.rados.0.smithi196.stdout:330: finishing rollback tid 0 to smithi19639362-28 2024-02-05T04:54:13.433 INFO:tasks.rados.rados.0.smithi196.stdout:331: finishing rollback tid 1 to smithi19639362-17 2024-02-05T04:54:13.433 INFO:tasks.rados.rados.0.smithi196.stdout:331: finishing rollback tid 2 to smithi19639362-17 2024-02-05T04:54:13.433 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 17 v 99 (ObjNum 71 snap 2 seq_num 71) dirty exists 2024-02-05T04:54:13.433 INFO:tasks.rados.rados.0.smithi196.stdout:330: finishing rollback tid 1 to smithi19639362-28 2024-02-05T04:54:13.433 INFO:tasks.rados.rados.0.smithi196.stdout:330: finishing rollback tid 2 to smithi19639362-28 2024-02-05T04:54:13.434 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 28 v 47 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-02-05T04:54:13.446 INFO:tasks.ceph.osd.0.smithi062.stderr:2024-02-05T04:54:13.444+0000 7fece8e03640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34688) UID: 0 2024-02-05T04:54:13.452 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:13.452 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:13.460 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-02-05T04:54:13.461 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-02-05T04:54:13.459+0000 7fe5ff27b640 -1 mon.a@0(leader).osd e81 definitely_dead 0 2024-02-05T04:54:13.547 INFO:tasks.ceph.osd.10.smithi150.stderr:2024-02-05T04:54:13.546+0000 7f3075a4d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 34018) UID: 0 2024-02-05T04:54:13.565 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:13.566 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:13.575 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-02-05T04:54:13.648 INFO:tasks.ceph.osd.7.smithi196.stderr:2024-02-05T04:54:13.647+0000 7fb281419640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34606) UID: 0 2024-02-05T04:54:13.681 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:13.681 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:13.690 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-02-05T04:54:13.748 INFO:tasks.ceph.osd.8.smithi062.stderr:2024-02-05T04:54:13.747+0000 7f9385bf2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 34693) UID: 0 2024-02-05T04:54:13.836 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:13.836 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:13.845 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-02-05T04:54:13.889 INFO:tasks.ceph.osd.3.smithi196.stderr:2024-02-05T04:54:13.888+0000 7fc8817c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34605) UID: 0 2024-02-05T04:54:13.950 INFO:tasks.ceph.osd.9.smithi091.stderr:2024-02-05T04:54:13.949+0000 7efc18827640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 34279) UID: 0 2024-02-05T04:54:13.951 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:13.951 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:13.959 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-02-05T04:54:14.052 INFO:tasks.ceph.osd.12.smithi062.stderr:2024-02-05T04:54:14.051+0000 7f5e5847e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 34695) UID: 0 2024-02-05T04:54:14.069 INFO:teuthology.orchestra.run.smithi062.stderr:marked out osd.13. 2024-02-05T04:54:14.075 INFO:tasks.rados.rados.0.smithi196.stdout:325: done (7 left) 2024-02-05T04:54:14.075 INFO:tasks.rados.rados.0.smithi196.stdout:326: done (6 left) 2024-02-05T04:54:14.075 INFO:tasks.rados.rados.0.smithi196.stdout:327: done (5 left) 2024-02-05T04:54:14.075 INFO:tasks.rados.rados.0.smithi196.stdout:328: done (4 left) 2024-02-05T04:54:14.075 INFO:tasks.rados.rados.0.smithi196.stdout:330: done (3 left) 2024-02-05T04:54:14.075 INFO:tasks.rados.rados.0.smithi196.stdout:331: done (2 left) 2024-02-05T04:54:14.075 INFO:tasks.rados.rados.0.smithi196.stdout:333: done (1 left) 2024-02-05T04:54:14.075 INFO:tasks.rados.rados.0.smithi196.stdout:334: delete oid 7 current snap is 28 2024-02-05T04:54:14.081 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-02-05T04:54:14.084 INFO:tasks.rados.rados.0.smithi196.stdout:334: done (1 left) 2024-02-05T04:54:14.084 INFO:tasks.rados.rados.0.smithi196.stdout:335: copy_from oid 47 from oid 25 current snap is 28 2024-02-05T04:54:14.084 INFO:tasks.rados.rados.0.smithi196.stdout:336: copy_from oid 39 from oid 29 current snap is 28 2024-02-05T04:54:14.084 INFO:tasks.rados.rados.0.smithi196.stdout:337: snap_remove snap 25 2024-02-05T04:54:14.090 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.13 2024-02-05T04:54:14.090 INFO:tasks.thrashosds.thrasher:remote for osd 13 is ubuntu@smithi091.front.sepia.ceph.com 2024-02-05T04:54:14.090 DEBUG:teuthology.orchestra.run.smithi091:> type ceph-objectstore-tool 2024-02-05T04:54:14.106 INFO:teuthology.orchestra.run.smithi091.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-02-05T04:54:14.106 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-13 --journal-path /var/lib/ceph/osd/ceph-13/journal --op list-pgs 2024-02-05T04:54:14.110 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:14.111 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:14.120 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-02-05T04:54:14.175 INFO:tasks.ceph.osd.0.smithi062.stderr:2024-02-05T04:54:14.173+0000 7fece8e03640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34688) UID: 0 2024-02-05T04:54:14.242 INFO:teuthology.orchestra.run.smithi062.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:14.242 INFO:teuthology.orchestra.run.smithi062.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:14.251 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-02-05T04:54:14.281 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T04:54:14.292 INFO:tasks.ceph.osd.4.smithi062.stderr:2024-02-05T04:54:14.290+0000 7f708c308640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34689) UID: 0 2024-02-05T04:54:14.354 INFO:tasks.ceph.osd.15.smithi196.stderr:2024-02-05T04:54:14.353+0000 7f1facc45640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 34611) UID: 0 2024-02-05T04:54:14.401 INFO:teuthology.orchestra.run.smithi062.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:14.402 INFO:teuthology.orchestra.run.smithi062.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:14.411 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-02-05T04:54:14.416 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0: {} 2024-02-05T04:54:14.417 INFO:teuthology.orchestra.run.smithi062.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T04:54:14.435 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1: {} 2024-02-05T04:54:14.435 INFO:teuthology.orchestra.run.smithi062.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T04:54:14.453 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2: {} 2024-02-05T04:54:14.454 INFO:teuthology.orchestra.run.smithi062.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T04:54:14.454 INFO:tasks.ceph.osd.9.smithi091.stderr:2024-02-05T04:54:14.453+0000 7efc18827640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 34279) UID: 0 2024-02-05T04:54:14.471 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3: {} 2024-02-05T04:54:14.471 INFO:teuthology.orchestra.run.smithi062.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T04:54:14.488 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4: {} 2024-02-05T04:54:14.489 INFO:teuthology.orchestra.run.smithi062.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T04:54:14.506 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5: {} 2024-02-05T04:54:14.506 INFO:teuthology.orchestra.run.smithi062.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T04:54:14.521 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-02-05T04:54:14.521 INFO:teuthology.orchestra.run.smithi062.stdout: "size": 20, 2024-02-05T04:54:14.521 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 600, 2024-02-05T04:54:14.521 INFO:teuthology.orchestra.run.smithi062.stdout: "ops": [ 2024-02-05T04:54:14.521 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.521 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.bs3 52/31 ECSubWrite(tid=200, reqid=osd.13.0:200, at_version=52'50, trim_to=0'0, roll_forward_to=51'48))", 2024-02-05T04:54:14.521 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:43.953702+0000", 2024-02-05T04:54:14.521 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 30.565763971999999, 2024-02-05T04:54:14.522 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0050156649999999999, 2024-02-05T04:54:14.522 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:14.522 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:54:14.522 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:14.522 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.522 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:14.522 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.953702+0000", 2024-02-05T04:54:14.522 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:14.522 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.522 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.522 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:14.522 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.953702+0000", 2024-02-05T04:54:14.523 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:14.523 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.523 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.523 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:14.523 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.953702+0000", 2024-02-05T04:54:14.523 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.11e-07 2024-02-05T04:54:14.523 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.523 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.523 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:14.523 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.953721+0000", 2024-02-05T04:54:14.523 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.8932000000000001e-05 2024-02-05T04:54:14.523 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.523 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.523 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:14.523 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.953732+0000", 2024-02-05T04:54:14.524 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1564e-05 2024-02-05T04:54:14.524 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.524 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.524 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:14.524 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.953741+0000", 2024-02-05T04:54:14.524 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.2050000000000008e-06 2024-02-05T04:54:14.524 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.524 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.524 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:14.524 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.953801+0000", 2024-02-05T04:54:14.524 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.9344e-05 2024-02-05T04:54:14.524 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.524 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.525 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:14.525 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.953812+0000", 2024-02-05T04:54:14.525 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0845e-05 2024-02-05T04:54:14.525 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.525 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.525 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:14.525 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.958673+0000", 2024-02-05T04:54:14.525 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0048608640000000003 2024-02-05T04:54:14.525 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.525 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.525 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:14.525 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.958717+0000", 2024-02-05T04:54:14.525 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.4700000000000002e-05 2024-02-05T04:54:14.525 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:14.525 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:14.526 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:14.526 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.526 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.526 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.fs4 52/31 ECSubWrite(tid=85, reqid=osd.11.0:85, at_version=52'40, trim_to=0'0, roll_forward_to=51'36))", 2024-02-05T04:54:14.526 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:43.954702+0000", 2024-02-05T04:54:14.526 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 30.564763821, 2024-02-05T04:54:14.526 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.005090982, 2024-02-05T04:54:14.526 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:14.526 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:54:14.526 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:14.526 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.526 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:14.526 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.954702+0000", 2024-02-05T04:54:14.526 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:14.526 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.526 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.527 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:14.527 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.954702+0000", 2024-02-05T04:54:14.527 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:14.527 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.527 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.527 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:14.527 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.954702+0000", 2024-02-05T04:54:14.527 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.9999999999999997e-07 2024-02-05T04:54:14.527 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.527 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.527 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:14.527 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.954717+0000", 2024-02-05T04:54:14.527 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.4905e-05 2024-02-05T04:54:14.527 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.527 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.527 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:14.527 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.954737+0000", 2024-02-05T04:54:14.528 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.9578000000000001e-05 2024-02-05T04:54:14.528 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.528 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.528 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:14.528 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.954745+0000", 2024-02-05T04:54:14.528 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.8299999999999996e-06 2024-02-05T04:54:14.528 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.528 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.528 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:14.528 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.954820+0000", 2024-02-05T04:54:14.528 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.4832000000000005e-05 2024-02-05T04:54:14.528 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.528 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.528 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:14.528 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.954827+0000", 2024-02-05T04:54:14.529 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.9349999999999997e-06 2024-02-05T04:54:14.529 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.529 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.529 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:14.529 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.959760+0000", 2024-02-05T04:54:14.529 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0049331669999999996 2024-02-05T04:54:14.529 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.529 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.529 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:14.529 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.959793+0000", 2024-02-05T04:54:14.529 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.3034999999999998e-05 2024-02-05T04:54:14.529 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:14.529 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:14.530 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:14.530 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.530 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.530 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.bs3 52/31 ECSubWrite(tid=202, reqid=osd.13.0:200, at_version=0'0, trim_to=0'0, roll_forward_to=52'50))", 2024-02-05T04:54:14.530 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:43.959201+0000", 2024-02-05T04:54:14.530 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 30.560264248999999, 2024-02-05T04:54:14.530 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0055030230000000001, 2024-02-05T04:54:14.530 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:14.530 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:54:14.530 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:14.530 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.530 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:14.530 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.959201+0000", 2024-02-05T04:54:14.530 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:14.531 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.531 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.531 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:14.531 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.959201+0000", 2024-02-05T04:54:14.531 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:14.531 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.531 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.531 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:14.531 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.959202+0000", 2024-02-05T04:54:14.531 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.4899999999999995e-07 2024-02-05T04:54:14.531 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.531 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.531 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:14.531 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.959211+0000", 2024-02-05T04:54:14.531 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.4350000000000003e-06 2024-02-05T04:54:14.531 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.532 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.532 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:14.532 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.959221+0000", 2024-02-05T04:54:14.532 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.8700000000000004e-06 2024-02-05T04:54:14.532 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.532 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.532 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:14.532 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.959227+0000", 2024-02-05T04:54:14.532 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.1349999999999997e-06 2024-02-05T04:54:14.532 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.532 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.532 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:14.532 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.959613+0000", 2024-02-05T04:54:14.532 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00038529800000000001 2024-02-05T04:54:14.533 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.533 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.533 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:14.533 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.959622+0000", 2024-02-05T04:54:14.533 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.4339999999999998e-06 2024-02-05T04:54:14.533 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.533 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.533 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:14.533 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.964670+0000", 2024-02-05T04:54:14.533 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.005047719 2024-02-05T04:54:14.533 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.533 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.533 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:14.533 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.964704+0000", 2024-02-05T04:54:14.534 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.4483000000000002e-05 2024-02-05T04:54:14.534 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:14.534 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:14.534 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:14.534 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.534 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.534 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.fs4 52/31 ECSubWrite(tid=87, reqid=osd.11.0:85, at_version=0'0, trim_to=0'0, roll_forward_to=52'40))", 2024-02-05T04:54:14.534 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:43.961973+0000", 2024-02-05T04:54:14.534 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 30.557492632999999, 2024-02-05T04:54:14.534 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0052598189999999998, 2024-02-05T04:54:14.534 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:14.534 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:54:14.534 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:14.534 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.535 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:14.535 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.961973+0000", 2024-02-05T04:54:14.535 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:14.535 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.535 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.535 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:14.535 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.961973+0000", 2024-02-05T04:54:14.535 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:14.535 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.535 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.535 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:14.535 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.961974+0000", 2024-02-05T04:54:14.535 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.9599999999999999e-07 2024-02-05T04:54:14.536 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.536 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.536 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:14.536 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.961983+0000", 2024-02-05T04:54:14.536 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.7270000000000002e-06 2024-02-05T04:54:14.536 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.536 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.536 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:14.536 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.961994+0000", 2024-02-05T04:54:14.536 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0778000000000001e-05 2024-02-05T04:54:14.536 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.536 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.536 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:14.537 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.962002+0000", 2024-02-05T04:54:14.537 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.5299999999999999e-06 2024-02-05T04:54:14.537 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.537 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.537 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:14.537 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.962052+0000", 2024-02-05T04:54:14.537 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.0080000000000002e-05 2024-02-05T04:54:14.537 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.537 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.537 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:14.537 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.962061+0000", 2024-02-05T04:54:14.537 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.8100000000000004e-06 2024-02-05T04:54:14.537 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.537 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.537 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:14.538 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.967197+0000", 2024-02-05T04:54:14.538 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0051366930000000003 2024-02-05T04:54:14.538 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.538 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.538 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:14.538 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.967233+0000", 2024-02-05T04:54:14.538 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.5505e-05 2024-02-05T04:54:14.538 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:14.538 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:14.538 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:14.539 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.539 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.539 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpRead(3.bs3 58/57 ECSubRead(tid=242, to_read={3:df6528f1:::smithi19639362-33:head=0,532480,0}, subchunks={3:df6528f1:::smithi19639362-33:head=[0,1]}, attrs_to_read=))", 2024-02-05T04:54:14.539 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:49.955701+0000", 2024-02-05T04:54:14.539 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 24.563764164999998, 2024-02-05T04:54:14.539 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0063665620000000001, 2024-02-05T04:54:14.539 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:14.539 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:54:14.539 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:14.539 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.539 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:14.539 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.955701+0000", 2024-02-05T04:54:14.540 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:14.540 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.540 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.540 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:14.540 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.955701+0000", 2024-02-05T04:54:14.540 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:14.540 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.540 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.540 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:14.540 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.955702+0000", 2024-02-05T04:54:14.540 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.0900000000000001e-07 2024-02-05T04:54:14.540 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.540 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.541 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:14.541 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.955710+0000", 2024-02-05T04:54:14.541 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.2220000000000002e-06 2024-02-05T04:54:14.541 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.541 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.541 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:14.541 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.955732+0000", 2024-02-05T04:54:14.541 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.2018000000000002e-05 2024-02-05T04:54:14.541 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.541 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.541 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:14.541 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.955745+0000", 2024-02-05T04:54:14.541 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.3025e-05 2024-02-05T04:54:14.541 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.541 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.542 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:14.542 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.955862+0000", 2024-02-05T04:54:14.542 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00011653400000000001 2024-02-05T04:54:14.542 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.542 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.542 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:14.542 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.962068+0000", 2024-02-05T04:54:14.542 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0062060539999999999 2024-02-05T04:54:14.542 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:14.542 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:14.542 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:14.542 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.542 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.542 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.4s3 58/57 ECSubWrite(tid=107, reqid=osd.1.0:107, at_version=58'59, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:14.543 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:49.956038+0000", 2024-02-05T04:54:14.543 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 24.563427307000001, 2024-02-05T04:54:14.543 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.005678163, 2024-02-05T04:54:14.543 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:14.543 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:54:14.543 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:14.543 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.543 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:14.543 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956038+0000", 2024-02-05T04:54:14.543 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:14.543 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.543 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.543 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:14.543 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956038+0000", 2024-02-05T04:54:14.543 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:14.544 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.544 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.544 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:14.544 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956039+0000", 2024-02-05T04:54:14.544 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.2200000000000003e-07 2024-02-05T04:54:14.544 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.544 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.544 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:14.544 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956068+0000", 2024-02-05T04:54:14.544 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.9013000000000002e-05 2024-02-05T04:54:14.544 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.544 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.544 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:14.545 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956082+0000", 2024-02-05T04:54:14.545 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.4066999999999999e-05 2024-02-05T04:54:14.545 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.545 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.545 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:14.545 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956091+0000", 2024-02-05T04:54:14.545 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.9539999999999993e-06 2024-02-05T04:54:14.545 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.545 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.545 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:14.545 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956177+0000", 2024-02-05T04:54:14.545 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.5548000000000002e-05 2024-02-05T04:54:14.545 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.545 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.545 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:14.546 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956187+0000", 2024-02-05T04:54:14.546 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0352e-05 2024-02-05T04:54:14.546 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.546 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.546 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:14.546 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.961666+0000", 2024-02-05T04:54:14.546 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0054792979999999996 2024-02-05T04:54:14.546 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.546 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.546 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:14.546 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.961716+0000", 2024-02-05T04:54:14.546 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.0108999999999997e-05 2024-02-05T04:54:14.546 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:14.546 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:14.547 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:14.547 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.547 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.547 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.4s3 58/57 ECSubWrite(tid=108, reqid=osd.1.0:108, at_version=58'61, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:14.547 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:49.956139+0000", 2024-02-05T04:54:14.547 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 24.563326732, 2024-02-05T04:54:14.547 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0079487310000000005, 2024-02-05T04:54:14.547 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:14.547 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:54:14.547 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:14.547 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.547 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:14.547 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956139+0000", 2024-02-05T04:54:14.547 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:14.548 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.548 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.548 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:14.548 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956139+0000", 2024-02-05T04:54:14.548 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:14.548 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.548 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.548 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:14.548 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956139+0000", 2024-02-05T04:54:14.548 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.0999999999999999e-07 2024-02-05T04:54:14.548 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.548 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.548 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:14.548 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956165+0000", 2024-02-05T04:54:14.549 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.5202999999999999e-05 2024-02-05T04:54:14.549 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.549 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.549 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:14.549 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956176+0000", 2024-02-05T04:54:14.549 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1523e-05 2024-02-05T04:54:14.549 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.549 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.549 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:14.549 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956184+0000", 2024-02-05T04:54:14.549 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.0940000000000008e-06 2024-02-05T04:54:14.549 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.549 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.549 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:14.549 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956668+0000", 2024-02-05T04:54:14.550 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000483535 2024-02-05T04:54:14.550 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.550 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.550 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:14.550 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956681+0000", 2024-02-05T04:54:14.550 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.2958e-05 2024-02-05T04:54:14.550 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.550 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.550 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:14.550 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.964043+0000", 2024-02-05T04:54:14.550 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0073623769999999998 2024-02-05T04:54:14.550 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.551 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.551 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:14.551 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.964088+0000", 2024-02-05T04:54:14.551 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.4530999999999998e-05 2024-02-05T04:54:14.551 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:14.551 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:14.551 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:14.551 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.551 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.551 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpRead(3.bs3 58/57 ECSubRead(tid=243, to_read={3:db93bf34:::smithi19639362-24:head=0,98304,0}, subchunks={3:db93bf34:::smithi19639362-24:head=[0,1]}, attrs_to_read=))", 2024-02-05T04:54:14.551 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:49.956639+0000", 2024-02-05T04:54:14.551 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 24.562826068, 2024-02-05T04:54:14.551 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0056018429999999996, 2024-02-05T04:54:14.551 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:14.552 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:54:14.552 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:14.552 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.552 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:14.552 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956639+0000", 2024-02-05T04:54:14.552 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:14.552 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.552 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.552 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:14.552 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956639+0000", 2024-02-05T04:54:14.552 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:14.552 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.552 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.552 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:14.553 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956640+0000", 2024-02-05T04:54:14.553 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.6800000000000009e-07 2024-02-05T04:54:14.553 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.553 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.553 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:14.553 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956650+0000", 2024-02-05T04:54:14.553 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.9739999999999997e-06 2024-02-05T04:54:14.553 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.553 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.553 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:14.553 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956681+0000", 2024-02-05T04:54:14.553 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.0655999999999999e-05 2024-02-05T04:54:14.553 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.553 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.553 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:14.554 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956707+0000", 2024-02-05T04:54:14.554 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.5469999999999998e-05 2024-02-05T04:54:14.554 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.554 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.554 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:14.554 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.962076+0000", 2024-02-05T04:54:14.554 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0053699300000000002 2024-02-05T04:54:14.554 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.554 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.554 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:14.554 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.962241+0000", 2024-02-05T04:54:14.554 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000164845 2024-02-05T04:54:14.554 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:14.554 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:14.555 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:14.555 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.555 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.555 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.14s3 68/67 ECSubWrite(tid=16, reqid=osd.3.0:16, at_version=68'77, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:14.555 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:59.962945+0000", 2024-02-05T04:54:14.555 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 14.556520731999999, 2024-02-05T04:54:14.555 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0053159289999999996, 2024-02-05T04:54:14.555 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:14.555 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:54:14.555 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:14.555 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.555 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:14.555 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.962945+0000", 2024-02-05T04:54:14.555 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:14.555 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.556 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.556 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:14.556 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.962945+0000", 2024-02-05T04:54:14.556 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:14.556 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.556 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.556 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:14.556 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.962946+0000", 2024-02-05T04:54:14.556 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.7599999999999996e-07 2024-02-05T04:54:14.556 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.556 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.556 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:14.556 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.962973+0000", 2024-02-05T04:54:14.556 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.6835e-05 2024-02-05T04:54:14.556 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.557 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.557 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:14.557 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.962987+0000", 2024-02-05T04:54:14.557 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.4788e-05 2024-02-05T04:54:14.557 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.557 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.557 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:14.557 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.962997+0000", 2024-02-05T04:54:14.557 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0169999999999999e-05 2024-02-05T04:54:14.557 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.557 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.557 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:14.557 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.963083+0000", 2024-02-05T04:54:14.557 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.5257e-05 2024-02-05T04:54:14.557 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.557 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.557 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:14.558 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.963097+0000", 2024-02-05T04:54:14.558 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.4477999999999999e-05 2024-02-05T04:54:14.558 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.558 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.558 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:14.558 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.968227+0000", 2024-02-05T04:54:14.558 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0051300190000000004 2024-02-05T04:54:14.558 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.558 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.558 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:14.558 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.968261+0000", 2024-02-05T04:54:14.558 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.3506000000000001e-05 2024-02-05T04:54:14.558 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:14.559 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:14.559 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:14.559 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.559 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.559 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.14s3 68/67 ECSubWrite(tid=17, reqid=osd.3.0:16, at_version=0'0, trim_to=0'0, roll_forward_to=68'77))", 2024-02-05T04:54:14.559 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:59.974045+0000", 2024-02-05T04:54:14.559 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 14.545420226999999, 2024-02-05T04:54:14.559 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0081087980000000004, 2024-02-05T04:54:14.559 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:14.559 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:54:14.559 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:14.559 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.560 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:14.560 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.974045+0000", 2024-02-05T04:54:14.560 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:14.560 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.560 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.560 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:14.560 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.974045+0000", 2024-02-05T04:54:14.560 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:14.560 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.560 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.560 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:14.560 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.974046+0000", 2024-02-05T04:54:14.560 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.5400000000000001e-07 2024-02-05T04:54:14.561 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.561 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.561 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:14.561 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.974056+0000", 2024-02-05T04:54:14.561 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.9960000000000005e-06 2024-02-05T04:54:14.561 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.561 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.561 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:14.561 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.974068+0000", 2024-02-05T04:54:14.561 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.2502e-05 2024-02-05T04:54:14.561 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.561 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.561 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:14.562 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.974078+0000", 2024-02-05T04:54:14.562 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.1889999999999996e-06 2024-02-05T04:54:14.562 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.562 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.562 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:14.562 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.974140+0000", 2024-02-05T04:54:14.562 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.2535999999999999e-05 2024-02-05T04:54:14.562 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.562 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.562 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:14.562 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.974151+0000", 2024-02-05T04:54:14.562 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1263e-05 2024-02-05T04:54:14.562 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.563 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.563 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:14.563 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.982119+0000", 2024-02-05T04:54:14.563 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0079679380000000008 2024-02-05T04:54:14.563 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.563 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.563 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:14.563 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.982154+0000", 2024-02-05T04:54:14.563 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.472e-05 2024-02-05T04:54:14.563 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:14.563 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:14.563 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:14.564 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.564 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.564 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.bs3 74/57 ECSubWrite(tid=340, reqid=client.4634.0:687, at_version=74'92, trim_to=0'0, roll_forward_to=74'88))", 2024-02-05T04:54:14.564 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:54:06.054652+0000", 2024-02-05T04:54:14.564 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 8.4648135609999997, 2024-02-05T04:54:14.564 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0058186510000000002, 2024-02-05T04:54:14.564 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:14.564 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:54:14.564 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:14.564 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.564 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:14.564 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.054652+0000", 2024-02-05T04:54:14.565 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:14.565 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.565 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.565 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:14.565 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.054652+0000", 2024-02-05T04:54:14.565 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:14.565 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.565 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.565 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:14.565 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.054653+0000", 2024-02-05T04:54:14.565 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.5000000000000002e-07 2024-02-05T04:54:14.565 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.565 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.565 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:14.566 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.055108+0000", 2024-02-05T04:54:14.566 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00045540499999999998 2024-02-05T04:54:14.566 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.566 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.566 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:14.566 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.055119+0000", 2024-02-05T04:54:14.566 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1286e-05 2024-02-05T04:54:14.566 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.566 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.566 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:14.566 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.055127+0000", 2024-02-05T04:54:14.566 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.61e-06 2024-02-05T04:54:14.566 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.566 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.566 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:14.567 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.055190+0000", 2024-02-05T04:54:14.567 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.2718999999999994e-05 2024-02-05T04:54:14.567 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.567 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.567 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:14.567 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.055200+0000", 2024-02-05T04:54:14.567 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.9720000000000004e-06 2024-02-05T04:54:14.567 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.567 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.567 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:14.567 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.060431+0000", 2024-02-05T04:54:14.567 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0052310940000000004 2024-02-05T04:54:14.567 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.568 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.568 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:14.568 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.060471+0000", 2024-02-05T04:54:14.568 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.9814999999999997e-05 2024-02-05T04:54:14.568 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:14.568 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:14.568 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:14.568 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.568 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.568 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.fs4 74/31 ECSubWrite(tid=157, reqid=osd.11.0:157, at_version=74'64, trim_to=0'0, roll_forward_to=72'62))", 2024-02-05T04:54:14.568 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:54:06.063576+0000", 2024-02-05T04:54:14.568 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 8.4558899079999996, 2024-02-05T04:54:14.568 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0048217599999999996, 2024-02-05T04:54:14.569 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:14.569 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:54:14.569 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:14.569 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.569 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:14.569 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.063576+0000", 2024-02-05T04:54:14.569 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:14.569 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.569 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.569 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:14.569 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.063576+0000", 2024-02-05T04:54:14.569 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:14.569 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.569 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.570 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:14.570 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.063576+0000", 2024-02-05T04:54:14.570 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.9200000000000001e-07 2024-02-05T04:54:14.570 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.570 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.570 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:14.570 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.063591+0000", 2024-02-05T04:54:14.570 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.4348000000000001e-05 2024-02-05T04:54:14.570 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.570 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.570 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:14.570 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.063598+0000", 2024-02-05T04:54:14.570 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.486e-06 2024-02-05T04:54:14.570 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.571 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.571 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:14.571 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.063603+0000", 2024-02-05T04:54:14.571 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.6600000000000003e-06 2024-02-05T04:54:14.571 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.571 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.571 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:14.571 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.063657+0000", 2024-02-05T04:54:14.571 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.4684e-05 2024-02-05T04:54:14.571 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.571 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.571 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:14.571 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.063667+0000", 2024-02-05T04:54:14.571 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.9599999999999995e-06 2024-02-05T04:54:14.571 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.572 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.572 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:14.572 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.068360+0000", 2024-02-05T04:54:14.572 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0046930349999999999 2024-02-05T04:54:14.572 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.572 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.572 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:14.572 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.068397+0000", 2024-02-05T04:54:14.572 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.6995000000000001e-05 2024-02-05T04:54:14.572 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:14.572 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:14.572 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:14.572 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.572 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.573 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:693 3.17s0 3:e8f6ce71:::smithi19639362-28:head [append 0~540672] snapc 18=[18,17,16,15,12,8,7] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-02-05T04:54:14.573 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:54:06.073867+0000", 2024-02-05T04:54:14.573 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 8.4455985400000007, 2024-02-05T04:54:14.573 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.013133568999999999, 2024-02-05T04:54:14.573 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:14.573 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:14.573 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:54:14.573 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:54:14.573 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:14.573 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 693 2024-02-05T04:54:14.573 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.573 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:14.573 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.573 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:14.573 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.073867+0000", 2024-02-05T04:54:14.574 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:14.574 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.574 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.574 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:14.574 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.073867+0000", 2024-02-05T04:54:14.574 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:14.574 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.574 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.574 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:14.574 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.073869+0000", 2024-02-05T04:54:14.574 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.59e-06 2024-02-05T04:54:14.574 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.574 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.574 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:14.574 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.074544+0000", 2024-02-05T04:54:14.575 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00067502299999999996 2024-02-05T04:54:14.575 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.575 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.575 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:14.575 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.074555+0000", 2024-02-05T04:54:14.575 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1780000000000001e-05 2024-02-05T04:54:14.575 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.575 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.575 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:14.576 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.074561+0000", 2024-02-05T04:54:14.576 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.8080000000000001e-06 2024-02-05T04:54:14.576 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.576 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.576 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:14.576 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.074606+0000", 2024-02-05T04:54:14.576 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.4468000000000002e-05 2024-02-05T04:54:14.576 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.576 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.576 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:54:14.576 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.074688+0000", 2024-02-05T04:54:14.576 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.2262999999999995e-05 2024-02-05T04:54:14.576 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.576 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.577 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:14.577 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.075525+0000", 2024-02-05T04:54:14.577 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000837023 2024-02-05T04:54:14.577 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.577 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.577 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:14.577 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.079967+0000", 2024-02-05T04:54:14.577 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0044417930000000003 2024-02-05T04:54:14.577 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.577 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.577 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:54:14.577 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.086948+0000", 2024-02-05T04:54:14.577 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0069808910000000004 2024-02-05T04:54:14.577 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.578 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.578 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:14.578 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.087001+0000", 2024-02-05T04:54:14.578 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.2930000000000003e-05 2024-02-05T04:54:14.578 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:14.578 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:14.578 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:14.578 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.578 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.578 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.bs3 74/57 ECSubWrite(tid=355, reqid=osd.13.0:355, at_version=74'100, trim_to=0'0, roll_forward_to=74'96))", 2024-02-05T04:54:14.578 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:54:06.077090+0000", 2024-02-05T04:54:14.578 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 8.4423756650000001, 2024-02-05T04:54:14.578 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0086224860000000004, 2024-02-05T04:54:14.578 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:14.579 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:54:14.579 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:14.579 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.579 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:14.579 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.077090+0000", 2024-02-05T04:54:14.579 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:14.579 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.579 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.579 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:14.579 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.077090+0000", 2024-02-05T04:54:14.579 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:14.579 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.579 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.579 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:14.580 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.077090+0000", 2024-02-05T04:54:14.580 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.2300000000000001e-07 2024-02-05T04:54:14.580 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.580 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.580 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:14.580 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.077103+0000", 2024-02-05T04:54:14.580 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.3005e-05 2024-02-05T04:54:14.580 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.580 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.580 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:14.580 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.077115+0000", 2024-02-05T04:54:14.580 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1331e-05 2024-02-05T04:54:14.580 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.580 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.581 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:14.581 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.077120+0000", 2024-02-05T04:54:14.581 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.6799999999999998e-06 2024-02-05T04:54:14.581 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.581 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.581 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:14.581 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.077280+0000", 2024-02-05T04:54:14.581 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000159588 2024-02-05T04:54:14.581 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.581 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.581 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:14.581 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.077291+0000", 2024-02-05T04:54:14.581 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1207e-05 2024-02-05T04:54:14.581 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.581 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.582 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:14.582 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.085681+0000", 2024-02-05T04:54:14.582 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0083902419999999991 2024-02-05T04:54:14.582 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.582 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.582 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:14.582 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.085712+0000", 2024-02-05T04:54:14.582 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.0910000000000001e-05 2024-02-05T04:54:14.582 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:14.582 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:14.582 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:14.582 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.582 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.582 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:694 3.17s0 3:e8f6ce71:::smithi19639362-28:head [append 0~720896] snapc 18=[18,17,16,15,12,8,7] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-02-05T04:54:14.582 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:54:06.077837+0000", 2024-02-05T04:54:14.583 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 8.4416286809999992, 2024-02-05T04:54:14.583 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0092709699999999999, 2024-02-05T04:54:14.583 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:14.583 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:14.583 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:54:14.583 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:54:14.583 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:14.583 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 694 2024-02-05T04:54:14.583 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.583 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:14.583 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.583 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:14.583 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.077837+0000", 2024-02-05T04:54:14.584 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:14.584 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.584 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.584 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:14.584 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.077837+0000", 2024-02-05T04:54:14.584 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:14.584 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.584 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.584 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:14.584 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.077838+0000", 2024-02-05T04:54:14.584 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.6700000000000002e-07 2024-02-05T04:54:14.584 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.584 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.584 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:14.585 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.078520+0000", 2024-02-05T04:54:14.585 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00068213300000000002 2024-02-05T04:54:14.585 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.585 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.585 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:14.585 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.078532+0000", 2024-02-05T04:54:14.585 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1907e-05 2024-02-05T04:54:14.585 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.585 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.585 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:14.585 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.078538+0000", 2024-02-05T04:54:14.585 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.6690000000000003e-06 2024-02-05T04:54:14.585 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.585 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.585 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:14.586 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.078589+0000", 2024-02-05T04:54:14.586 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.0983999999999998e-05 2024-02-05T04:54:14.586 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.586 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.586 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:54:14.586 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.078656+0000", 2024-02-05T04:54:14.586 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.7088000000000005e-05 2024-02-05T04:54:14.586 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.586 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.586 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:14.586 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.079247+0000", 2024-02-05T04:54:14.586 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00059124199999999996 2024-02-05T04:54:14.586 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.586 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.586 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:14.587 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.080425+0000", 2024-02-05T04:54:14.587 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0011778680000000001 2024-02-05T04:54:14.587 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.587 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.587 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:54:14.587 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.087070+0000", 2024-02-05T04:54:14.587 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0066450069999999997 2024-02-05T04:54:14.587 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.587 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.587 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:14.587 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.087108+0000", 2024-02-05T04:54:14.587 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.8105000000000002e-05 2024-02-05T04:54:14.587 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:14.587 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:14.587 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:14.588 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.588 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.588 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:695 3.17s0 3:e8f6ce71:::smithi19639362-28:head [append 0~786432] snapc 18=[18,17,16,15,12,8,7] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-02-05T04:54:14.588 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:54:06.082921+0000", 2024-02-05T04:54:14.588 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 8.4365445010000002, 2024-02-05T04:54:14.588 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0059995420000000001, 2024-02-05T04:54:14.588 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:14.588 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:14.588 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:54:14.588 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:54:14.588 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:14.588 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 695 2024-02-05T04:54:14.588 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.588 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:14.589 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.589 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:14.589 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.082921+0000", 2024-02-05T04:54:14.589 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:14.589 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.589 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.589 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:14.589 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.082921+0000", 2024-02-05T04:54:14.589 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:14.589 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.589 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.589 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:14.589 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.082922+0000", 2024-02-05T04:54:14.589 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.215e-06 2024-02-05T04:54:14.590 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.590 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.590 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:14.590 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.083627+0000", 2024-02-05T04:54:14.590 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00070471499999999996 2024-02-05T04:54:14.590 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.590 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.590 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:14.590 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.083639+0000", 2024-02-05T04:54:14.590 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1691e-05 2024-02-05T04:54:14.590 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.590 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.590 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:14.590 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.083644+0000", 2024-02-05T04:54:14.591 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.6930000000000004e-06 2024-02-05T04:54:14.591 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.591 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.591 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:14.591 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.083686+0000", 2024-02-05T04:54:14.591 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.1567999999999999e-05 2024-02-05T04:54:14.591 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.591 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.591 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:54:14.591 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.083758+0000", 2024-02-05T04:54:14.591 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.2107999999999995e-05 2024-02-05T04:54:14.591 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.591 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.592 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:14.592 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.084390+0000", 2024-02-05T04:54:14.592 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00063148200000000005 2024-02-05T04:54:14.592 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.592 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.592 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:14.592 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.087118+0000", 2024-02-05T04:54:14.592 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0027284560000000002 2024-02-05T04:54:14.592 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.592 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.592 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:54:14.592 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.088858+0000", 2024-02-05T04:54:14.593 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0017397090000000001 2024-02-05T04:54:14.593 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.593 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.593 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:14.593 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.088921+0000", 2024-02-05T04:54:14.593 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.2904999999999995e-05 2024-02-05T04:54:14.593 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:14.593 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:14.593 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:14.593 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.593 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.593 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:696 3.17s0 3:e8f6ce71:::smithi19639362-28:head [append 0~49152] snapc 18=[18,17,16,15,12,8,7] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-02-05T04:54:14.593 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:54:06.083659+0000", 2024-02-05T04:54:14.593 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 8.4358068720000006, 2024-02-05T04:54:14.594 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0082693929999999999, 2024-02-05T04:54:14.594 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:14.594 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:14.594 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:54:14.594 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:54:14.594 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:14.594 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 696 2024-02-05T04:54:14.594 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.594 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:14.594 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.594 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:14.594 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.083659+0000", 2024-02-05T04:54:14.594 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:14.594 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.594 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.595 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:14.595 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.083659+0000", 2024-02-05T04:54:14.595 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:14.595 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.595 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.595 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:14.595 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.083659+0000", 2024-02-05T04:54:14.595 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.75e-07 2024-02-05T04:54:14.595 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.595 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.595 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:14.595 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.083683+0000", 2024-02-05T04:54:14.595 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.3626999999999999e-05 2024-02-05T04:54:14.595 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.596 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.596 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:14.596 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.083690+0000", 2024-02-05T04:54:14.596 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.4370000000000001e-06 2024-02-05T04:54:14.596 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.596 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.596 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:14.596 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.083695+0000", 2024-02-05T04:54:14.596 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.5029999999999999e-06 2024-02-05T04:54:14.596 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.596 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.596 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:14.596 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.085844+0000", 2024-02-05T04:54:14.596 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0021489119999999998 2024-02-05T04:54:14.597 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.597 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.597 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:54:14.597 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.085950+0000", 2024-02-05T04:54:14.597 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000106159 2024-02-05T04:54:14.597 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.597 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.597 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:14.597 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.086449+0000", 2024-02-05T04:54:14.597 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00049940300000000004 2024-02-05T04:54:14.597 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.597 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.597 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:14.597 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.091749+0000", 2024-02-05T04:54:14.598 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0053000180000000001 2024-02-05T04:54:14.598 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.598 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.598 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:54:14.598 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.091890+0000", 2024-02-05T04:54:14.598 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00014029299999999999 2024-02-05T04:54:14.598 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.598 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.598 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:14.598 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.091928+0000", 2024-02-05T04:54:14.598 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.8365999999999998e-05 2024-02-05T04:54:14.598 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:14.598 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:14.598 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:14.598 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.598 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.599 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:697 3.17s0 3:e8f6ce71:::smithi19639362-28:head [setxattr _header (57) in=7b] snapc 18=[18,17,16,15,12,8,7] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-02-05T04:54:14.599 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:54:06.083703+0000", 2024-02-05T04:54:14.599 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 8.4357621009999999, 2024-02-05T04:54:14.599 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0084306329999999999, 2024-02-05T04:54:14.599 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:14.599 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:14.599 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:54:14.599 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:54:14.599 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:14.599 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 697 2024-02-05T04:54:14.599 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.599 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:14.599 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.599 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:14.600 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.083703+0000", 2024-02-05T04:54:14.600 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:14.600 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.600 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.600 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:14.600 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.083703+0000", 2024-02-05T04:54:14.600 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:14.600 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.600 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.600 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:14.600 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.083704+0000", 2024-02-05T04:54:14.600 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.6000000000000004e-07 2024-02-05T04:54:14.600 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.600 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.600 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:14.601 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.083708+0000", 2024-02-05T04:54:14.601 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.6610000000000002e-06 2024-02-05T04:54:14.601 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.601 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.601 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:14.601 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.083714+0000", 2024-02-05T04:54:14.601 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.4309999999999999e-06 2024-02-05T04:54:14.601 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.601 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.601 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:14.601 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.083719+0000", 2024-02-05T04:54:14.601 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.9049999999999996e-06 2024-02-05T04:54:14.601 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.601 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.601 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:14.601 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.087267+0000", 2024-02-05T04:54:14.602 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0035483390000000002 2024-02-05T04:54:14.602 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.602 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.602 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:54:14.602 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.087324+0000", 2024-02-05T04:54:14.602 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.6459999999999998e-05 2024-02-05T04:54:14.602 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.602 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.602 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:14.602 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.087567+0000", 2024-02-05T04:54:14.602 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000243651 2024-02-05T04:54:14.602 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.602 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.602 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:14.602 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.091770+0000", 2024-02-05T04:54:14.603 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0042029190000000003 2024-02-05T04:54:14.603 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.603 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.603 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:54:14.603 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.092049+0000", 2024-02-05T04:54:14.603 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00027828100000000001 2024-02-05T04:54:14.603 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.603 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.603 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:14.603 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.092134+0000", 2024-02-05T04:54:14.603 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.5426000000000005e-05 2024-02-05T04:54:14.603 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:14.603 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:14.603 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:14.604 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.604 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.604 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:698 3.17s0 3:e8f6ce71:::smithi19639362-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e74)", 2024-02-05T04:54:14.604 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:54:06.083731+0000", 2024-02-05T04:54:14.604 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 8.4357342309999996, 2024-02-05T04:54:14.604 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0094381120000000002, 2024-02-05T04:54:14.604 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:14.604 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "started", 2024-02-05T04:54:14.604 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:54:14.604 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:54:14.604 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:14.604 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 698 2024-02-05T04:54:14.604 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.604 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:14.605 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.605 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:14.605 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.083731+0000", 2024-02-05T04:54:14.605 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:14.605 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.605 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.605 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:14.605 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.083731+0000", 2024-02-05T04:54:14.605 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:14.605 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.605 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.605 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:14.605 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.083732+0000", 2024-02-05T04:54:14.605 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.6199999999999999e-07 2024-02-05T04:54:14.606 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.606 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.606 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:14.606 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.083733+0000", 2024-02-05T04:54:14.606 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.6869999999999999e-06 2024-02-05T04:54:14.606 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.606 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.606 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:14.606 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.083739+0000", 2024-02-05T04:54:14.606 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.48e-06 2024-02-05T04:54:14.606 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.606 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.606 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:14.606 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.083743+0000", 2024-02-05T04:54:14.606 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.8120000000000001e-06 2024-02-05T04:54:14.607 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.607 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.607 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:14.607 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.087790+0000", 2024-02-05T04:54:14.607 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.004047246 2024-02-05T04:54:14.607 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.607 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.607 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for rw locks", 2024-02-05T04:54:14.607 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.087843+0000", 2024-02-05T04:54:14.607 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.3545999999999999e-05 2024-02-05T04:54:14.607 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.607 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.607 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:14.607 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.092445+0000", 2024-02-05T04:54:14.608 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0046010909999999999 2024-02-05T04:54:14.608 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.608 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.608 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:54:14.608 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.092504+0000", 2024-02-05T04:54:14.608 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.9431000000000001e-05 2024-02-05T04:54:14.608 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.608 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.608 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:14.608 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.093169+0000", 2024-02-05T04:54:14.608 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00066545700000000005 2024-02-05T04:54:14.608 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:14.608 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:14.608 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:14.609 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.609 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.609 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.fs4 78/31 ECSubWrite(tid=192, reqid=client.4634.0:788, at_version=78'94, trim_to=0'0, roll_forward_to=78'92))", 2024-02-05T04:54:14.609 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:54:10.090739+0000", 2024-02-05T04:54:14.609 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 4.4287267750000003, 2024-02-05T04:54:14.609 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0049663069999999997, 2024-02-05T04:54:14.609 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:14.609 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:54:14.609 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:14.609 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.609 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:14.609 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:10.090739+0000", 2024-02-05T04:54:14.609 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:14.609 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.610 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.610 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:14.610 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:10.090739+0000", 2024-02-05T04:54:14.610 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:14.610 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.610 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.610 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:14.610 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:10.090739+0000", 2024-02-05T04:54:14.610 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.4499999999999997e-07 2024-02-05T04:54:14.610 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.610 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.610 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:14.610 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:10.090809+0000", 2024-02-05T04:54:14.610 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.0233000000000003e-05 2024-02-05T04:54:14.611 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.611 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.611 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:14.611 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:10.090819+0000", 2024-02-05T04:54:14.611 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.6909999999999992e-06 2024-02-05T04:54:14.611 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.611 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.611 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:14.611 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:10.090825+0000", 2024-02-05T04:54:14.611 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.699e-06 2024-02-05T04:54:14.611 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.611 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.611 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:14.612 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:10.090872+0000", 2024-02-05T04:54:14.612 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.7278999999999999e-05 2024-02-05T04:54:14.612 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.612 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.612 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:14.612 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:10.090878+0000", 2024-02-05T04:54:14.612 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.57e-06 2024-02-05T04:54:14.612 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.612 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.612 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:14.612 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:10.095679+0000", 2024-02-05T04:54:14.612 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.004801446 2024-02-05T04:54:14.612 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:14.612 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:14.613 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:14.613 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:10.095705+0000", 2024-02-05T04:54:14.613 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.5944000000000001e-05 2024-02-05T04:54:14.613 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:14.613 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:14.613 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:14.613 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:14.613 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:14.613 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-02-05T04:54:14.614 INFO:teuthology.orchestra.run.smithi062.stdout:osd.6: {} 2024-02-05T04:54:14.614 INFO:teuthology.orchestra.run.smithi062.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T04:54:14.615 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-02-05T04:54:14.616 INFO:teuthology.orchestra.run.smithi062.stdout:osd.7: {} 2024-02-05T04:54:14.616 INFO:teuthology.orchestra.run.smithi062.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T04:54:14.616 INFO:teuthology.orchestra.run.smithi062.stdout:osd.8: {} 2024-02-05T04:54:14.617 INFO:teuthology.orchestra.run.smithi062.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-05T04:54:14.617 INFO:teuthology.orchestra.run.smithi062.stdout:osd.9: {} 2024-02-05T04:54:14.617 INFO:teuthology.orchestra.run.smithi062.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-05T04:54:14.618 INFO:teuthology.orchestra.run.smithi062.stdout:osd.10: {} 2024-02-05T04:54:14.618 INFO:teuthology.orchestra.run.smithi062.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-05T04:54:14.618 INFO:teuthology.orchestra.run.smithi062.stdout:osd.11: {} 2024-02-05T04:54:14.618 INFO:teuthology.orchestra.run.smithi062.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-05T04:54:14.619 INFO:tasks.ceph.osd.7.smithi196.stderr:2024-02-05T04:54:14.614+0000 7fb281419640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34606) UID: 0 2024-02-05T04:54:14.630 INFO:teuthology.orchestra.run.smithi062.stdout:osd.12: {} 2024-02-05T04:54:14.631 INFO:teuthology.orchestra.run.smithi062.stderr:osd.12: osd_enable_op_tracker = '' 2024-02-05T04:54:14.631 INFO:teuthology.orchestra.run.smithi062.stderr:Error ENXIO: problem getting command descriptions from osd.13 2024-02-05T04:54:14.648 INFO:teuthology.orchestra.run.smithi062.stdout:osd.14: {} 2024-02-05T04:54:14.648 INFO:teuthology.orchestra.run.smithi062.stderr:osd.14: osd_enable_op_tracker = '' 2024-02-05T04:54:14.666 INFO:teuthology.orchestra.run.smithi062.stdout:osd.15: {} 2024-02-05T04:54:14.666 INFO:teuthology.orchestra.run.smithi062.stderr:osd.15: osd_enable_op_tracker = '' 2024-02-05T04:54:14.669 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-05T04:54:14.670 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-05T04:54:14.715 INFO:tasks.ceph.osd.6.smithi150.stderr:2024-02-05T04:54:14.715+0000 7f6bea9f0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34015) UID: 0 2024-02-05T04:54:14.738 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-02-05T04:54:14.738 INFO:teuthology.orchestra.run.smithi062.stdout: "ops": [], 2024-02-05T04:54:14.738 INFO:teuthology.orchestra.run.smithi062.stdout: "num_ops": 0 2024-02-05T04:54:14.738 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-02-05T04:54:14.746 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-02-05T04:54:14.816 INFO:tasks.ceph.osd.0.smithi062.stderr:2024-02-05T04:54:14.814+0000 7fece8e03640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34688) UID: 0 2024-02-05T04:54:14.853 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-02-05T04:54:14.853 INFO:teuthology.orchestra.run.smithi062.stdout: "ops": [], 2024-02-05T04:54:14.853 INFO:teuthology.orchestra.run.smithi062.stdout: "complaint_time": 30, 2024-02-05T04:54:14.853 INFO:teuthology.orchestra.run.smithi062.stdout: "num_blocked_ops": 0 2024-02-05T04:54:14.853 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-02-05T04:54:14.862 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-02-05T04:54:14.916 INFO:tasks.ceph.osd.11.smithi196.stderr:2024-02-05T04:54:14.915+0000 7f85a8d86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 34609) UID: 0 2024-02-05T04:54:15.011 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-02-05T04:54:15.012 INFO:teuthology.orchestra.run.smithi062.stdout: "size": 20, 2024-02-05T04:54:15.012 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 600, 2024-02-05T04:54:15.012 INFO:teuthology.orchestra.run.smithi062.stdout: "ops": [ 2024-02-05T04:54:15.012 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.012 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:218 3.3s0 3:ce48de55:::smithi19639362-16:head [append 0~638976] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:54:15.012 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:29.132294+0000", 2024-02-05T04:54:15.012 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 45.877200623999997, 2024-02-05T04:54:15.012 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0081869909999999994, 2024-02-05T04:54:15.012 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:15.012 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:15.012 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:54:15.012 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:54:15.012 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:15.013 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 218 2024-02-05T04:54:15.013 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.013 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:15.013 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.013 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:15.013 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.132294+0000", 2024-02-05T04:54:15.013 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:15.013 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.013 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.013 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:15.013 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.132294+0000", 2024-02-05T04:54:15.013 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:15.013 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.014 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.014 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:15.014 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.132296+0000", 2024-02-05T04:54:15.014 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.886e-06 2024-02-05T04:54:15.014 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.014 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.014 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:15.014 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.133151+0000", 2024-02-05T04:54:15.014 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00085535699999999995 2024-02-05T04:54:15.014 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.014 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.014 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:15.014 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.133165+0000", 2024-02-05T04:54:15.014 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.396e-05 2024-02-05T04:54:15.015 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.015 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.015 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:15.015 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.133182+0000", 2024-02-05T04:54:15.015 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.6787e-05 2024-02-05T04:54:15.015 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.015 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.015 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:15.015 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.133229+0000", 2024-02-05T04:54:15.015 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.7509999999999997e-05 2024-02-05T04:54:15.015 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.015 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.015 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:54:15.015 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.133324+0000", 2024-02-05T04:54:15.015 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.4950000000000004e-05 2024-02-05T04:54:15.016 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.016 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.016 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:15.016 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.134392+0000", 2024-02-05T04:54:15.016 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0010674219999999999 2024-02-05T04:54:15.016 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.016 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.016 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:15.016 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.138240+0000", 2024-02-05T04:54:15.016 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0038487930000000001 2024-02-05T04:54:15.016 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.016 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.016 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:54:15.016 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.140400+0000", 2024-02-05T04:54:15.017 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0021592690000000001 2024-02-05T04:54:15.017 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.017 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.017 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:15.017 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.140481+0000", 2024-02-05T04:54:15.017 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.1057000000000007e-05 2024-02-05T04:54:15.017 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.017 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:15.017 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.017 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.017 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.017 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:219 3.3s0 3:ce48de55:::smithi19639362-16:head [append 0~376832] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:54:15.017 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:29.134822+0000", 2024-02-05T04:54:15.017 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 45.874671948, 2024-02-05T04:54:15.018 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0088602019999999993, 2024-02-05T04:54:15.018 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:15.018 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:15.018 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:54:15.018 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:54:15.018 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:15.018 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 219 2024-02-05T04:54:15.018 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.018 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:15.018 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.018 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:15.018 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.134822+0000", 2024-02-05T04:54:15.018 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:15.018 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.019 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.019 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:15.019 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.134822+0000", 2024-02-05T04:54:15.019 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:15.019 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.019 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.019 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:15.019 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.134825+0000", 2024-02-05T04:54:15.019 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.1229999999999998e-06 2024-02-05T04:54:15.019 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.019 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.019 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:15.019 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.135417+0000", 2024-02-05T04:54:15.020 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00059254900000000005 2024-02-05T04:54:15.020 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.020 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.020 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:15.020 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.135430+0000", 2024-02-05T04:54:15.020 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.2663000000000001e-05 2024-02-05T04:54:15.020 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.020 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.020 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:15.020 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.135436+0000", 2024-02-05T04:54:15.020 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.6959999999999999e-06 2024-02-05T04:54:15.020 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.020 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.020 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:15.021 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.135834+0000", 2024-02-05T04:54:15.021 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000397382 2024-02-05T04:54:15.021 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.021 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.021 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:54:15.021 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.135967+0000", 2024-02-05T04:54:15.021 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00013333 2024-02-05T04:54:15.021 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.021 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.021 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:15.021 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.136753+0000", 2024-02-05T04:54:15.021 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00078585700000000005 2024-02-05T04:54:15.021 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.021 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.021 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:15.022 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.138286+0000", 2024-02-05T04:54:15.022 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0015333390000000001 2024-02-05T04:54:15.022 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.022 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.022 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:54:15.022 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.143611+0000", 2024-02-05T04:54:15.022 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0053249639999999997 2024-02-05T04:54:15.022 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.022 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.022 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:15.022 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.143683+0000", 2024-02-05T04:54:15.022 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.1298999999999996e-05 2024-02-05T04:54:15.022 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.022 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:15.023 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.023 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.023 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.023 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:220 3.3s0 3:ce48de55:::smithi19639362-16:head [setxattr _header (56) in=7b] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:54:15.023 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:29.135450+0000", 2024-02-05T04:54:15.023 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 45.874044026, 2024-02-05T04:54:15.023 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0084937420000000003, 2024-02-05T04:54:15.023 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:15.023 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:15.023 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:54:15.023 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:54:15.023 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:15.023 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 220 2024-02-05T04:54:15.023 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.024 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:15.024 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.024 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:15.024 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.135450+0000", 2024-02-05T04:54:15.024 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:15.024 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.024 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.024 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:15.024 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.135450+0000", 2024-02-05T04:54:15.024 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:15.024 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.024 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.024 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:15.025 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.135451+0000", 2024-02-05T04:54:15.025 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.4799999999999997e-07 2024-02-05T04:54:15.025 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.025 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.025 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:15.025 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.135454+0000", 2024-02-05T04:54:15.025 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.3539999999999998e-06 2024-02-05T04:54:15.025 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.025 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.025 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:15.025 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.135458+0000", 2024-02-05T04:54:15.025 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.1509999999999997e-06 2024-02-05T04:54:15.025 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.025 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.025 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:15.026 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.135461+0000", 2024-02-05T04:54:15.026 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.1650000000000002e-06 2024-02-05T04:54:15.026 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.026 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.026 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:15.026 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.137469+0000", 2024-02-05T04:54:15.026 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0020076410000000001 2024-02-05T04:54:15.026 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.026 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.026 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:54:15.026 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.137586+0000", 2024-02-05T04:54:15.026 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000117599 2024-02-05T04:54:15.027 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.027 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.027 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:15.027 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.138018+0000", 2024-02-05T04:54:15.027 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00043188200000000002 2024-02-05T04:54:15.027 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.027 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.027 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:15.027 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.140234+0000", 2024-02-05T04:54:15.027 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0022155159999999998 2024-02-05T04:54:15.027 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.027 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.027 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:54:15.027 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.143857+0000", 2024-02-05T04:54:15.028 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0036236749999999998 2024-02-05T04:54:15.028 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.028 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.028 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:15.028 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.143944+0000", 2024-02-05T04:54:15.028 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.6910999999999995e-05 2024-02-05T04:54:15.028 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.028 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:15.028 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.028 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.028 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.028 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:221 3.3s0 3:ce48de55:::smithi19639362-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:54:15.028 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:29.135468+0000", 2024-02-05T04:54:15.029 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 45.874026888000003, 2024-02-05T04:54:15.029 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.010019444000000001, 2024-02-05T04:54:15.029 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:15.029 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "started", 2024-02-05T04:54:15.029 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:54:15.029 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:54:15.029 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:15.029 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 221 2024-02-05T04:54:15.029 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.029 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:15.029 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.029 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:15.029 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.135468+0000", 2024-02-05T04:54:15.029 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:15.030 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.030 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.030 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:15.030 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.135468+0000", 2024-02-05T04:54:15.030 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:15.030 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.030 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.030 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:15.030 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.135468+0000", 2024-02-05T04:54:15.030 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.4299999999999999e-07 2024-02-05T04:54:15.030 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.030 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.030 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:15.030 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.135469+0000", 2024-02-05T04:54:15.031 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.457e-06 2024-02-05T04:54:15.031 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.031 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.031 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:15.031 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.135475+0000", 2024-02-05T04:54:15.031 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.9939999999999997e-06 2024-02-05T04:54:15.031 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.031 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.031 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:15.031 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.135480+0000", 2024-02-05T04:54:15.031 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.0989999999999998e-06 2024-02-05T04:54:15.031 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.031 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.032 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:15.032 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.138320+0000", 2024-02-05T04:54:15.032 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0028394739999999998 2024-02-05T04:54:15.032 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.032 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.032 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for rw locks", 2024-02-05T04:54:15.032 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.138401+0000", 2024-02-05T04:54:15.032 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.0966999999999999e-05 2024-02-05T04:54:15.032 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.032 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.032 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:15.032 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.144381+0000", 2024-02-05T04:54:15.032 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0059798389999999998 2024-02-05T04:54:15.032 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.033 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.033 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:54:15.033 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.144442+0000", 2024-02-05T04:54:15.033 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.1311999999999998e-05 2024-02-05T04:54:15.033 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.033 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.033 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:15.033 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.145487+0000", 2024-02-05T04:54:15.033 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0010450590000000001 2024-02-05T04:54:15.033 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.033 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:15.033 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.033 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.033 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.034 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:390 3.8s0 3:170878b2:::smithi19639362-30:head [copy-from ver 20] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-02-05T04:54:15.034 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:34.171935+0000", 2024-02-05T04:54:15.034 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 40.837558928, 2024-02-05T04:54:15.034 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.014076916, 2024-02-05T04:54:15.034 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:15.034 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:15.034 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:54:15.034 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:54:15.034 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:15.035 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 390 2024-02-05T04:54:15.035 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.035 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:15.035 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.035 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:15.035 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.171935+0000", 2024-02-05T04:54:15.035 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:15.035 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.035 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.035 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:15.036 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.171935+0000", 2024-02-05T04:54:15.036 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:15.036 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.036 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.036 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:15.036 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.171937+0000", 2024-02-05T04:54:15.036 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.8959999999999999e-06 2024-02-05T04:54:15.036 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.036 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.036 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:15.036 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.171944+0000", 2024-02-05T04:54:15.036 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.5729999999999996e-06 2024-02-05T04:54:15.036 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.036 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.037 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:15.037 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.171956+0000", 2024-02-05T04:54:15.037 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.2499e-05 2024-02-05T04:54:15.037 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.037 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.037 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:15.037 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.171964+0000", 2024-02-05T04:54:15.037 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.6450000000000005e-06 2024-02-05T04:54:15.037 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.037 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.037 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:15.037 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.172025+0000", 2024-02-05T04:54:15.037 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.0713e-05 2024-02-05T04:54:15.037 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.037 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.038 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:54:15.038 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.172112+0000", 2024-02-05T04:54:15.038 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.7421000000000002e-05 2024-02-05T04:54:15.038 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.038 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.038 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:15.038 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.180774+0000", 2024-02-05T04:54:15.038 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.008661986 2024-02-05T04:54:15.038 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.038 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.038 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:15.038 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.182980+0000", 2024-02-05T04:54:15.038 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0022060389999999999 2024-02-05T04:54:15.038 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.038 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.038 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:54:15.039 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.185921+0000", 2024-02-05T04:54:15.039 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0029412179999999998 2024-02-05T04:54:15.039 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.039 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.039 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:15.039 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.186012+0000", 2024-02-05T04:54:15.039 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.0926000000000003e-05 2024-02-05T04:54:15.039 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.039 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:15.039 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.039 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.039 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.039 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:391 3.8s0 3:170878b2:::smithi19639362-30:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-02-05T04:54:15.039 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:34.172014+0000", 2024-02-05T04:54:15.039 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 40.837480444999997, 2024-02-05T04:54:15.040 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.010857311999999999, 2024-02-05T04:54:15.040 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:15.040 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "started", 2024-02-05T04:54:15.040 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:54:15.040 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:54:15.040 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:15.040 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 391 2024-02-05T04:54:15.040 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.040 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:15.040 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.041 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:15.041 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.172014+0000", 2024-02-05T04:54:15.041 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:15.041 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.041 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.041 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:15.041 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.172014+0000", 2024-02-05T04:54:15.041 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:15.041 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.041 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.041 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:15.041 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.172015+0000", 2024-02-05T04:54:15.041 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.5199999999999996e-07 2024-02-05T04:54:15.042 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.042 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.042 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:15.042 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.172018+0000", 2024-02-05T04:54:15.042 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.5319999999999998e-06 2024-02-05T04:54:15.042 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.042 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.042 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:15.042 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.172028+0000", 2024-02-05T04:54:15.042 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.5990000000000008e-06 2024-02-05T04:54:15.042 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.042 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.042 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:15.042 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.172034+0000", 2024-02-05T04:54:15.043 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.742e-06 2024-02-05T04:54:15.043 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.043 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.043 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:15.043 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.172290+0000", 2024-02-05T04:54:15.043 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00025550300000000002 2024-02-05T04:54:15.043 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.043 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.043 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for blocked object", 2024-02-05T04:54:15.043 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.172342+0000", 2024-02-05T04:54:15.043 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.1657999999999999e-05 2024-02-05T04:54:15.043 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.043 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.043 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:15.044 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.182668+0000", 2024-02-05T04:54:15.044 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.010326223000000001 2024-02-05T04:54:15.044 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.044 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.044 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:54:15.044 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.182773+0000", 2024-02-05T04:54:15.044 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00010530399999999999 2024-02-05T04:54:15.044 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.044 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.044 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:15.044 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.182871+0000", 2024-02-05T04:54:15.044 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.8098999999999997e-05 2024-02-05T04:54:15.044 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.044 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:15.045 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.045 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.045 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.045 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.4s5 43/31 ECSubWrite(tid=46, reqid=osd.1.0:46, at_version=43'29, trim_to=0'0, roll_forward_to=42'27))", 2024-02-05T04:54:15.045 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:35.182637+0000", 2024-02-05T04:54:15.045 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 39.826857638, 2024-02-05T04:54:15.045 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0075236640000000002, 2024-02-05T04:54:15.045 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:15.045 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:54:15.045 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:15.045 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.045 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:15.045 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.182637+0000", 2024-02-05T04:54:15.045 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:15.046 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.046 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.046 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:15.046 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.182637+0000", 2024-02-05T04:54:15.046 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:15.046 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.046 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.046 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:15.046 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.182637+0000", 2024-02-05T04:54:15.046 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.3000000000000002e-07 2024-02-05T04:54:15.046 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.046 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.046 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:15.047 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.182698+0000", 2024-02-05T04:54:15.047 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.1062000000000006e-05 2024-02-05T04:54:15.047 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.047 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.047 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:15.047 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.182708+0000", 2024-02-05T04:54:15.047 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0186999999999999e-05 2024-02-05T04:54:15.047 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.047 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.047 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:15.047 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.182715+0000", 2024-02-05T04:54:15.047 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.0040000000000002e-06 2024-02-05T04:54:15.047 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.047 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.047 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:15.048 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.182796+0000", 2024-02-05T04:54:15.048 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.0364000000000005e-05 2024-02-05T04:54:15.048 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.048 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.048 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:15.048 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.182805+0000", 2024-02-05T04:54:15.048 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.5990000000000008e-06 2024-02-05T04:54:15.048 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.048 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.048 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:15.048 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.190124+0000", 2024-02-05T04:54:15.048 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0073191009999999997 2024-02-05T04:54:15.048 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.049 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.049 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:15.049 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.190160+0000", 2024-02-05T04:54:15.049 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.6016999999999998e-05 2024-02-05T04:54:15.049 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.049 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:15.049 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.049 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.049 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.049 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:591 3.8s0 3:170878b2:::smithi19639362-30:head [append 0~0] snapc 13=[13,12,11,f,e,d,b,8,7] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-02-05T04:54:15.049 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:54.007323+0000", 2024-02-05T04:54:15.049 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 21.002171800999999, 2024-02-05T04:54:15.049 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.99264834700000004, 2024-02-05T04:54:15.049 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:15.049 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:15.050 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:54:15.050 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:54:15.050 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:15.050 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 591 2024-02-05T04:54:15.050 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.050 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:15.050 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.050 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:15.050 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.007323+0000", 2024-02-05T04:54:15.050 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:15.050 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.050 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.050 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:15.050 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.007323+0000", 2024-02-05T04:54:15.051 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:15.051 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.051 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.051 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:15.051 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.007324+0000", 2024-02-05T04:54:15.051 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.8339999999999999e-06 2024-02-05T04:54:15.051 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.051 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.051 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:15.051 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.007333+0000", 2024-02-05T04:54:15.051 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.5790000000000004e-06 2024-02-05T04:54:15.051 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.051 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.051 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:15.051 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.007353+0000", 2024-02-05T04:54:15.052 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.9568999999999998e-05 2024-02-05T04:54:15.052 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.052 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.052 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:15.052 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.007363+0000", 2024-02-05T04:54:15.052 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0319000000000001e-05 2024-02-05T04:54:15.052 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.052 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.052 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:15.052 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.007446+0000", 2024-02-05T04:54:15.052 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.2924999999999996e-05 2024-02-05T04:54:15.052 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.052 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.052 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for peered", 2024-02-05T04:54:15.053 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.007450+0000", 2024-02-05T04:54:15.053 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.5909999999999998e-06 2024-02-05T04:54:15.053 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.053 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.053 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:15.053 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.986231+0000", 2024-02-05T04:54:15.053 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.97878106300000001 2024-02-05T04:54:15.053 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.053 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.053 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:54:15.053 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.986401+0000", 2024-02-05T04:54:15.053 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00016971600000000001 2024-02-05T04:54:15.053 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.053 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.054 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:15.054 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.987097+0000", 2024-02-05T04:54:15.054 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00069556800000000003 2024-02-05T04:54:15.054 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.054 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.054 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:15.054 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.990602+0000", 2024-02-05T04:54:15.054 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0035056549999999999 2024-02-05T04:54:15.054 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.054 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.054 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:54:15.054 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.999810+0000", 2024-02-05T04:54:15.054 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0092078690000000005 2024-02-05T04:54:15.054 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.055 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.055 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:15.055 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.999971+0000", 2024-02-05T04:54:15.055 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00016065900000000001 2024-02-05T04:54:15.055 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.055 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:15.055 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.055 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.055 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.055 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:592 3.8s0 3:170878b2:::smithi19639362-30:head [rollback 8] snapc 13=[13,12,11,f,e,d,b,8,7] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-02-05T04:54:15.055 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:54.007387+0000", 2024-02-05T04:54:15.055 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 21.002107391999999, 2024-02-05T04:54:15.055 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.99478262799999995, 2024-02-05T04:54:15.055 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:15.056 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:15.056 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:54:15.056 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:54:15.056 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:15.056 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 592 2024-02-05T04:54:15.056 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.056 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:15.056 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.056 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:15.056 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.007387+0000", 2024-02-05T04:54:15.056 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:15.056 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.056 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.057 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:15.057 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.007387+0000", 2024-02-05T04:54:15.057 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:15.057 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.057 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.057 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:15.057 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.007387+0000", 2024-02-05T04:54:15.057 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.3799999999999998e-07 2024-02-05T04:54:15.057 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.057 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.057 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:15.057 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.007390+0000", 2024-02-05T04:54:15.057 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.4030000000000001e-06 2024-02-05T04:54:15.057 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.057 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.058 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:15.058 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.007397+0000", 2024-02-05T04:54:15.058 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.272e-06 2024-02-05T04:54:15.058 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.058 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.058 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:15.058 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.007402+0000", 2024-02-05T04:54:15.058 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.7400000000000004e-06 2024-02-05T04:54:15.058 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.058 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.058 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:15.058 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.007491+0000", 2024-02-05T04:54:15.058 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.9462999999999994e-05 2024-02-05T04:54:15.058 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.058 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.059 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for peered", 2024-02-05T04:54:15.059 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.007493+0000", 2024-02-05T04:54:15.059 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.7540000000000001e-06 2024-02-05T04:54:15.059 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.059 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.059 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:15.059 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.988725+0000", 2024-02-05T04:54:15.059 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.98123155299999998 2024-02-05T04:54:15.059 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.059 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.059 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:54:15.059 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.988828+0000", 2024-02-05T04:54:15.059 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000103463 2024-02-05T04:54:15.059 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.060 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.060 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:15.060 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.989372+0000", 2024-02-05T04:54:15.060 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00054408600000000005 2024-02-05T04:54:15.060 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.060 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.060 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:15.060 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.994357+0000", 2024-02-05T04:54:15.060 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0049851440000000004 2024-02-05T04:54:15.060 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.060 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.060 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:54:15.060 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:55.002096+0000", 2024-02-05T04:54:15.060 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.007738389 2024-02-05T04:54:15.061 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.061 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.061 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:15.061 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:55.002170+0000", 2024-02-05T04:54:15.061 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.3923000000000004e-05 2024-02-05T04:54:15.061 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.061 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:15.061 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.061 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.061 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.061 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:593 3.8s0 3:170878b2:::smithi19639362-30:head [append 0~0] snapc 13=[13,12,11,f,e,d,b,8,7] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-02-05T04:54:15.061 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:54.007407+0000", 2024-02-05T04:54:15.061 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 21.002086904999999, 2024-02-05T04:54:15.061 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.99488064899999995, 2024-02-05T04:54:15.061 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:15.062 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:15.062 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:54:15.062 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:54:15.062 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:15.062 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 593 2024-02-05T04:54:15.062 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.062 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:15.062 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.062 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:15.062 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.007407+0000", 2024-02-05T04:54:15.062 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:15.062 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.062 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.062 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:15.063 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.007407+0000", 2024-02-05T04:54:15.063 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:15.063 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.063 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.063 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:15.063 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.007408+0000", 2024-02-05T04:54:15.063 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.0400000000000002e-07 2024-02-05T04:54:15.063 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.063 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.063 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:15.063 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.007413+0000", 2024-02-05T04:54:15.063 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.9790000000000001e-06 2024-02-05T04:54:15.063 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.063 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.063 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:15.064 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.007418+0000", 2024-02-05T04:54:15.064 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.8180000000000003e-06 2024-02-05T04:54:15.064 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.064 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.064 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:15.064 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.007422+0000", 2024-02-05T04:54:15.064 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.1930000000000003e-06 2024-02-05T04:54:15.064 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.064 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.064 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:15.064 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.007524+0000", 2024-02-05T04:54:15.064 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000101808 2024-02-05T04:54:15.065 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.065 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.065 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for peered", 2024-02-05T04:54:15.065 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.007525+0000", 2024-02-05T04:54:15.065 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.2330000000000001e-06 2024-02-05T04:54:15.065 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.065 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.065 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:15.065 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.989757+0000", 2024-02-05T04:54:15.065 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.98223166900000003 2024-02-05T04:54:15.065 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.065 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.065 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:54:15.066 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.989846+0000", 2024-02-05T04:54:15.066 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.9580999999999995e-05 2024-02-05T04:54:15.066 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.066 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.066 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:15.066 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.990320+0000", 2024-02-05T04:54:15.066 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00047333 2024-02-05T04:54:15.066 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.066 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.066 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:15.066 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.994359+0000", 2024-02-05T04:54:15.066 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0040392400000000004 2024-02-05T04:54:15.066 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.067 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.067 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:54:15.067 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:55.002244+0000", 2024-02-05T04:54:15.067 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0078852279999999993 2024-02-05T04:54:15.067 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.067 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.067 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:15.067 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:55.002288+0000", 2024-02-05T04:54:15.067 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.4165999999999997e-05 2024-02-05T04:54:15.067 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.067 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:15.067 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.067 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.067 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.067 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:629 3.3s0 3:c46da155:::smithi19639362-15:head [append 0~770048] snapc 16=[16,15,12,11,f,b,8,7] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-02-05T04:54:15.068 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:54:00.957118+0000", 2024-02-05T04:54:15.068 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 14.052376803, 2024-02-05T04:54:15.068 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.016343434, 2024-02-05T04:54:15.068 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:15.068 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:15.068 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:54:15.068 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:54:15.068 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:15.068 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 629 2024-02-05T04:54:15.068 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.068 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:15.068 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.068 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:15.069 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.957118+0000", 2024-02-05T04:54:15.069 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:15.069 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.069 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.069 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:15.069 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.957118+0000", 2024-02-05T04:54:15.069 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:15.069 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.069 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.069 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:15.069 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.957121+0000", 2024-02-05T04:54:15.069 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.0910000000000001e-06 2024-02-05T04:54:15.069 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.069 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.069 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:15.070 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.958839+0000", 2024-02-05T04:54:15.070 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.001718682 2024-02-05T04:54:15.070 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.070 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.070 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:15.070 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.958876+0000", 2024-02-05T04:54:15.070 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.6293e-05 2024-02-05T04:54:15.070 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.070 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.070 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:15.070 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.958889+0000", 2024-02-05T04:54:15.070 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.3664e-05 2024-02-05T04:54:15.070 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.070 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.070 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:15.071 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.958976+0000", 2024-02-05T04:54:15.071 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.6207999999999999e-05 2024-02-05T04:54:15.071 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.071 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.071 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "op must wait for map", 2024-02-05T04:54:15.071 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.958983+0000", 2024-02-05T04:54:15.071 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.7260000000000002e-06 2024-02-05T04:54:15.071 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.071 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.071 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:15.071 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.964806+0000", 2024-02-05T04:54:15.071 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0058228869999999997 2024-02-05T04:54:15.071 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.071 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.072 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:54:15.072 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.964928+0000", 2024-02-05T04:54:15.072 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000121997 2024-02-05T04:54:15.072 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.072 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.072 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:15.072 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.965928+0000", 2024-02-05T04:54:15.072 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0010002850000000001 2024-02-05T04:54:15.072 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.072 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.072 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:15.072 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.973092+0000", 2024-02-05T04:54:15.072 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0071639190000000004 2024-02-05T04:54:15.072 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.073 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.073 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:54:15.073 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.973366+0000", 2024-02-05T04:54:15.073 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000273467 2024-02-05T04:54:15.073 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.073 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.073 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:15.073 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.973461+0000", 2024-02-05T04:54:15.073 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.5215000000000002e-05 2024-02-05T04:54:15.073 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.073 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:15.073 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.073 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.073 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.074 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:630 3.3s0 3:c46da155:::smithi19639362-15:head [append 0~704512] snapc 16=[16,15,12,11,f,b,8,7] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-02-05T04:54:15.074 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:54:00.961143+0000", 2024-02-05T04:54:15.074 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 14.048351363, 2024-02-05T04:54:15.074 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.014004825, 2024-02-05T04:54:15.074 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:15.074 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:15.074 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:54:15.074 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:54:15.074 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:15.074 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 630 2024-02-05T04:54:15.074 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.074 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:15.074 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.074 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:15.075 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.961143+0000", 2024-02-05T04:54:15.075 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:15.075 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.075 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.075 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:15.075 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.961143+0000", 2024-02-05T04:54:15.075 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:15.075 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.075 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.075 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:15.075 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.961145+0000", 2024-02-05T04:54:15.075 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.716e-06 2024-02-05T04:54:15.075 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.075 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.075 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:15.076 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.962094+0000", 2024-02-05T04:54:15.076 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00094879700000000003 2024-02-05T04:54:15.076 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.076 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.076 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:15.076 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.962105+0000", 2024-02-05T04:54:15.076 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1209999999999999e-05 2024-02-05T04:54:15.076 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.076 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.076 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:15.076 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.962111+0000", 2024-02-05T04:54:15.076 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.9510000000000003e-06 2024-02-05T04:54:15.076 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.076 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.077 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:15.077 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.962167+0000", 2024-02-05T04:54:15.077 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.6020999999999998e-05 2024-02-05T04:54:15.077 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.077 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.077 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting_for_map not empty", 2024-02-05T04:54:15.077 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.962172+0000", 2024-02-05T04:54:15.077 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.6860000000000003e-06 2024-02-05T04:54:15.077 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.077 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.077 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:15.077 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.966743+0000", 2024-02-05T04:54:15.077 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0045707639999999997 2024-02-05T04:54:15.077 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.077 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.077 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:54:15.078 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.966868+0000", 2024-02-05T04:54:15.078 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00012479099999999999 2024-02-05T04:54:15.078 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.078 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.078 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:15.078 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.967757+0000", 2024-02-05T04:54:15.078 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00088885699999999995 2024-02-05T04:54:15.078 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.078 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.078 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:15.078 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.974750+0000", 2024-02-05T04:54:15.078 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0069934250000000002 2024-02-05T04:54:15.078 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.079 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.079 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:54:15.079 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.975055+0000", 2024-02-05T04:54:15.079 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00030481699999999998 2024-02-05T04:54:15.079 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.079 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.079 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:15.079 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.975148+0000", 2024-02-05T04:54:15.079 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.2789999999999998e-05 2024-02-05T04:54:15.079 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.079 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:15.079 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.079 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.079 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.079 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:631 3.3s0 3:c46da155:::smithi19639362-15:head [append 0~458752] snapc 16=[16,15,12,11,f,b,8,7] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-02-05T04:54:15.080 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:54:00.964145+0000", 2024-02-05T04:54:15.080 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 14.045349879, 2024-02-05T04:54:15.080 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.012240795, 2024-02-05T04:54:15.080 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:15.080 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:15.080 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:54:15.080 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:54:15.080 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:15.080 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 631 2024-02-05T04:54:15.080 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.080 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:15.080 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.080 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:15.080 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.964145+0000", 2024-02-05T04:54:15.081 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:15.081 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.081 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.081 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:15.081 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.964145+0000", 2024-02-05T04:54:15.081 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:15.081 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.081 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.081 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:15.081 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.964146+0000", 2024-02-05T04:54:15.081 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.5689999999999999e-06 2024-02-05T04:54:15.081 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.081 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.081 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:15.081 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.965007+0000", 2024-02-05T04:54:15.082 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00086071499999999996 2024-02-05T04:54:15.082 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.082 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.082 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:15.082 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.965017+0000", 2024-02-05T04:54:15.082 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0643e-05 2024-02-05T04:54:15.082 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.082 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.082 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:15.082 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.965024+0000", 2024-02-05T04:54:15.082 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.4659999999999996e-06 2024-02-05T04:54:15.082 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.082 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.082 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:15.083 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.970390+0000", 2024-02-05T04:54:15.083 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00536587 2024-02-05T04:54:15.083 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.083 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.083 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:54:15.083 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.970537+0000", 2024-02-05T04:54:15.083 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00014697899999999999 2024-02-05T04:54:15.083 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.083 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.083 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:15.083 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.971373+0000", 2024-02-05T04:54:15.083 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00083609699999999995 2024-02-05T04:54:15.083 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.083 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.083 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:15.084 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.975431+0000", 2024-02-05T04:54:15.084 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0040576910000000004 2024-02-05T04:54:15.084 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.084 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.084 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:54:15.084 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.976344+0000", 2024-02-05T04:54:15.084 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00091364799999999998 2024-02-05T04:54:15.084 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.084 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.084 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:15.084 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.976385+0000", 2024-02-05T04:54:15.084 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.1117e-05 2024-02-05T04:54:15.084 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.084 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:15.084 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.085 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.085 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.085 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:632 3.3s0 3:c46da155:::smithi19639362-15:head [setxattr _header (57) in=7b] snapc 16=[16,15,12,11,f,b,8,7] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-02-05T04:54:15.085 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:54:00.965035+0000", 2024-02-05T04:54:15.085 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 14.044459145999999, 2024-02-05T04:54:15.085 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.011543045, 2024-02-05T04:54:15.085 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:15.085 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:15.085 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:54:15.085 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:54:15.085 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:15.085 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 632 2024-02-05T04:54:15.085 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.085 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:15.086 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.086 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:15.086 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.965035+0000", 2024-02-05T04:54:15.086 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:15.086 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.086 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.086 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:15.086 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.965035+0000", 2024-02-05T04:54:15.086 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:15.086 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.086 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.086 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:15.086 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.965036+0000", 2024-02-05T04:54:15.086 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.99e-07 2024-02-05T04:54:15.087 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.087 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.087 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:15.087 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.965038+0000", 2024-02-05T04:54:15.087 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.604e-06 2024-02-05T04:54:15.087 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.087 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.087 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:15.087 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.965043+0000", 2024-02-05T04:54:15.087 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.0999999999999997e-06 2024-02-05T04:54:15.087 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.087 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.087 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:15.087 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.965046+0000", 2024-02-05T04:54:15.088 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.1410000000000001e-06 2024-02-05T04:54:15.088 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.088 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.088 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:15.088 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.972375+0000", 2024-02-05T04:54:15.088 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0073292299999999999 2024-02-05T04:54:15.088 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.088 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.088 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:54:15.088 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.972473+0000", 2024-02-05T04:54:15.088 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.8414999999999999e-05 2024-02-05T04:54:15.088 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.088 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.088 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:15.088 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.972832+0000", 2024-02-05T04:54:15.089 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00035884900000000001 2024-02-05T04:54:15.089 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.089 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.089 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:15.089 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.975432+0000", 2024-02-05T04:54:15.089 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.002600265 2024-02-05T04:54:15.089 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.089 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.089 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:54:15.089 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.976514+0000", 2024-02-05T04:54:15.089 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.001081551 2024-02-05T04:54:15.089 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.089 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.089 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:15.090 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.976578+0000", 2024-02-05T04:54:15.090 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.4290999999999998e-05 2024-02-05T04:54:15.090 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.090 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:15.090 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.090 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.090 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.090 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:633 3.3s0 3:c46da155:::smithi19639362-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e69)", 2024-02-05T04:54:15.090 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:54:00.965050+0000", 2024-02-05T04:54:15.090 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 14.044444427, 2024-02-05T04:54:15.090 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.013116340000000001, 2024-02-05T04:54:15.090 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:15.090 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "started", 2024-02-05T04:54:15.090 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:54:15.090 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:54:15.091 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:15.091 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 633 2024-02-05T04:54:15.091 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.091 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:15.091 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.091 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:15.091 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.965050+0000", 2024-02-05T04:54:15.091 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:15.091 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.091 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.091 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:15.091 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.965050+0000", 2024-02-05T04:54:15.091 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:15.091 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.091 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.092 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:15.092 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.965050+0000", 2024-02-05T04:54:15.092 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.6800000000000002e-07 2024-02-05T04:54:15.092 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.092 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.092 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:15.092 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.965052+0000", 2024-02-05T04:54:15.092 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.4249999999999999e-06 2024-02-05T04:54:15.092 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.092 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.092 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:15.092 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.965055+0000", 2024-02-05T04:54:15.092 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.8290000000000001e-06 2024-02-05T04:54:15.092 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.092 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.093 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:15.093 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.965058+0000", 2024-02-05T04:54:15.093 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.9560000000000002e-06 2024-02-05T04:54:15.093 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.093 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.093 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:15.093 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.973682+0000", 2024-02-05T04:54:15.093 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0086237540000000008 2024-02-05T04:54:15.093 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.093 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.093 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for rw locks", 2024-02-05T04:54:15.093 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.973747+0000", 2024-02-05T04:54:15.093 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.4393000000000005e-05 2024-02-05T04:54:15.093 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.093 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.094 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:15.094 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.976885+0000", 2024-02-05T04:54:15.094 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0031381040000000001 2024-02-05T04:54:15.094 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.094 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.094 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:54:15.094 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.976961+0000", 2024-02-05T04:54:15.094 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.6206999999999997e-05 2024-02-05T04:54:15.094 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.094 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.094 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:15.094 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.978166+0000", 2024-02-05T04:54:15.094 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.001205404 2024-02-05T04:54:15.094 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.095 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:15.095 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.095 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.095 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.095 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.6s2 74/62 ECSubWrite(tid=337, reqid=osd.13.0:337, at_version=74'38, trim_to=0'0, roll_forward_to=72'36))", 2024-02-05T04:54:15.095 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:54:06.043590+0000", 2024-02-05T04:54:15.095 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 8.9659047960000002, 2024-02-05T04:54:15.095 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0078961069999999994, 2024-02-05T04:54:15.095 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:15.095 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:54:15.095 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:15.095 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.095 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:15.095 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.043590+0000", 2024-02-05T04:54:15.095 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:15.096 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.096 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.096 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:15.096 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.043590+0000", 2024-02-05T04:54:15.096 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:15.096 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.096 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.096 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:15.096 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.043590+0000", 2024-02-05T04:54:15.096 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.6800000000000002e-07 2024-02-05T04:54:15.096 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.096 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.096 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:15.096 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.043616+0000", 2024-02-05T04:54:15.097 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.6449e-05 2024-02-05T04:54:15.097 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.097 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.097 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:15.097 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.043626+0000", 2024-02-05T04:54:15.097 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.1900000000000001e-06 2024-02-05T04:54:15.097 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.097 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.097 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:15.097 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.043634+0000", 2024-02-05T04:54:15.097 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.4530000000000002e-06 2024-02-05T04:54:15.097 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.097 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.097 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:15.098 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.043691+0000", 2024-02-05T04:54:15.098 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.7466999999999997e-05 2024-02-05T04:54:15.098 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.098 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.098 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:15.098 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.043701+0000", 2024-02-05T04:54:15.098 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.4860000000000003e-06 2024-02-05T04:54:15.098 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.098 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.098 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:15.098 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.051445+0000", 2024-02-05T04:54:15.098 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0077437119999999998 2024-02-05T04:54:15.098 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.098 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.098 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:15.099 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.051486+0000", 2024-02-05T04:54:15.099 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.1081999999999997e-05 2024-02-05T04:54:15.099 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.099 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:15.099 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.099 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.099 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.099 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.17s3 74/67 ECSubWrite(tid=4, reqid=osd.0.0:4, at_version=74'32, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:15.099 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:54:06.045714+0000", 2024-02-05T04:54:15.099 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 8.9637799719999993, 2024-02-05T04:54:15.099 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0091829640000000001, 2024-02-05T04:54:15.099 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:15.099 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:54:15.099 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:15.100 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.100 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:15.100 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.045714+0000", 2024-02-05T04:54:15.100 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:15.100 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.100 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.100 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:15.100 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.045714+0000", 2024-02-05T04:54:15.100 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:15.100 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.100 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.100 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:15.100 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.045715+0000", 2024-02-05T04:54:15.100 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.1699999999999998e-07 2024-02-05T04:54:15.101 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.101 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.101 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:15.101 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.045736+0000", 2024-02-05T04:54:15.101 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.0944000000000001e-05 2024-02-05T04:54:15.101 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.101 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.101 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:15.101 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.045747+0000", 2024-02-05T04:54:15.101 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1151e-05 2024-02-05T04:54:15.101 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.101 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.101 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:15.101 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.045755+0000", 2024-02-05T04:54:15.102 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.0299999999999994e-06 2024-02-05T04:54:15.102 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.102 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.102 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:15.102 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.045838+0000", 2024-02-05T04:54:15.102 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.3269000000000005e-05 2024-02-05T04:54:15.102 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.102 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.102 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:15.102 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.045848+0000", 2024-02-05T04:54:15.102 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.1099999999999992e-06 2024-02-05T04:54:15.102 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.102 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.102 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:15.102 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.054858+0000", 2024-02-05T04:54:15.102 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0090107869999999993 2024-02-05T04:54:15.103 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.103 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.103 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:15.103 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.054897+0000", 2024-02-05T04:54:15.103 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.9056e-05 2024-02-05T04:54:15.103 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.103 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:15.103 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.103 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.103 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.103 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.es1 74/57 ECSubWrite(tid=98, reqid=osd.2.0:97, at_version=0'0, trim_to=0'0, roll_forward_to=74'68))", 2024-02-05T04:54:15.103 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:54:06.046317+0000", 2024-02-05T04:54:15.103 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 8.9631773839999997, 2024-02-05T04:54:15.103 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0086387900000000004, 2024-02-05T04:54:15.103 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:15.104 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:54:15.104 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:15.104 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.104 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:15.104 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.046317+0000", 2024-02-05T04:54:15.104 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:15.104 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.104 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.104 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:15.104 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.046317+0000", 2024-02-05T04:54:15.104 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:15.104 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.104 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.104 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:15.105 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.046318+0000", 2024-02-05T04:54:15.105 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.0900000000000001e-07 2024-02-05T04:54:15.105 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.105 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.105 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:15.105 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.046328+0000", 2024-02-05T04:54:15.105 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0336000000000001e-05 2024-02-05T04:54:15.105 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.105 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.105 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:15.105 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.046345+0000", 2024-02-05T04:54:15.105 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.6697999999999998e-05 2024-02-05T04:54:15.105 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.105 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.105 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:15.106 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.046354+0000", 2024-02-05T04:54:15.106 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.9770000000000006e-06 2024-02-05T04:54:15.106 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.106 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.106 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:15.106 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.046435+0000", 2024-02-05T04:54:15.106 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.1154000000000003e-05 2024-02-05T04:54:15.106 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.106 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.106 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:15.106 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.046444+0000", 2024-02-05T04:54:15.106 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.4700000000000008e-06 2024-02-05T04:54:15.106 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.106 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.107 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:15.107 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.054915+0000", 2024-02-05T04:54:15.107 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0084709250000000007 2024-02-05T04:54:15.107 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.107 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.107 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:15.107 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.054956+0000", 2024-02-05T04:54:15.107 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.0521e-05 2024-02-05T04:54:15.107 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.107 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:15.107 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.107 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.107 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.107 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.4s5 74/62 ECSubWrite(tid=132, reqid=osd.1.0:132, at_version=74'81, trim_to=0'0, roll_forward_to=72'79))", 2024-02-05T04:54:15.107 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:54:06.052059+0000", 2024-02-05T04:54:15.108 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 8.9574349229999992, 2024-02-05T04:54:15.108 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0080075749999999994, 2024-02-05T04:54:15.108 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:15.108 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:54:15.108 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:15.108 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.108 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:15.108 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.052059+0000", 2024-02-05T04:54:15.108 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:15.108 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.108 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.108 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:15.108 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.052059+0000", 2024-02-05T04:54:15.109 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:15.109 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.109 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.109 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:15.109 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.052060+0000", 2024-02-05T04:54:15.109 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.3099999999999997e-07 2024-02-05T04:54:15.109 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.109 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.109 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:15.109 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.052080+0000", 2024-02-05T04:54:15.109 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.9690999999999998e-05 2024-02-05T04:54:15.109 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.109 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.109 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:15.109 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.052092+0000", 2024-02-05T04:54:15.110 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.2381000000000001e-05 2024-02-05T04:54:15.110 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.110 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.110 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:15.110 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.052101+0000", 2024-02-05T04:54:15.110 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.7309999999999999e-06 2024-02-05T04:54:15.110 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.110 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.110 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:15.110 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.052161+0000", 2024-02-05T04:54:15.110 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.0084000000000003e-05 2024-02-05T04:54:15.110 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.110 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.110 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:15.111 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.052171+0000", 2024-02-05T04:54:15.111 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0025e-05 2024-02-05T04:54:15.111 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.111 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.111 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:15.111 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.060030+0000", 2024-02-05T04:54:15.111 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0078588890000000008 2024-02-05T04:54:15.111 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.111 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.111 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:15.111 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.060067+0000", 2024-02-05T04:54:15.111 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.7143e-05 2024-02-05T04:54:15.111 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.111 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:15.111 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.112 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.112 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.112 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:794 3.3s0 3:c46da155:::smithi19639362-15:head [append 0~720896] snapc 1b=[1b,19,18,17,16,12,8,7] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-02-05T04:54:15.112 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:54:11.082842+0000", 2024-02-05T04:54:15.112 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 3.9266527180000002, 2024-02-05T04:54:15.112 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0077717130000000004, 2024-02-05T04:54:15.112 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:15.112 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:15.112 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:54:15.112 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:54:15.112 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:15.112 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 794 2024-02-05T04:54:15.112 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.112 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:15.113 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.113 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:15.113 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:11.082842+0000", 2024-02-05T04:54:15.113 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:15.113 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.113 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.113 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:15.113 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:11.082842+0000", 2024-02-05T04:54:15.113 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:15.113 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.113 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.113 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:15.113 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:11.082845+0000", 2024-02-05T04:54:15.113 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.8930000000000001e-06 2024-02-05T04:54:15.113 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.113 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.114 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:15.114 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:11.083713+0000", 2024-02-05T04:54:15.114 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00086801200000000004 2024-02-05T04:54:15.114 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.114 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.114 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:15.114 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:11.083738+0000", 2024-02-05T04:54:15.114 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.5823999999999999e-05 2024-02-05T04:54:15.114 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.114 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.114 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:15.114 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:11.083751+0000", 2024-02-05T04:54:15.114 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.2843999999999999e-05 2024-02-05T04:54:15.114 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.114 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.115 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:15.115 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:11.083818+0000", 2024-02-05T04:54:15.115 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.6307000000000001e-05 2024-02-05T04:54:15.115 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.115 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.115 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "op must wait for map", 2024-02-05T04:54:15.115 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:11.083824+0000", 2024-02-05T04:54:15.115 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.1820000000000003e-06 2024-02-05T04:54:15.115 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.115 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.115 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:15.115 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:11.085345+0000", 2024-02-05T04:54:15.115 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.001521047 2024-02-05T04:54:15.115 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.116 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.116 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:54:15.116 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:11.085453+0000", 2024-02-05T04:54:15.116 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000108039 2024-02-05T04:54:15.116 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.116 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.116 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:15.116 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:11.086618+0000", 2024-02-05T04:54:15.116 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0011647109999999999 2024-02-05T04:54:15.116 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.116 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.116 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:15.116 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:11.088676+0000", 2024-02-05T04:54:15.116 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0020582230000000001 2024-02-05T04:54:15.116 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.117 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.117 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:54:15.117 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:11.090562+0000", 2024-02-05T04:54:15.117 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.001886176 2024-02-05T04:54:15.117 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.117 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.117 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:15.117 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:11.090613+0000", 2024-02-05T04:54:15.117 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.1455000000000001e-05 2024-02-05T04:54:15.117 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.117 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:15.117 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.117 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.117 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:15.118 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-02-05T04:54:15.118 INFO:tasks.rados.rados.0.smithi196.stdout:337: done (3 left) 2024-02-05T04:54:15.119 INFO:tasks.rados.rados.0.smithi196.stdout:338: delete oid 6 current snap is 28 2024-02-05T04:54:15.120 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-02-05T04:54:15.121 INFO:teuthology.orchestra.run.smithi062.stderr:noscrub is unset 2024-02-05T04:54:15.121 INFO:tasks.ceph.osd.15.smithi196.stderr:2024-02-05T04:54:15.118+0000 7f1facc45640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 34611) UID: 0 2024-02-05T04:54:15.220 INFO:tasks.ceph.osd.8.smithi062.stderr:2024-02-05T04:54:15.218+0000 7f9385bf2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 34693) UID: 0 2024-02-05T04:54:15.268 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-02-05T04:54:15.268 INFO:teuthology.orchestra.run.smithi062.stdout: "ops": [], 2024-02-05T04:54:15.268 INFO:teuthology.orchestra.run.smithi062.stdout: "num_ops": 0 2024-02-05T04:54:15.268 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-02-05T04:54:15.277 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-02-05T04:54:15.301 INFO:teuthology.orchestra.run.smithi091.stdout:3.ds5 2024-02-05T04:54:15.301 INFO:teuthology.orchestra.run.smithi091.stdout:3.bs0 2024-02-05T04:54:15.301 INFO:teuthology.orchestra.run.smithi091.stdout:3.6s0 2024-02-05T04:54:15.302 INFO:teuthology.orchestra.run.smithi091.stdout:3.cs1 2024-02-05T04:54:15.302 INFO:teuthology.orchestra.run.smithi091.stdout:3.5s1 2024-02-05T04:54:15.302 INFO:teuthology.orchestra.run.smithi091.stdout:3.es4 2024-02-05T04:54:15.302 INFO:teuthology.orchestra.run.smithi091.stdout:3.9s2 2024-02-05T04:54:15.302 INFO:teuthology.orchestra.run.smithi091.stdout:3.7s0 2024-02-05T04:54:15.302 INFO:teuthology.orchestra.run.smithi091.stdout:3.4s6 2024-02-05T04:54:15.302 INFO:teuthology.orchestra.run.smithi091.stdout:3.1s0 2024-02-05T04:54:15.302 INFO:teuthology.orchestra.run.smithi091.stdout:3.18s3 2024-02-05T04:54:15.302 INFO:teuthology.orchestra.run.smithi091.stdout:3.14s6 2024-02-05T04:54:15.361 INFO:tasks.ceph.osd.0.smithi062.stderr:2024-02-05T04:54:15.359+0000 7fece8e03640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34688) UID: 0 2024-02-05T04:54:15.421 INFO:tasks.ceph.osd.2.smithi150.stderr:2024-02-05T04:54:15.421+0000 7f7337bf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 34014) UID: 0 2024-02-05T04:54:15.424 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-02-05T04:54:15.424 INFO:teuthology.orchestra.run.smithi062.stdout: "ops": [], 2024-02-05T04:54:15.424 INFO:teuthology.orchestra.run.smithi062.stdout: "complaint_time": 30, 2024-02-05T04:54:15.424 INFO:teuthology.orchestra.run.smithi062.stdout: "num_blocked_ops": 0 2024-02-05T04:54:15.424 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-02-05T04:54:15.433 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-02-05T04:54:15.523 INFO:tasks.ceph.osd.2.smithi150.stderr:2024-02-05T04:54:15.522+0000 7f7337bf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 34014) UID: 0 2024-02-05T04:54:15.582 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-02-05T04:54:15.582 INFO:teuthology.orchestra.run.smithi062.stdout: "size": 20, 2024-02-05T04:54:15.582 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 600, 2024-02-05T04:54:15.582 INFO:teuthology.orchestra.run.smithi062.stdout: "ops": [ 2024-02-05T04:54:15.582 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.582 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4209.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-02-05T04:54:15.582 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:08.919510+0000", 2024-02-05T04:54:15.583 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 66.660529170999993, 2024-02-05T04:54:15.583 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.70688627900000001, 2024-02-05T04:54:15.583 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:15.583 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:15.583 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:54:15.583 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4209", 2024-02-05T04:54:15.583 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3672061555", 2024-02-05T04:54:15.583 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 1 2024-02-05T04:54:15.583 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.583 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:15.583 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.583 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:15.583 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:08.919510+0000", 2024-02-05T04:54:15.583 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:15.584 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.584 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.584 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:15.584 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:08.919510+0000", 2024-02-05T04:54:15.584 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:15.584 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.584 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.584 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:15.584 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:08.919512+0000", 2024-02-05T04:54:15.584 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1620000000000001e-06 2024-02-05T04:54:15.584 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.584 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.584 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:15.584 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:08.919525+0000", 2024-02-05T04:54:15.585 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.3369e-05 2024-02-05T04:54:15.585 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.585 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.585 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:15.585 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:08.919536+0000", 2024-02-05T04:54:15.585 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0621e-05 2024-02-05T04:54:15.585 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.585 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.585 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:15.585 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:08.919550+0000", 2024-02-05T04:54:15.585 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.403e-05 2024-02-05T04:54:15.585 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.585 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.585 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:15.586 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:08.919625+0000", 2024-02-05T04:54:15.586 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.5328999999999997e-05 2024-02-05T04:54:15.586 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.586 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.586 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for peered", 2024-02-05T04:54:15.586 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:08.919634+0000", 2024-02-05T04:54:15.586 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.4879999999999996e-06 2024-02-05T04:54:15.586 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.586 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.586 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:15.586 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.624946+0000", 2024-02-05T04:54:15.586 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.70531134600000001 2024-02-05T04:54:15.586 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.586 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.586 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:54:15.587 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.625129+0000", 2024-02-05T04:54:15.587 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00018288199999999999 2024-02-05T04:54:15.587 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.587 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.587 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for subops from 2", 2024-02-05T04:54:15.587 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.625339+0000", 2024-02-05T04:54:15.587 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00021016900000000001 2024-02-05T04:54:15.587 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.587 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.587 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "op_commit", 2024-02-05T04:54:15.587 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.625989+0000", 2024-02-05T04:54:15.587 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00065006700000000005 2024-02-05T04:54:15.587 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.587 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.587 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_commit_rec", 2024-02-05T04:54:15.588 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.626301+0000", 2024-02-05T04:54:15.588 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00031220800000000002 2024-02-05T04:54:15.588 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.588 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.588 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:54:15.588 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.626352+0000", 2024-02-05T04:54:15.588 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.0973000000000001e-05 2024-02-05T04:54:15.588 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.588 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.588 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:15.588 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.626397+0000", 2024-02-05T04:54:15.588 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.4635000000000002e-05 2024-02-05T04:54:15.588 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.588 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:15.589 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.589 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.589 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.589 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4209.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-02-05T04:54:15.589 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:09.696463+0000", 2024-02-05T04:54:15.589 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 65.883576939999998, 2024-02-05T04:54:15.589 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0095939449999999996, 2024-02-05T04:54:15.589 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:15.589 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:15.589 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:54:15.589 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4209", 2024-02-05T04:54:15.589 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3672061555", 2024-02-05T04:54:15.589 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 85 2024-02-05T04:54:15.589 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.590 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:15.590 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.590 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:15.590 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.696463+0000", 2024-02-05T04:54:15.590 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:15.590 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.590 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.590 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:15.590 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.696463+0000", 2024-02-05T04:54:15.590 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:15.590 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.590 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.590 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:15.590 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.696463+0000", 2024-02-05T04:54:15.590 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.9400000000000001e-07 2024-02-05T04:54:15.591 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.591 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.591 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:15.591 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.696465+0000", 2024-02-05T04:54:15.591 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.1019999999999999e-06 2024-02-05T04:54:15.591 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.591 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.591 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:15.591 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.696470+0000", 2024-02-05T04:54:15.591 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.6099999999999999e-06 2024-02-05T04:54:15.591 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.591 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.591 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:15.591 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.696473+0000", 2024-02-05T04:54:15.592 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.8949999999999998e-06 2024-02-05T04:54:15.592 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.592 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.592 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:15.592 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.697422+0000", 2024-02-05T04:54:15.592 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00094939000000000004 2024-02-05T04:54:15.592 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.592 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.592 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:54:15.592 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.697489+0000", 2024-02-05T04:54:15.592 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.7293999999999996e-05 2024-02-05T04:54:15.592 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.592 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.592 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for subops from 2", 2024-02-05T04:54:15.593 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.697604+0000", 2024-02-05T04:54:15.593 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000114288 2024-02-05T04:54:15.593 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.593 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.593 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "op_commit", 2024-02-05T04:54:15.593 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.702424+0000", 2024-02-05T04:54:15.593 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0048202649999999998 2024-02-05T04:54:15.593 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.593 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.593 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_commit_rec", 2024-02-05T04:54:15.593 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.706007+0000", 2024-02-05T04:54:15.593 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.003583059 2024-02-05T04:54:15.593 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.593 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.594 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:54:15.594 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.706037+0000", 2024-02-05T04:54:15.594 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.9944999999999999e-05 2024-02-05T04:54:15.594 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.594 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.594 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:15.594 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.706057+0000", 2024-02-05T04:54:15.594 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.9803e-05 2024-02-05T04:54:15.594 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.594 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:15.594 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.594 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.594 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.594 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4209.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-02-05T04:54:15.594 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:09.696477+0000", 2024-02-05T04:54:15.595 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 65.883562585999996, 2024-02-05T04:54:15.595 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0096605600000000003, 2024-02-05T04:54:15.595 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:15.595 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:15.595 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:54:15.595 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4209", 2024-02-05T04:54:15.595 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3672061555", 2024-02-05T04:54:15.595 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 86 2024-02-05T04:54:15.595 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.595 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:15.595 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.595 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:15.595 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.696477+0000", 2024-02-05T04:54:15.595 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:15.595 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.596 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.596 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:15.596 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.696477+0000", 2024-02-05T04:54:15.596 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:15.596 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.596 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.596 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:15.596 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.696477+0000", 2024-02-05T04:54:15.596 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.3200000000000001e-07 2024-02-05T04:54:15.596 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.596 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.596 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:15.596 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.696480+0000", 2024-02-05T04:54:15.596 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.3779999999999999e-06 2024-02-05T04:54:15.597 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.597 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.597 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:15.597 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.696483+0000", 2024-02-05T04:54:15.597 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.23e-06 2024-02-05T04:54:15.597 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.597 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.597 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:15.597 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.696485+0000", 2024-02-05T04:54:15.597 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.4200000000000001e-06 2024-02-05T04:54:15.597 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.597 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.597 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:15.597 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.697977+0000", 2024-02-05T04:54:15.597 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0014921299999999999 2024-02-05T04:54:15.598 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.598 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.598 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:54:15.598 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.698018+0000", 2024-02-05T04:54:15.598 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.0676999999999997e-05 2024-02-05T04:54:15.598 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.598 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.598 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for subops from 2", 2024-02-05T04:54:15.598 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.698082+0000", 2024-02-05T04:54:15.598 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.3466000000000005e-05 2024-02-05T04:54:15.598 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.598 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.598 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "op_commit", 2024-02-05T04:54:15.598 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.702654+0000", 2024-02-05T04:54:15.598 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0045719710000000002 2024-02-05T04:54:15.599 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.599 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.599 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_commit_rec", 2024-02-05T04:54:15.599 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.706093+0000", 2024-02-05T04:54:15.599 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0034398749999999998 2024-02-05T04:54:15.599 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.599 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.599 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:54:15.599 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.706120+0000", 2024-02-05T04:54:15.599 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.6801999999999999e-05 2024-02-05T04:54:15.599 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.599 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.599 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:15.599 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.706138+0000", 2024-02-05T04:54:15.600 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.7379000000000001e-05 2024-02-05T04:54:15.600 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.600 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:15.600 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.600 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.600 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.600 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4209.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-02-05T04:54:15.600 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:09.696526+0000", 2024-02-05T04:54:15.600 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 65.883513876999999, 2024-02-05T04:54:15.600 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0096876010000000005, 2024-02-05T04:54:15.600 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:15.600 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:15.600 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:54:15.600 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4209", 2024-02-05T04:54:15.601 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3672061555", 2024-02-05T04:54:15.601 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 87 2024-02-05T04:54:15.601 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.601 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:15.601 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.601 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:15.601 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.696526+0000", 2024-02-05T04:54:15.601 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:15.601 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.601 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.601 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:15.601 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.696526+0000", 2024-02-05T04:54:15.601 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:15.601 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.601 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.602 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:15.602 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.696526+0000", 2024-02-05T04:54:15.602 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.2899999999999999e-07 2024-02-05T04:54:15.602 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.602 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.602 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:15.602 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.696602+0000", 2024-02-05T04:54:15.602 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.5636999999999994e-05 2024-02-05T04:54:15.602 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.602 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.602 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:15.602 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.696607+0000", 2024-02-05T04:54:15.602 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.463e-06 2024-02-05T04:54:15.602 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.603 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.603 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:15.603 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.696611+0000", 2024-02-05T04:54:15.603 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.4199999999999999e-06 2024-02-05T04:54:15.603 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.603 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.603 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:15.603 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.698261+0000", 2024-02-05T04:54:15.603 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0016498039999999999 2024-02-05T04:54:15.603 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.603 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.603 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:54:15.603 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.698300+0000", 2024-02-05T04:54:15.603 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.9468000000000002e-05 2024-02-05T04:54:15.604 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.604 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.604 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for subops from 2", 2024-02-05T04:54:15.604 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.698363+0000", 2024-02-05T04:54:15.604 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.3360000000000003e-05 2024-02-05T04:54:15.604 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.604 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.604 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "op_commit", 2024-02-05T04:54:15.604 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.702658+0000", 2024-02-05T04:54:15.604 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0042946929999999996 2024-02-05T04:54:15.604 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.604 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.604 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_commit_rec", 2024-02-05T04:54:15.604 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.706170+0000", 2024-02-05T04:54:15.605 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0035117070000000002 2024-02-05T04:54:15.605 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.605 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.605 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:54:15.605 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.706196+0000", 2024-02-05T04:54:15.605 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.6129e-05 2024-02-05T04:54:15.605 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.605 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.605 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:15.605 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.706213+0000", 2024-02-05T04:54:15.605 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.7490999999999999e-05 2024-02-05T04:54:15.605 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.605 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:15.605 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.606 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.606 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.606 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4209.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-02-05T04:54:15.606 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:09.696618+0000", 2024-02-05T04:54:15.606 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 65.883421580999993, 2024-02-05T04:54:15.606 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0096608100000000006, 2024-02-05T04:54:15.606 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:15.606 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:15.606 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:54:15.606 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4209", 2024-02-05T04:54:15.606 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3672061555", 2024-02-05T04:54:15.606 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 88 2024-02-05T04:54:15.606 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.606 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:15.607 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.607 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:15.607 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.696618+0000", 2024-02-05T04:54:15.607 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:15.607 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.607 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.607 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:15.607 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.696618+0000", 2024-02-05T04:54:15.607 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:15.607 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.607 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.607 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:15.607 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.696618+0000", 2024-02-05T04:54:15.608 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.7099999999999998e-07 2024-02-05T04:54:15.608 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.608 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.608 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:15.608 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.696621+0000", 2024-02-05T04:54:15.608 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.401e-06 2024-02-05T04:54:15.608 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.608 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.608 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:15.608 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.696624+0000", 2024-02-05T04:54:15.608 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.4680000000000001e-06 2024-02-05T04:54:15.608 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.608 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.608 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:15.609 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.696627+0000", 2024-02-05T04:54:15.609 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.525e-06 2024-02-05T04:54:15.609 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.609 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.609 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:15.609 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.698695+0000", 2024-02-05T04:54:15.609 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0020686290000000002 2024-02-05T04:54:15.609 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.609 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.609 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:54:15.609 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.698736+0000", 2024-02-05T04:54:15.609 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.0555e-05 2024-02-05T04:54:15.609 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.610 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.610 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for subops from 2", 2024-02-05T04:54:15.610 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.698800+0000", 2024-02-05T04:54:15.610 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.4437000000000007e-05 2024-02-05T04:54:15.610 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.610 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.610 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "op_commit", 2024-02-05T04:54:15.610 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.702661+0000", 2024-02-05T04:54:15.610 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0038606640000000002 2024-02-05T04:54:15.610 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.610 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.610 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_commit_rec", 2024-02-05T04:54:15.610 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.706238+0000", 2024-02-05T04:54:15.611 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0035770250000000002 2024-02-05T04:54:15.611 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.611 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.611 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:54:15.611 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.706263+0000", 2024-02-05T04:54:15.611 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.4797999999999999e-05 2024-02-05T04:54:15.611 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.611 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.611 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:15.611 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.706279+0000", 2024-02-05T04:54:15.611 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.6036999999999999e-05 2024-02-05T04:54:15.611 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.611 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:15.611 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.612 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.612 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.612 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4209.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-02-05T04:54:15.612 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:09.850779+0000", 2024-02-05T04:54:15.612 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 65.729260664999998, 2024-02-05T04:54:15.612 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0095974279999999999, 2024-02-05T04:54:15.612 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:15.612 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:15.612 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:54:15.612 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4209", 2024-02-05T04:54:15.612 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3672061555", 2024-02-05T04:54:15.612 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 110 2024-02-05T04:54:15.612 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.613 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:15.613 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.613 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:15.613 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.850779+0000", 2024-02-05T04:54:15.613 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:15.613 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.613 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.613 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:15.613 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.850779+0000", 2024-02-05T04:54:15.613 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:15.613 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.613 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.613 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:15.613 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.850780+0000", 2024-02-05T04:54:15.614 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.5199999999999997e-07 2024-02-05T04:54:15.614 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.614 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.614 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:15.614 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.850784+0000", 2024-02-05T04:54:15.614 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.0099999999999997e-06 2024-02-05T04:54:15.614 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.614 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.614 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:15.614 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.850791+0000", 2024-02-05T04:54:15.614 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.2259999999999999e-06 2024-02-05T04:54:15.614 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.614 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.614 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:15.615 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.850796+0000", 2024-02-05T04:54:15.615 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.2089999999999996e-06 2024-02-05T04:54:15.615 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.615 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.615 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:15.615 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.853636+0000", 2024-02-05T04:54:15.615 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0028403199999999999 2024-02-05T04:54:15.615 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.615 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.615 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:54:15.615 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.853688+0000", 2024-02-05T04:54:15.615 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.1391000000000003e-05 2024-02-05T04:54:15.615 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.615 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.615 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for subops from 2", 2024-02-05T04:54:15.616 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.853766+0000", 2024-02-05T04:54:15.616 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.7875999999999998e-05 2024-02-05T04:54:15.616 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.616 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.616 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "op_commit", 2024-02-05T04:54:15.616 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.855430+0000", 2024-02-05T04:54:15.616 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.001664603 2024-02-05T04:54:15.616 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.616 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.616 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_commit_rec", 2024-02-05T04:54:15.616 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.860285+0000", 2024-02-05T04:54:15.616 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0048545230000000003 2024-02-05T04:54:15.616 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.616 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.617 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:54:15.617 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.860340+0000", 2024-02-05T04:54:15.617 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.5531000000000001e-05 2024-02-05T04:54:15.617 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.617 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.617 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:15.617 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.860376+0000", 2024-02-05T04:54:15.617 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.6186999999999998e-05 2024-02-05T04:54:15.617 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.617 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:15.617 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.617 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.617 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.617 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4209.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-02-05T04:54:15.618 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:09.850805+0000", 2024-02-05T04:54:15.618 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 65.729234855000001, 2024-02-05T04:54:15.618 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0096654600000000007, 2024-02-05T04:54:15.618 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:15.618 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:15.618 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:54:15.618 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4209", 2024-02-05T04:54:15.618 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3672061555", 2024-02-05T04:54:15.618 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 111 2024-02-05T04:54:15.618 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.618 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:15.618 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.618 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:15.618 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.850805+0000", 2024-02-05T04:54:15.619 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:15.619 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.619 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.619 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:15.619 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.850805+0000", 2024-02-05T04:54:15.619 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:15.619 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.619 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.619 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:15.619 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.850805+0000", 2024-02-05T04:54:15.619 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.3099999999999998e-07 2024-02-05T04:54:15.619 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.619 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.619 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:15.619 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.850809+0000", 2024-02-05T04:54:15.620 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.6370000000000001e-06 2024-02-05T04:54:15.620 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.620 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.620 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:15.620 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.850817+0000", 2024-02-05T04:54:15.620 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.3909999999999998e-06 2024-02-05T04:54:15.620 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.620 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.620 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:15.620 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.850831+0000", 2024-02-05T04:54:15.620 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.3942e-05 2024-02-05T04:54:15.620 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.620 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.620 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:15.620 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.854167+0000", 2024-02-05T04:54:15.621 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0033358530000000002 2024-02-05T04:54:15.621 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.621 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.621 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:54:15.621 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.854212+0000", 2024-02-05T04:54:15.621 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.5393999999999999e-05 2024-02-05T04:54:15.621 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.621 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.621 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for subops from 2", 2024-02-05T04:54:15.621 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.854290+0000", 2024-02-05T04:54:15.621 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.7189999999999998e-05 2024-02-05T04:54:15.621 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.621 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.621 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "op_commit", 2024-02-05T04:54:15.621 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.857996+0000", 2024-02-05T04:54:15.622 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0037059889999999998 2024-02-05T04:54:15.622 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.622 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.622 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_commit_rec", 2024-02-05T04:54:15.622 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.860419+0000", 2024-02-05T04:54:15.622 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0024236309999999999 2024-02-05T04:54:15.622 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.622 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.622 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:54:15.622 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.860451+0000", 2024-02-05T04:54:15.622 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.1433000000000002e-05 2024-02-05T04:54:15.622 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.622 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.622 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:15.623 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.860470+0000", 2024-02-05T04:54:15.623 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.9568999999999998e-05 2024-02-05T04:54:15.623 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.623 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:15.623 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.623 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.623 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.623 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4209.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-02-05T04:54:15.623 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:09.850841+0000", 2024-02-05T04:54:15.623 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 65.729198737999994, 2024-02-05T04:54:15.623 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0096977799999999996, 2024-02-05T04:54:15.623 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:15.623 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:15.623 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:54:15.624 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4209", 2024-02-05T04:54:15.624 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3672061555", 2024-02-05T04:54:15.624 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 112 2024-02-05T04:54:15.624 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.624 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:15.624 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.624 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:15.624 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.850841+0000", 2024-02-05T04:54:15.624 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:15.624 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.624 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.624 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:15.624 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.850841+0000", 2024-02-05T04:54:15.624 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:15.624 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.625 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.625 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:15.625 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.850841+0000", 2024-02-05T04:54:15.625 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.3000000000000001e-07 2024-02-05T04:54:15.625 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.625 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.625 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:15.625 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.850961+0000", 2024-02-05T04:54:15.625 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00011995 2024-02-05T04:54:15.625 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.625 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.625 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:15.625 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.850981+0000", 2024-02-05T04:54:15.625 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.9323000000000001e-05 2024-02-05T04:54:15.626 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.626 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.626 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:15.626 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.850987+0000", 2024-02-05T04:54:15.626 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.5729999999999996e-06 2024-02-05T04:54:15.626 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.626 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.626 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:15.626 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.854544+0000", 2024-02-05T04:54:15.626 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0035569809999999999 2024-02-05T04:54:15.626 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.626 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.626 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:54:15.626 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.854626+0000", 2024-02-05T04:54:15.626 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.1725999999999996e-05 2024-02-05T04:54:15.627 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.627 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.627 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for subops from 2", 2024-02-05T04:54:15.627 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.854754+0000", 2024-02-05T04:54:15.627 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00012792899999999999 2024-02-05T04:54:15.627 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.627 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.627 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "op_commit", 2024-02-05T04:54:15.627 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.857998+0000", 2024-02-05T04:54:15.627 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0032445070000000002 2024-02-05T04:54:15.627 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.627 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.627 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_commit_rec", 2024-02-05T04:54:15.627 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.860493+0000", 2024-02-05T04:54:15.628 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0024945700000000002 2024-02-05T04:54:15.628 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.628 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.628 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:54:15.628 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.860520+0000", 2024-02-05T04:54:15.628 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.7141999999999999e-05 2024-02-05T04:54:15.628 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.628 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.628 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:15.628 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.860539+0000", 2024-02-05T04:54:15.628 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.8649000000000001e-05 2024-02-05T04:54:15.628 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.628 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:15.628 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.629 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.629 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.629 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4209.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-02-05T04:54:15.629 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:09.850999+0000", 2024-02-05T04:54:15.629 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 65.729040491000006, 2024-02-05T04:54:15.629 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0095989739999999997, 2024-02-05T04:54:15.629 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:15.629 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:15.629 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:54:15.629 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4209", 2024-02-05T04:54:15.629 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3672061555", 2024-02-05T04:54:15.629 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 113 2024-02-05T04:54:15.629 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.629 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:15.629 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.630 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:15.630 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.850999+0000", 2024-02-05T04:54:15.630 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:15.630 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.630 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.630 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:15.630 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.850999+0000", 2024-02-05T04:54:15.630 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:15.630 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.630 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.630 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:15.630 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851000+0000", 2024-02-05T04:54:15.630 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.0900000000000002e-07 2024-02-05T04:54:15.630 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.631 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.631 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:15.631 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851004+0000", 2024-02-05T04:54:15.631 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.4669999999999998e-06 2024-02-05T04:54:15.631 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.631 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.631 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:15.631 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851014+0000", 2024-02-05T04:54:15.631 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.7540000000000001e-06 2024-02-05T04:54:15.631 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.631 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.631 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:15.631 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851019+0000", 2024-02-05T04:54:15.631 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.2610000000000001e-06 2024-02-05T04:54:15.631 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.632 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.632 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:15.632 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.855104+0000", 2024-02-05T04:54:15.632 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0040849609999999998 2024-02-05T04:54:15.632 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.632 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.632 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:54:15.632 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.855150+0000", 2024-02-05T04:54:15.632 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.6143999999999997e-05 2024-02-05T04:54:15.632 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.632 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.632 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for subops from 2", 2024-02-05T04:54:15.632 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.855224+0000", 2024-02-05T04:54:15.632 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.4024999999999997e-05 2024-02-05T04:54:15.633 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.633 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.633 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "op_commit", 2024-02-05T04:54:15.633 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.858002+0000", 2024-02-05T04:54:15.633 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.002777336 2024-02-05T04:54:15.633 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.633 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.633 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_commit_rec", 2024-02-05T04:54:15.633 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.860559+0000", 2024-02-05T04:54:15.633 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0025570910000000001 2024-02-05T04:54:15.633 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.633 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.633 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:54:15.633 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.860583+0000", 2024-02-05T04:54:15.633 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.4562999999999998e-05 2024-02-05T04:54:15.633 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.634 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.634 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:15.634 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.860598+0000", 2024-02-05T04:54:15.634 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.4863e-05 2024-02-05T04:54:15.634 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.634 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:15.634 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.634 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.634 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.634 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4209.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-02-05T04:54:15.634 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:09.851028+0000", 2024-02-05T04:54:15.634 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 65.729011307999997, 2024-02-05T04:54:15.634 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0097085690000000002, 2024-02-05T04:54:15.634 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:15.634 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:15.635 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:54:15.635 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4209", 2024-02-05T04:54:15.635 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3672061555", 2024-02-05T04:54:15.635 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 114 2024-02-05T04:54:15.635 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.635 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:15.635 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.635 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:15.635 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851028+0000", 2024-02-05T04:54:15.635 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:15.635 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.635 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.635 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:15.635 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851028+0000", 2024-02-05T04:54:15.635 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:15.636 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.636 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.636 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:15.636 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851029+0000", 2024-02-05T04:54:15.636 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.9599999999999999e-07 2024-02-05T04:54:15.636 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.636 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.636 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:15.636 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851033+0000", 2024-02-05T04:54:15.636 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.0169999999999998e-06 2024-02-05T04:54:15.636 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.636 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.636 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:15.636 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851044+0000", 2024-02-05T04:54:15.637 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1388e-05 2024-02-05T04:54:15.637 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.637 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.637 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:15.637 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851050+0000", 2024-02-05T04:54:15.637 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.7250000000000002e-06 2024-02-05T04:54:15.637 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.637 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.637 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:15.637 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.855468+0000", 2024-02-05T04:54:15.637 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0044182170000000003 2024-02-05T04:54:15.637 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.637 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.637 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:54:15.637 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.855549+0000", 2024-02-05T04:54:15.638 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.1248999999999994e-05 2024-02-05T04:54:15.638 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.638 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.638 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for subops from 2", 2024-02-05T04:54:15.638 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.855675+0000", 2024-02-05T04:54:15.638 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000126029 2024-02-05T04:54:15.638 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.638 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.638 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "op_commit", 2024-02-05T04:54:15.638 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.858005+0000", 2024-02-05T04:54:15.638 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0023293329999999998 2024-02-05T04:54:15.638 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.638 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.638 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_commit_rec", 2024-02-05T04:54:15.639 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.860645+0000", 2024-02-05T04:54:15.639 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0026406960000000001 2024-02-05T04:54:15.639 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.639 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.639 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:54:15.639 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.860701+0000", 2024-02-05T04:54:15.639 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.5492999999999999e-05 2024-02-05T04:54:15.639 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.639 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.639 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:15.639 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.860737+0000", 2024-02-05T04:54:15.639 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.5926000000000001e-05 2024-02-05T04:54:15.639 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.639 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:15.639 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.639 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.640 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.640 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4209.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-02-05T04:54:15.640 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:09.851060+0000", 2024-02-05T04:54:15.640 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 65.728979507999995, 2024-02-05T04:54:15.640 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.011939682, 2024-02-05T04:54:15.640 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:15.640 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:15.640 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:54:15.640 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4209", 2024-02-05T04:54:15.640 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3672061555", 2024-02-05T04:54:15.640 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 115 2024-02-05T04:54:15.640 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.640 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:15.641 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.641 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:15.641 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851060+0000", 2024-02-05T04:54:15.641 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:15.641 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.641 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.641 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:15.641 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851060+0000", 2024-02-05T04:54:15.641 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:15.641 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.641 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.641 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:15.641 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851061+0000", 2024-02-05T04:54:15.641 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.5900000000000002e-07 2024-02-05T04:54:15.641 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.642 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.642 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:15.642 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851142+0000", 2024-02-05T04:54:15.642 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.1631000000000005e-05 2024-02-05T04:54:15.642 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.642 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.642 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:15.642 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851150+0000", 2024-02-05T04:54:15.642 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.2679999999999994e-06 2024-02-05T04:54:15.642 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.642 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.642 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:15.642 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851156+0000", 2024-02-05T04:54:15.642 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.7200000000000003e-06 2024-02-05T04:54:15.643 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.643 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.643 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:15.643 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.856546+0000", 2024-02-05T04:54:15.643 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0053899839999999996 2024-02-05T04:54:15.643 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.643 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.643 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:54:15.643 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.856623+0000", 2024-02-05T04:54:15.643 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.6921000000000004e-05 2024-02-05T04:54:15.643 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.643 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.643 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for subops from 2", 2024-02-05T04:54:15.643 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.856748+0000", 2024-02-05T04:54:15.643 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00012522999999999999 2024-02-05T04:54:15.643 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.644 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.644 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "op_commit", 2024-02-05T04:54:15.644 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.858007+0000", 2024-02-05T04:54:15.644 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0012587779999999999 2024-02-05T04:54:15.644 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.644 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.644 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_commit_rec", 2024-02-05T04:54:15.644 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.862907+0000", 2024-02-05T04:54:15.644 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0048998729999999999 2024-02-05T04:54:15.644 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.644 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.644 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:54:15.644 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.862964+0000", 2024-02-05T04:54:15.644 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.7391e-05 2024-02-05T04:54:15.645 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.645 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.645 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:15.645 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.863000+0000", 2024-02-05T04:54:15.645 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.5426999999999998e-05 2024-02-05T04:54:15.645 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.645 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:15.645 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.645 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.645 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.645 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4209.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-02-05T04:54:15.645 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:09.851167+0000", 2024-02-05T04:54:15.645 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 65.728872663999994, 2024-02-05T04:54:15.646 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.011918911000000001, 2024-02-05T04:54:15.646 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:15.646 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:15.646 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:54:15.646 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4209", 2024-02-05T04:54:15.646 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3672061555", 2024-02-05T04:54:15.646 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 116 2024-02-05T04:54:15.646 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.646 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:15.646 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.646 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:15.646 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851167+0000", 2024-02-05T04:54:15.646 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:15.646 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.646 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.646 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:15.646 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851167+0000", 2024-02-05T04:54:15.647 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:15.647 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.647 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.647 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:15.647 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851167+0000", 2024-02-05T04:54:15.647 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.9999999999999998e-07 2024-02-05T04:54:15.647 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.647 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.647 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:15.647 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851172+0000", 2024-02-05T04:54:15.647 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.0400000000000003e-06 2024-02-05T04:54:15.647 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.647 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.647 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:15.647 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851179+0000", 2024-02-05T04:54:15.647 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.063e-06 2024-02-05T04:54:15.648 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.648 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.648 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:15.648 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851184+0000", 2024-02-05T04:54:15.648 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.1220000000000003e-06 2024-02-05T04:54:15.648 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.648 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.648 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:15.648 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.857205+0000", 2024-02-05T04:54:15.648 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0060208630000000004 2024-02-05T04:54:15.648 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.648 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.648 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:54:15.648 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.857278+0000", 2024-02-05T04:54:15.649 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.3089000000000004e-05 2024-02-05T04:54:15.649 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.649 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.649 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for subops from 2", 2024-02-05T04:54:15.649 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.857412+0000", 2024-02-05T04:54:15.649 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00013471800000000001 2024-02-05T04:54:15.649 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.649 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.649 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "op_commit", 2024-02-05T04:54:15.649 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.858009+0000", 2024-02-05T04:54:15.649 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00059697400000000001 2024-02-05T04:54:15.649 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.649 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.649 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_commit_rec", 2024-02-05T04:54:15.649 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.863035+0000", 2024-02-05T04:54:15.650 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0050251489999999996 2024-02-05T04:54:15.650 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.650 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.650 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:54:15.650 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.863066+0000", 2024-02-05T04:54:15.650 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.1680999999999997e-05 2024-02-05T04:54:15.650 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.650 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.650 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:15.650 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.863086+0000", 2024-02-05T04:54:15.650 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.9712e-05 2024-02-05T04:54:15.650 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.650 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:15.650 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.650 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.651 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.651 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4209.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-02-05T04:54:15.651 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:09.851192+0000", 2024-02-05T04:54:15.651 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 65.728847662000007, 2024-02-05T04:54:15.651 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.014624346999999999, 2024-02-05T04:54:15.651 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:15.651 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:15.651 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:54:15.651 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4209", 2024-02-05T04:54:15.651 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3672061555", 2024-02-05T04:54:15.651 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 117 2024-02-05T04:54:15.651 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.651 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:15.651 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.652 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:15.652 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851192+0000", 2024-02-05T04:54:15.652 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:15.652 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.652 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.652 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:15.652 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851192+0000", 2024-02-05T04:54:15.652 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:15.652 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.652 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.652 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:15.652 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851192+0000", 2024-02-05T04:54:15.652 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.5900000000000003e-07 2024-02-05T04:54:15.653 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.653 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.653 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:15.653 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851200+0000", 2024-02-05T04:54:15.653 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.2420000000000003e-06 2024-02-05T04:54:15.653 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.653 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.653 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:15.653 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851206+0000", 2024-02-05T04:54:15.653 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.2419999999999997e-06 2024-02-05T04:54:15.653 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.653 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.653 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:15.653 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.851211+0000", 2024-02-05T04:54:15.653 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.7339999999999999e-06 2024-02-05T04:54:15.654 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.654 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.654 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:15.654 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.857705+0000", 2024-02-05T04:54:15.654 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0064942890000000003 2024-02-05T04:54:15.654 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.654 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.654 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:54:15.654 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.857756+0000", 2024-02-05T04:54:15.654 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.1360999999999998e-05 2024-02-05T04:54:15.654 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.654 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.654 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for subops from 2", 2024-02-05T04:54:15.654 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.857841+0000", 2024-02-05T04:54:15.655 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.4610000000000005e-05 2024-02-05T04:54:15.655 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.655 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.655 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "op_commit", 2024-02-05T04:54:15.655 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.858257+0000", 2024-02-05T04:54:15.655 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00041574599999999999 2024-02-05T04:54:15.655 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.655 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.655 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_commit_rec", 2024-02-05T04:54:15.655 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.865722+0000", 2024-02-05T04:54:15.655 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.007465902 2024-02-05T04:54:15.655 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.655 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.655 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:54:15.656 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.865782+0000", 2024-02-05T04:54:15.656 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.9879e-05 2024-02-05T04:54:15.656 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.656 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.656 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:15.656 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.865816+0000", 2024-02-05T04:54:15.656 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.3983000000000003e-05 2024-02-05T04:54:15.656 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.656 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:15.656 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.656 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.656 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.656 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4209.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-02-05T04:54:15.656 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:09.993364+0000", 2024-02-05T04:54:15.657 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 65.586675560000003, 2024-02-05T04:54:15.657 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0097019250000000001, 2024-02-05T04:54:15.657 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:15.657 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:15.657 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:54:15.657 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4209", 2024-02-05T04:54:15.657 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3672061555", 2024-02-05T04:54:15.657 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 138 2024-02-05T04:54:15.657 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.657 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:15.657 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.657 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:15.657 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.993364+0000", 2024-02-05T04:54:15.657 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:15.657 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.658 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.658 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:15.658 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.993364+0000", 2024-02-05T04:54:15.658 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:15.658 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.658 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.658 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:15.658 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.993364+0000", 2024-02-05T04:54:15.658 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.8700000000000002e-07 2024-02-05T04:54:15.658 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.658 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.658 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:15.658 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.993368+0000", 2024-02-05T04:54:15.658 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.6720000000000002e-06 2024-02-05T04:54:15.658 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.659 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.659 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:15.659 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.993372+0000", 2024-02-05T04:54:15.659 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.625e-06 2024-02-05T04:54:15.659 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.659 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.659 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:15.659 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.993379+0000", 2024-02-05T04:54:15.659 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.1210000000000001e-06 2024-02-05T04:54:15.659 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.659 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.659 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:15.659 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.000600+0000", 2024-02-05T04:54:15.659 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0072211829999999999 2024-02-05T04:54:15.659 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.659 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.660 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:54:15.660 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.000672+0000", 2024-02-05T04:54:15.660 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.1967999999999999e-05 2024-02-05T04:54:15.660 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.660 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.660 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for subops from 2", 2024-02-05T04:54:15.660 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.000765+0000", 2024-02-05T04:54:15.660 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.3139000000000004e-05 2024-02-05T04:54:15.660 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.660 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.660 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "op_commit", 2024-02-05T04:54:15.660 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.002487+0000", 2024-02-05T04:54:15.660 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.001721783 2024-02-05T04:54:15.660 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.661 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.661 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_commit_rec", 2024-02-05T04:54:15.661 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.003017+0000", 2024-02-05T04:54:15.661 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00053045200000000005 2024-02-05T04:54:15.661 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.661 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.661 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:54:15.661 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.003047+0000", 2024-02-05T04:54:15.661 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.9229000000000001e-05 2024-02-05T04:54:15.661 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.661 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.661 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:15.661 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.003066+0000", 2024-02-05T04:54:15.661 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.9466e-05 2024-02-05T04:54:15.662 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.662 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:15.662 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.662 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.662 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.662 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4209.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-02-05T04:54:15.662 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:09.993384+0000", 2024-02-05T04:54:15.662 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 65.586655798999999, 2024-02-05T04:54:15.662 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.013170685999999999, 2024-02-05T04:54:15.662 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:15.662 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:15.662 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:54:15.662 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4209", 2024-02-05T04:54:15.662 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3672061555", 2024-02-05T04:54:15.662 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 139 2024-02-05T04:54:15.663 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.663 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:15.663 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.663 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:15.663 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.993384+0000", 2024-02-05T04:54:15.663 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:15.663 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.663 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.663 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:15.663 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.993384+0000", 2024-02-05T04:54:15.663 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:15.663 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.663 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.663 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:15.663 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.993384+0000", 2024-02-05T04:54:15.664 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.1400000000000001e-07 2024-02-05T04:54:15.664 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.664 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.664 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:15.664 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.993386+0000", 2024-02-05T04:54:15.664 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.9829999999999999e-06 2024-02-05T04:54:15.664 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.664 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.664 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:15.664 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.993390+0000", 2024-02-05T04:54:15.664 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.4209999999999997e-06 2024-02-05T04:54:15.664 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.664 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.664 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:15.665 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.993393+0000", 2024-02-05T04:54:15.665 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.508e-06 2024-02-05T04:54:15.665 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.665 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.665 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:15.665 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.001088+0000", 2024-02-05T04:54:15.665 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0076952230000000002 2024-02-05T04:54:15.665 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.665 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.665 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:54:15.665 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.001134+0000", 2024-02-05T04:54:15.665 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.5467999999999999e-05 2024-02-05T04:54:15.665 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.666 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.666 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for subops from 2", 2024-02-05T04:54:15.666 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.001203+0000", 2024-02-05T04:54:15.666 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.9509000000000001e-05 2024-02-05T04:54:15.666 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.666 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.666 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "op_commit", 2024-02-05T04:54:15.666 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.002497+0000", 2024-02-05T04:54:15.666 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0012934000000000001 2024-02-05T04:54:15.666 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.666 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.666 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_commit_rec", 2024-02-05T04:54:15.666 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.006502+0000", 2024-02-05T04:54:15.666 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0040053980000000003 2024-02-05T04:54:15.666 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.667 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.667 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:54:15.667 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.006534+0000", 2024-02-05T04:54:15.667 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.2351999999999998e-05 2024-02-05T04:54:15.667 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.667 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.667 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:15.667 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.006555+0000", 2024-02-05T04:54:15.667 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.0210000000000001e-05 2024-02-05T04:54:15.667 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.667 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:15.667 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.667 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.667 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.667 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4209.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-02-05T04:54:15.668 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:09.993397+0000", 2024-02-05T04:54:15.668 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 65.586642552000001, 2024-02-05T04:54:15.668 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.013240236000000001, 2024-02-05T04:54:15.668 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:15.668 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:15.668 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:54:15.668 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4209", 2024-02-05T04:54:15.668 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3672061555", 2024-02-05T04:54:15.668 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 140 2024-02-05T04:54:15.668 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.668 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:15.668 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.668 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:15.669 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.993397+0000", 2024-02-05T04:54:15.669 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:15.669 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.669 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.669 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:15.669 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.993397+0000", 2024-02-05T04:54:15.669 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:15.669 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.669 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.669 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:15.669 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.993397+0000", 2024-02-05T04:54:15.669 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.2700000000000001e-07 2024-02-05T04:54:15.669 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.669 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.670 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:15.670 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.993449+0000", 2024-02-05T04:54:15.670 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.1613000000000002e-05 2024-02-05T04:54:15.670 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.670 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.670 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:15.670 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.993453+0000", 2024-02-05T04:54:15.670 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.245e-06 2024-02-05T04:54:15.670 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.670 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.670 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:15.670 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.993456+0000", 2024-02-05T04:54:15.670 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.942e-06 2024-02-05T04:54:15.670 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.670 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.671 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:15.671 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.001438+0000", 2024-02-05T04:54:15.671 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0079815589999999992 2024-02-05T04:54:15.671 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.671 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.671 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:54:15.671 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.001514+0000", 2024-02-05T04:54:15.671 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.6038999999999995e-05 2024-02-05T04:54:15.671 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.671 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.671 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for subops from 2", 2024-02-05T04:54:15.671 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.001642+0000", 2024-02-05T04:54:15.671 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000128713 2024-02-05T04:54:15.671 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.671 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.672 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "op_commit", 2024-02-05T04:54:15.672 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.002506+0000", 2024-02-05T04:54:15.672 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00086372900000000004 2024-02-05T04:54:15.672 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.672 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.672 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_commit_rec", 2024-02-05T04:54:15.672 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.006589+0000", 2024-02-05T04:54:15.672 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0040831690000000002 2024-02-05T04:54:15.672 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.672 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.672 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:54:15.672 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.006620+0000", 2024-02-05T04:54:15.672 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.0817e-05 2024-02-05T04:54:15.672 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.672 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.673 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:15.673 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.006637+0000", 2024-02-05T04:54:15.673 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.7183000000000001e-05 2024-02-05T04:54:15.673 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.673 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:15.673 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.673 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.673 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.673 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4209.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-02-05T04:54:15.673 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:09.993462+0000", 2024-02-05T04:54:15.673 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 65.586577762999994, 2024-02-05T04:54:15.673 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.013244182, 2024-02-05T04:54:15.673 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:15.673 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:15.674 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:54:15.674 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4209", 2024-02-05T04:54:15.674 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3672061555", 2024-02-05T04:54:15.674 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 141 2024-02-05T04:54:15.674 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.674 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:15.674 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.674 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:15.674 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.993462+0000", 2024-02-05T04:54:15.674 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:15.674 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.674 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.674 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:15.674 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.993462+0000", 2024-02-05T04:54:15.675 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:15.675 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.675 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.675 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:15.675 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.993462+0000", 2024-02-05T04:54:15.675 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.6100000000000002e-07 2024-02-05T04:54:15.675 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.675 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.675 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:15.675 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.993464+0000", 2024-02-05T04:54:15.675 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.0480000000000001e-06 2024-02-05T04:54:15.675 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.675 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.675 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:15.675 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.993467+0000", 2024-02-05T04:54:15.675 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.1480000000000002e-06 2024-02-05T04:54:15.676 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.676 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.676 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:15.676 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:09.993470+0000", 2024-02-05T04:54:15.676 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.5320000000000001e-06 2024-02-05T04:54:15.676 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.676 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.676 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:15.676 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.002184+0000", 2024-02-05T04:54:15.676 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0087141649999999994 2024-02-05T04:54:15.676 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.676 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.676 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:54:15.676 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.002227+0000", 2024-02-05T04:54:15.676 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.3272000000000002e-05 2024-02-05T04:54:15.677 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.677 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.677 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for subops from 2", 2024-02-05T04:54:15.677 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.002296+0000", 2024-02-05T04:54:15.677 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.8268000000000004e-05 2024-02-05T04:54:15.677 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.677 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.677 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "op_commit", 2024-02-05T04:54:15.677 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.002752+0000", 2024-02-05T04:54:15.677 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00045654199999999999 2024-02-05T04:54:15.677 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.677 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.677 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_commit_rec", 2024-02-05T04:54:15.677 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.006663+0000", 2024-02-05T04:54:15.677 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.003911126 2024-02-05T04:54:15.678 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.678 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.678 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:54:15.678 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.006689+0000", 2024-02-05T04:54:15.678 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.6061999999999999e-05 2024-02-05T04:54:15.678 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.678 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.678 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:15.678 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:10.006706+0000", 2024-02-05T04:54:15.678 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.6758000000000001e-05 2024-02-05T04:54:15.678 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.678 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:15.678 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.678 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.679 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.679 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.es2 38/31 ECSubWrite(tid=25, reqid=osd.2.0:25, at_version=38'20, trim_to=0'0, roll_forward_to=37'16))", 2024-02-05T04:54:15.679 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:30.134965+0000", 2024-02-05T04:54:15.679 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 45.445074290000001, 2024-02-05T04:54:15.679 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.010779733, 2024-02-05T04:54:15.679 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:15.679 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:54:15.679 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:15.679 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.679 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:15.679 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.134965+0000", 2024-02-05T04:54:15.679 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:15.679 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.679 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.680 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:15.680 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.134965+0000", 2024-02-05T04:54:15.680 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:15.680 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.680 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.680 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:15.680 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.134966+0000", 2024-02-05T04:54:15.680 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.4399999999999999e-07 2024-02-05T04:54:15.680 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.680 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.680 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:15.680 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.134990+0000", 2024-02-05T04:54:15.680 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.3895000000000001e-05 2024-02-05T04:54:15.680 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.681 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.681 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:15.681 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.135002+0000", 2024-02-05T04:54:15.681 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1868000000000001e-05 2024-02-05T04:54:15.681 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.681 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.681 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:15.681 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.135011+0000", 2024-02-05T04:54:15.681 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.8920000000000006e-06 2024-02-05T04:54:15.681 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.681 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.681 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:15.681 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.135354+0000", 2024-02-05T04:54:15.681 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00034297400000000001 2024-02-05T04:54:15.682 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.682 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.682 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:15.682 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.135364+0000", 2024-02-05T04:54:15.682 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0205e-05 2024-02-05T04:54:15.682 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.682 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.682 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:15.682 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.145707+0000", 2024-02-05T04:54:15.682 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.010343250999999999 2024-02-05T04:54:15.682 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.682 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.682 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:15.682 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.145745+0000", 2024-02-05T04:54:15.682 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.7904000000000002e-05 2024-02-05T04:54:15.683 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.683 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:15.683 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.683 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.683 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.683 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.2s5 38/31 ECSubWrite(tid=28, reqid=osd.12.0:28, at_version=38'22, trim_to=0'0, roll_forward_to=37'18))", 2024-02-05T04:54:15.683 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:30.144335+0000", 2024-02-05T04:54:15.683 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 45.435705096, 2024-02-05T04:54:15.683 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0094635199999999996, 2024-02-05T04:54:15.683 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:15.683 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:54:15.683 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:15.683 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.683 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:15.683 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.144335+0000", 2024-02-05T04:54:15.683 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:15.684 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.684 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.684 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:15.684 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.144335+0000", 2024-02-05T04:54:15.684 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:15.684 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.684 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.684 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:15.684 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.144335+0000", 2024-02-05T04:54:15.684 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.5600000000000004e-07 2024-02-05T04:54:15.684 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.684 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.684 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:15.684 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.144373+0000", 2024-02-05T04:54:15.684 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.7731000000000002e-05 2024-02-05T04:54:15.685 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.685 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.685 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:15.685 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.144387+0000", 2024-02-05T04:54:15.685 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.3788000000000001e-05 2024-02-05T04:54:15.685 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.685 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.685 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:15.685 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.144394+0000", 2024-02-05T04:54:15.685 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.2450000000000001e-06 2024-02-05T04:54:15.685 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.685 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.685 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:15.685 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.146480+0000", 2024-02-05T04:54:15.685 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0020859120000000001 2024-02-05T04:54:15.686 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.686 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.686 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:15.686 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.146504+0000", 2024-02-05T04:54:15.686 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.4091999999999999e-05 2024-02-05T04:54:15.686 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.686 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.686 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:15.686 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.153756+0000", 2024-02-05T04:54:15.686 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0072520470000000002 2024-02-05T04:54:15.686 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.686 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.686 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:15.686 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.153798+0000", 2024-02-05T04:54:15.686 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.1848999999999999e-05 2024-02-05T04:54:15.686 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.687 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:15.687 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.687 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.687 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.687 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.4s3 43/31 ECSubWrite(tid=46, reqid=osd.1.0:46, at_version=43'29, trim_to=0'0, roll_forward_to=42'27))", 2024-02-05T04:54:15.687 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:35.182639+0000", 2024-02-05T04:54:15.687 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 40.397400388999998, 2024-02-05T04:54:15.687 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.010458696999999999, 2024-02-05T04:54:15.687 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:15.687 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:54:15.687 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:15.687 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.687 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:15.687 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.182639+0000", 2024-02-05T04:54:15.687 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:15.687 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.688 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.688 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:15.688 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.182639+0000", 2024-02-05T04:54:15.688 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:15.688 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.688 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.688 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:15.688 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.182639+0000", 2024-02-05T04:54:15.688 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.0599999999999999e-07 2024-02-05T04:54:15.688 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.688 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.688 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:15.688 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.182658+0000", 2024-02-05T04:54:15.688 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.8646000000000001e-05 2024-02-05T04:54:15.688 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.689 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.689 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:15.689 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.182670+0000", 2024-02-05T04:54:15.689 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1629e-05 2024-02-05T04:54:15.689 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.689 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.689 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:15.689 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.182678+0000", 2024-02-05T04:54:15.689 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.7740000000000004e-06 2024-02-05T04:54:15.689 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.689 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.689 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:15.689 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.182738+0000", 2024-02-05T04:54:15.689 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.0123e-05 2024-02-05T04:54:15.690 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.690 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.690 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:15.690 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.182748+0000", 2024-02-05T04:54:15.690 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0078000000000001e-05 2024-02-05T04:54:15.690 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.690 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.690 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:15.690 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.193064+0000", 2024-02-05T04:54:15.690 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.010316561 2024-02-05T04:54:15.690 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:15.690 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:15.690 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:15.690 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.193098+0000", 2024-02-05T04:54:15.690 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.3680000000000003e-05 2024-02-05T04:54:15.691 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.691 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:15.691 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.691 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:15.691 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:15.691 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-02-05T04:54:15.693 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-02-05T04:54:15.693 INFO:tasks.ceph.osd.3.smithi196.stderr:2024-02-05T04:54:15.692+0000 7fc8817c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34605) UID: 0 2024-02-05T04:54:15.793 INFO:tasks.ceph.osd.9.smithi091.stderr:2024-02-05T04:54:15.792+0000 7efc18827640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 34279) UID: 0 2024-02-05T04:54:15.801 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-02-05T04:54:15.801 INFO:teuthology.orchestra.run.smithi062.stdout: "ops": [], 2024-02-05T04:54:15.801 INFO:teuthology.orchestra.run.smithi062.stdout: "num_ops": 0 2024-02-05T04:54:15.801 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-02-05T04:54:15.810 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-02-05T04:54:15.823 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-13 --journal-path /var/lib/ceph/osd/ceph-13/journal --op export --pgid 3.7s0 --file /var/log/ceph/exp.3.7s0.13 2024-02-05T04:54:15.894 INFO:tasks.ceph.osd.11.smithi196.stderr:2024-02-05T04:54:15.894+0000 7f85a8d86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 34609) UID: 0 2024-02-05T04:54:15.916 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-02-05T04:54:15.917 INFO:teuthology.orchestra.run.smithi062.stdout: "ops": [], 2024-02-05T04:54:15.917 INFO:teuthology.orchestra.run.smithi062.stdout: "complaint_time": 30, 2024-02-05T04:54:15.917 INFO:teuthology.orchestra.run.smithi062.stdout: "num_blocked_ops": 0 2024-02-05T04:54:15.917 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-02-05T04:54:15.925 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-02-05T04:54:15.995 INFO:tasks.ceph.osd.4.smithi062.stderr:2024-02-05T04:54:15.993+0000 7f708c308640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34689) UID: 0 2024-02-05T04:54:16.035 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-02-05T04:54:16.036 INFO:teuthology.orchestra.run.smithi062.stdout: "size": 20, 2024-02-05T04:54:16.036 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 600, 2024-02-05T04:54:16.036 INFO:teuthology.orchestra.run.smithi062.stdout: "ops": [ 2024-02-05T04:54:16.036 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.036 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:251 3.2s0 3:47e81a58:::smithi19639362-13:head [rollback 3] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:54:16.036 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:29.199025+0000", 2024-02-05T04:54:16.036 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 46.834816848000003, 2024-02-05T04:54:16.036 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.010965871, 2024-02-05T04:54:16.036 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:16.036 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:16.036 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:54:16.036 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:54:16.036 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:16.037 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 251 2024-02-05T04:54:16.037 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.037 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:16.037 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.037 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:16.037 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.199025+0000", 2024-02-05T04:54:16.037 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:16.037 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.037 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.037 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:16.037 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.199025+0000", 2024-02-05T04:54:16.037 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:16.037 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.037 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.037 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:16.038 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.199025+0000", 2024-02-05T04:54:16.038 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.6700000000000003e-07 2024-02-05T04:54:16.038 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.038 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.038 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:16.038 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.199030+0000", 2024-02-05T04:54:16.038 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.9620000000000001e-06 2024-02-05T04:54:16.038 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.038 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.038 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:16.038 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.199039+0000", 2024-02-05T04:54:16.038 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.5660000000000007e-06 2024-02-05T04:54:16.038 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.038 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.038 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:16.038 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.199045+0000", 2024-02-05T04:54:16.039 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.9229999999999999e-06 2024-02-05T04:54:16.039 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.039 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.039 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:16.039 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.200375+0000", 2024-02-05T04:54:16.039 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0013305610000000001 2024-02-05T04:54:16.039 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.039 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.039 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:54:16.039 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.200435+0000", 2024-02-05T04:54:16.039 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.9790999999999997e-05 2024-02-05T04:54:16.039 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.039 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.039 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:16.040 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.200748+0000", 2024-02-05T04:54:16.040 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00031337299999999998 2024-02-05T04:54:16.040 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.040 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.040 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:16.040 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.203751+0000", 2024-02-05T04:54:16.040 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.003003019 2024-02-05T04:54:16.040 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.040 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.040 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:54:16.040 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.209915+0000", 2024-02-05T04:54:16.040 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0061635099999999996 2024-02-05T04:54:16.040 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.040 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.041 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:16.041 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.209990+0000", 2024-02-05T04:54:16.041 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.5599000000000006e-05 2024-02-05T04:54:16.041 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:16.041 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:16.041 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:16.041 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.041 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.041 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:252 3.2s0 3:47e81a58:::smithi19639362-13:head [append 0~0] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:54:16.041 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:29.199055+0000", 2024-02-05T04:54:16.041 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 46.834786686999998, 2024-02-05T04:54:16.041 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.011150785, 2024-02-05T04:54:16.041 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:16.041 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:16.042 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:54:16.042 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:54:16.042 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:16.042 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 252 2024-02-05T04:54:16.042 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.042 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:16.042 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.042 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:16.042 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.199055+0000", 2024-02-05T04:54:16.042 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:16.042 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.042 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.042 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:16.042 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.199055+0000", 2024-02-05T04:54:16.043 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:16.043 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.043 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.043 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:16.043 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.199055+0000", 2024-02-05T04:54:16.043 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.3500000000000002e-07 2024-02-05T04:54:16.043 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.043 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.043 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:16.043 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.199058+0000", 2024-02-05T04:54:16.043 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.779e-06 2024-02-05T04:54:16.043 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.043 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.043 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:16.043 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.199065+0000", 2024-02-05T04:54:16.044 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.5139999999999998e-06 2024-02-05T04:54:16.044 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.044 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.044 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:16.044 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.199069+0000", 2024-02-05T04:54:16.044 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.5820000000000003e-06 2024-02-05T04:54:16.044 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.044 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.044 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:16.044 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.201028+0000", 2024-02-05T04:54:16.044 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.001959199 2024-02-05T04:54:16.044 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.044 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.044 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:54:16.045 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.201092+0000", 2024-02-05T04:54:16.045 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.3375999999999998e-05 2024-02-05T04:54:16.045 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.045 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.045 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:16.045 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.201409+0000", 2024-02-05T04:54:16.045 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00031745599999999998 2024-02-05T04:54:16.045 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.045 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.045 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:16.045 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.203757+0000", 2024-02-05T04:54:16.045 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0023475219999999999 2024-02-05T04:54:16.045 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.045 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.045 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:54:16.046 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.210134+0000", 2024-02-05T04:54:16.046 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0063771130000000002 2024-02-05T04:54:16.046 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.046 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.046 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:16.046 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.210206+0000", 2024-02-05T04:54:16.046 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.1809000000000003e-05 2024-02-05T04:54:16.046 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:16.046 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:16.046 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:16.046 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.046 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.046 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:253 3.2s0 3:41121c54:::smithi19639362-23:head [append 0~688128] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:54:16.046 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:29.205018+0000", 2024-02-05T04:54:16.047 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 46.828823802999999, 2024-02-05T04:54:16.047 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0073893079999999998, 2024-02-05T04:54:16.047 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:16.047 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:16.047 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:54:16.047 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:54:16.047 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:16.047 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 253 2024-02-05T04:54:16.047 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.047 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:16.047 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.047 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:16.047 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.205018+0000", 2024-02-05T04:54:16.047 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:16.047 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.047 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.048 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:16.048 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.205018+0000", 2024-02-05T04:54:16.048 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:16.048 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.048 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.048 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:16.048 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.205019+0000", 2024-02-05T04:54:16.048 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.282e-06 2024-02-05T04:54:16.048 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.048 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.048 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:16.048 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.205752+0000", 2024-02-05T04:54:16.048 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00073318000000000005 2024-02-05T04:54:16.048 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.049 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.049 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:16.049 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.205764+0000", 2024-02-05T04:54:16.049 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1936000000000001e-05 2024-02-05T04:54:16.049 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.049 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.049 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:16.049 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.205769+0000", 2024-02-05T04:54:16.049 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.2979999999999999e-06 2024-02-05T04:54:16.049 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.049 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.049 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:16.049 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.205811+0000", 2024-02-05T04:54:16.049 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.1891000000000003e-05 2024-02-05T04:54:16.050 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.050 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.050 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:54:16.050 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.205885+0000", 2024-02-05T04:54:16.050 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.3384000000000001e-05 2024-02-05T04:54:16.050 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.050 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.050 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:16.050 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.206604+0000", 2024-02-05T04:54:16.050 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000719343 2024-02-05T04:54:16.050 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.050 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.050 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:16.050 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.210004+0000", 2024-02-05T04:54:16.050 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0034002799999999999 2024-02-05T04:54:16.051 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.051 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.051 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:54:16.051 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.212332+0000", 2024-02-05T04:54:16.051 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0023273209999999998 2024-02-05T04:54:16.051 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.051 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.051 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:16.051 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.212407+0000", 2024-02-05T04:54:16.051 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.5393000000000001e-05 2024-02-05T04:54:16.051 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:16.051 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:16.051 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:16.051 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.051 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.052 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:254 3.2s0 3:41121c54:::smithi19639362-23:head [append 0~376832] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:54:16.052 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:29.207235+0000", 2024-02-05T04:54:16.052 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 46.826606763000001, 2024-02-05T04:54:16.052 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0064199799999999996, 2024-02-05T04:54:16.052 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:16.052 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:16.052 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:54:16.052 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:54:16.052 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:16.052 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 254 2024-02-05T04:54:16.052 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.052 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:16.052 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.052 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:16.053 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.207235+0000", 2024-02-05T04:54:16.053 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:16.053 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.053 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.053 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:16.053 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.207235+0000", 2024-02-05T04:54:16.053 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:16.053 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.053 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.053 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:16.053 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.207236+0000", 2024-02-05T04:54:16.053 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.37e-06 2024-02-05T04:54:16.053 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.054 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.054 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:16.054 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.207708+0000", 2024-02-05T04:54:16.054 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00047221200000000001 2024-02-05T04:54:16.054 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.054 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.054 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:16.054 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.207721+0000", 2024-02-05T04:54:16.054 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.2446000000000001e-05 2024-02-05T04:54:16.054 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.054 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.054 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:16.054 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.207730+0000", 2024-02-05T04:54:16.055 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.4870000000000008e-06 2024-02-05T04:54:16.055 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.055 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.055 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:16.055 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.207796+0000", 2024-02-05T04:54:16.055 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.5852999999999994e-05 2024-02-05T04:54:16.055 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.055 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.055 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:54:16.055 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.207899+0000", 2024-02-05T04:54:16.055 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00010255900000000001 2024-02-05T04:54:16.055 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.055 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.055 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:16.056 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.208541+0000", 2024-02-05T04:54:16.056 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00064210899999999997 2024-02-05T04:54:16.056 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.056 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.056 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:16.056 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.212149+0000", 2024-02-05T04:54:16.056 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0036078170000000001 2024-02-05T04:54:16.056 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.056 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.056 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:54:16.056 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.213599+0000", 2024-02-05T04:54:16.056 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.001450935 2024-02-05T04:54:16.056 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.057 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.057 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:16.057 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.213655+0000", 2024-02-05T04:54:16.057 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.5192000000000003e-05 2024-02-05T04:54:16.057 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:16.057 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:16.057 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:16.057 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.057 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.057 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:256 3.2s0 3:41121c54:::smithi19639362-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:54:16.057 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:29.207798+0000", 2024-02-05T04:54:16.057 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 46.826043374999998, 2024-02-05T04:54:16.057 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0068739830000000002, 2024-02-05T04:54:16.057 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:16.058 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "started", 2024-02-05T04:54:16.058 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:54:16.058 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:54:16.058 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:16.058 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 256 2024-02-05T04:54:16.058 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.058 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:16.058 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.058 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:16.058 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.207798+0000", 2024-02-05T04:54:16.058 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:16.058 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.058 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.058 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:16.058 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.207798+0000", 2024-02-05T04:54:16.059 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:16.059 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.059 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.059 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:16.059 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.207799+0000", 2024-02-05T04:54:16.059 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.5899999999999996e-07 2024-02-05T04:54:16.059 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.059 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.059 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:16.059 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.207808+0000", 2024-02-05T04:54:16.059 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.7720000000000006e-06 2024-02-05T04:54:16.059 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.059 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.059 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:16.059 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.207818+0000", 2024-02-05T04:54:16.059 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.1050000000000001e-06 2024-02-05T04:54:16.060 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.060 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.060 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:16.060 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.207823+0000", 2024-02-05T04:54:16.060 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.8409999999999996e-06 2024-02-05T04:54:16.060 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.060 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.060 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:16.060 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.210496+0000", 2024-02-05T04:54:16.060 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0026729879999999998 2024-02-05T04:54:16.060 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.060 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.060 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for rw locks", 2024-02-05T04:54:16.060 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.210593+0000", 2024-02-05T04:54:16.061 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.6873999999999994e-05 2024-02-05T04:54:16.061 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.061 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.061 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:16.061 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.213830+0000", 2024-02-05T04:54:16.061 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0032368560000000002 2024-02-05T04:54:16.061 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.061 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.061 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:54:16.061 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.213901+0000", 2024-02-05T04:54:16.061 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.0813000000000001e-05 2024-02-05T04:54:16.061 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.061 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.061 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:16.061 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.214672+0000", 2024-02-05T04:54:16.062 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00077117500000000003 2024-02-05T04:54:16.062 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:16.062 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:16.062 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:16.062 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.062 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.062 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:257 3.2s0 3:4f006f6c:::smithi19639362-3:head [stat,delete] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:54:16.062 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:29.207834+0000", 2024-02-05T04:54:16.062 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 46.826007336000004, 2024-02-05T04:54:16.062 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0085459170000000001, 2024-02-05T04:54:16.062 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:16.062 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:16.062 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:54:16.062 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:54:16.063 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:16.063 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 257 2024-02-05T04:54:16.063 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.063 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:16.063 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.063 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:16.063 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.207834+0000", 2024-02-05T04:54:16.063 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:16.063 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.063 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.063 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:16.063 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.207834+0000", 2024-02-05T04:54:16.063 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:16.063 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.064 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.064 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:16.064 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.207835+0000", 2024-02-05T04:54:16.064 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.13e-07 2024-02-05T04:54:16.064 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.064 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.064 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:16.064 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.207838+0000", 2024-02-05T04:54:16.064 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.96e-06 2024-02-05T04:54:16.064 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.064 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.064 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:16.064 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.207845+0000", 2024-02-05T04:54:16.064 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.6699999999999994e-06 2024-02-05T04:54:16.065 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.065 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.065 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:16.065 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.207851+0000", 2024-02-05T04:54:16.065 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.896e-06 2024-02-05T04:54:16.065 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.065 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.065 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:16.065 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.210632+0000", 2024-02-05T04:54:16.065 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00278087 2024-02-05T04:54:16.065 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.065 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.065 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:54:16.065 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.210723+0000", 2024-02-05T04:54:16.065 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.1265000000000001e-05 2024-02-05T04:54:16.066 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.066 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.066 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:16.066 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.211433+0000", 2024-02-05T04:54:16.066 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00070996899999999996 2024-02-05T04:54:16.066 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.066 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.066 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:16.066 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.214530+0000", 2024-02-05T04:54:16.066 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.003096634 2024-02-05T04:54:16.066 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.066 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.066 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:54:16.066 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.216308+0000", 2024-02-05T04:54:16.066 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00177809 2024-02-05T04:54:16.066 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.066 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.067 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:16.067 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.216380+0000", 2024-02-05T04:54:16.067 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.1950000000000001e-05 2024-02-05T04:54:16.067 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:16.067 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:16.067 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:16.067 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.067 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.067 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:281 3.2s0 3:41121c54:::smithi19639362-23:head [read 0~1441792,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-02-05T04:54:16.067 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:30.195307+0000", 2024-02-05T04:54:16.067 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 45.838534170000003, 2024-02-05T04:54:16.067 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0063372669999999997, 2024-02-05T04:54:16.067 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:16.067 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "started", 2024-02-05T04:54:16.068 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:54:16.068 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:54:16.068 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:16.068 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 281 2024-02-05T04:54:16.068 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.068 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:16.068 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.068 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:16.068 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.195307+0000", 2024-02-05T04:54:16.068 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:16.068 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.068 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.068 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:16.068 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.195307+0000", 2024-02-05T04:54:16.069 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:16.069 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.069 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.069 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:16.069 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.195308+0000", 2024-02-05T04:54:16.069 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.41e-07 2024-02-05T04:54:16.069 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.069 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.069 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:16.069 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.195309+0000", 2024-02-05T04:54:16.069 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.5659999999999999e-06 2024-02-05T04:54:16.069 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.069 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.069 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:16.070 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.195312+0000", 2024-02-05T04:54:16.070 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.242e-06 2024-02-05T04:54:16.070 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.070 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.070 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:16.070 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.195316+0000", 2024-02-05T04:54:16.070 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.2559999999999998e-06 2024-02-05T04:54:16.070 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.070 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.070 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:16.070 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.195897+0000", 2024-02-05T04:54:16.070 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00058133200000000005 2024-02-05T04:54:16.070 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.070 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.070 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:54:16.071 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.195943+0000", 2024-02-05T04:54:16.071 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.6452999999999997e-05 2024-02-05T04:54:16.071 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.071 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.071 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:16.071 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:30.201645+0000", 2024-02-05T04:54:16.071 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.005701177 2024-02-05T04:54:16.071 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:16.071 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:16.071 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:16.071 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.071 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.071 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(osd.7.23:2 3.2s0 3:41121c54:::smithi19639362-23:head [assert-version v29,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-02-05T04:54:16.071 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:49.168864+0000", 2024-02-05T04:54:16.071 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 26.864977230000001, 2024-02-05T04:54:16.071 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.79151548400000005, 2024-02-05T04:54:16.071 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:16.072 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "started", 2024-02-05T04:54:16.072 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:16.072 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.072 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:16.072 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.168864+0000", 2024-02-05T04:54:16.072 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:16.072 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.072 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.072 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:16.072 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.168864+0000", 2024-02-05T04:54:16.072 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:16.072 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.072 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.072 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:16.073 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.168866+0000", 2024-02-05T04:54:16.073 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.6899999999999999e-06 2024-02-05T04:54:16.073 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.073 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.073 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:16.073 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.168872+0000", 2024-02-05T04:54:16.073 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.2720000000000003e-06 2024-02-05T04:54:16.073 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.073 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.073 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:16.073 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.168883+0000", 2024-02-05T04:54:16.073 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1294000000000001e-05 2024-02-05T04:54:16.073 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.073 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.073 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:16.073 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.168894+0000", 2024-02-05T04:54:16.074 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.047e-05 2024-02-05T04:54:16.074 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.074 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.074 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:16.074 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.168957+0000", 2024-02-05T04:54:16.074 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.3275000000000007e-05 2024-02-05T04:54:16.074 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.074 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.074 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for peered", 2024-02-05T04:54:16.074 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.168961+0000", 2024-02-05T04:54:16.074 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.5499999999999999e-06 2024-02-05T04:54:16.074 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.074 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.074 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:16.075 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.955268+0000", 2024-02-05T04:54:16.075 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.78630735900000004 2024-02-05T04:54:16.075 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.075 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.075 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:54:16.075 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.955409+0000", 2024-02-05T04:54:16.076 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00014131199999999999 2024-02-05T04:54:16.076 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.076 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.076 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:16.076 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.960380+0000", 2024-02-05T04:54:16.076 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0049702619999999996 2024-02-05T04:54:16.076 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:16.076 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:16.076 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:16.076 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.077 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.077 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.14s4 68/67 ECSubWrite(tid=15, reqid=osd.3.0:15, at_version=68'75, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:16.077 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:59.964017+0000", 2024-02-05T04:54:16.077 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 16.069824520000001, 2024-02-05T04:54:16.077 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0071728579999999998, 2024-02-05T04:54:16.077 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:16.077 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:54:16.077 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:16.077 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.077 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:16.077 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.964017+0000", 2024-02-05T04:54:16.077 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:16.077 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.077 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.077 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:16.078 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.964017+0000", 2024-02-05T04:54:16.078 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:16.078 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.078 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.078 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:16.078 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.964018+0000", 2024-02-05T04:54:16.078 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.023e-06 2024-02-05T04:54:16.078 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.078 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.078 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:16.078 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.964043+0000", 2024-02-05T04:54:16.078 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.544e-05 2024-02-05T04:54:16.078 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.078 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.079 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:16.079 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.964067+0000", 2024-02-05T04:54:16.079 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.3210999999999999e-05 2024-02-05T04:54:16.079 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.079 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.079 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:16.079 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.964077+0000", 2024-02-05T04:54:16.079 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0635999999999999e-05 2024-02-05T04:54:16.079 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.079 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.079 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:16.079 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.964155+0000", 2024-02-05T04:54:16.079 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.7333e-05 2024-02-05T04:54:16.079 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.080 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.080 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:16.080 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.964165+0000", 2024-02-05T04:54:16.080 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0176e-05 2024-02-05T04:54:16.080 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.080 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.080 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:16.080 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.971152+0000", 2024-02-05T04:54:16.080 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0069875609999999998 2024-02-05T04:54:16.080 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.080 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.080 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:16.080 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.971190+0000", 2024-02-05T04:54:16.080 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.7478000000000003e-05 2024-02-05T04:54:16.081 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:16.081 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:16.081 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:16.081 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.081 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.081 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.14s4 68/67 ECSubWrite(tid=16, reqid=osd.3.0:16, at_version=68'77, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:16.081 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:59.964125+0000", 2024-02-05T04:54:16.081 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 16.069716183000001, 2024-02-05T04:54:16.081 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0076732850000000002, 2024-02-05T04:54:16.081 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:16.081 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:54:16.081 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:16.081 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.081 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:16.081 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.964125+0000", 2024-02-05T04:54:16.081 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:16.082 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.082 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.082 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:16.082 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.964125+0000", 2024-02-05T04:54:16.082 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:16.082 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.082 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.082 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:16.082 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.964126+0000", 2024-02-05T04:54:16.082 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.9599999999999999e-07 2024-02-05T04:54:16.082 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.082 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.082 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:16.082 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.964155+0000", 2024-02-05T04:54:16.082 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.9456999999999999e-05 2024-02-05T04:54:16.082 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.083 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.083 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:16.083 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.964165+0000", 2024-02-05T04:54:16.083 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.6390000000000004e-06 2024-02-05T04:54:16.083 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.083 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.083 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:16.083 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.964173+0000", 2024-02-05T04:54:16.083 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.6930000000000007e-06 2024-02-05T04:54:16.083 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.083 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.083 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:16.083 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.964579+0000", 2024-02-05T04:54:16.083 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00040651800000000003 2024-02-05T04:54:16.084 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.084 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.084 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:16.085 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.964591+0000", 2024-02-05T04:54:16.085 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.2155e-05 2024-02-05T04:54:16.085 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.085 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.085 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:16.085 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.971767+0000", 2024-02-05T04:54:16.085 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0071754180000000002 2024-02-05T04:54:16.085 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.085 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.085 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:16.085 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.971799+0000", 2024-02-05T04:54:16.085 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.1909000000000002e-05 2024-02-05T04:54:16.085 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:16.085 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:16.085 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:16.086 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.086 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.086 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.14s4 68/67 ECSubWrite(tid=17, reqid=osd.3.0:16, at_version=0'0, trim_to=0'0, roll_forward_to=68'77))", 2024-02-05T04:54:16.086 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:59.973935+0000", 2024-02-05T04:54:16.086 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 16.059906018, 2024-02-05T04:54:16.086 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0082042380000000008, 2024-02-05T04:54:16.086 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:16.086 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:54:16.086 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:16.086 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.086 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:16.086 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.973935+0000", 2024-02-05T04:54:16.086 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:16.086 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.086 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.087 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:16.087 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.973935+0000", 2024-02-05T04:54:16.087 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:16.087 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.087 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.087 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:16.087 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.973936+0000", 2024-02-05T04:54:16.087 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.9400000000000004e-07 2024-02-05T04:54:16.087 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.087 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.087 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:16.087 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.973947+0000", 2024-02-05T04:54:16.087 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0438999999999999e-05 2024-02-05T04:54:16.087 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.087 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.088 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:16.088 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.973956+0000", 2024-02-05T04:54:16.088 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.6639999999999993e-06 2024-02-05T04:54:16.088 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.088 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.088 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:16.088 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.973965+0000", 2024-02-05T04:54:16.088 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.8279999999999992e-06 2024-02-05T04:54:16.088 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.088 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.088 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:16.088 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.974014+0000", 2024-02-05T04:54:16.088 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.901e-05 2024-02-05T04:54:16.089 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.089 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.089 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:16.089 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.974026+0000", 2024-02-05T04:54:16.089 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1606999999999999e-05 2024-02-05T04:54:16.089 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.089 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.089 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:16.089 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.982102+0000", 2024-02-05T04:54:16.089 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0080761210000000003 2024-02-05T04:54:16.089 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.089 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.089 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:16.089 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.982140+0000", 2024-02-05T04:54:16.089 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.7675000000000001e-05 2024-02-05T04:54:16.090 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:16.090 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:16.090 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:16.090 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.090 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.090 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.bs6 69/57 ECSubWrite(tid=297, reqid=osd.13.0:294, at_version=0'0, trim_to=0'0, roll_forward_to=69'66))", 2024-02-05T04:54:16.090 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:54:00.961526+0000", 2024-02-05T04:54:16.090 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 15.072315897999999, 2024-02-05T04:54:16.090 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0075936379999999998, 2024-02-05T04:54:16.090 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:16.090 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:54:16.090 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:16.090 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.090 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:16.090 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.961526+0000", 2024-02-05T04:54:16.091 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:16.091 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.091 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.091 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:16.091 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.961526+0000", 2024-02-05T04:54:16.091 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:16.091 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.091 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.091 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:16.091 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.961526+0000", 2024-02-05T04:54:16.091 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.5699999999999998e-07 2024-02-05T04:54:16.091 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.091 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.091 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:16.091 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.961536+0000", 2024-02-05T04:54:16.092 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.6390000000000004e-06 2024-02-05T04:54:16.092 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.092 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.092 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:16.092 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.961546+0000", 2024-02-05T04:54:16.092 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.8300000000000008e-06 2024-02-05T04:54:16.092 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.092 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.092 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:16.092 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.961555+0000", 2024-02-05T04:54:16.092 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.1409999999999994e-06 2024-02-05T04:54:16.092 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.092 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.092 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:16.093 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.961817+0000", 2024-02-05T04:54:16.093 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00026277899999999999 2024-02-05T04:54:16.093 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.093 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.093 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:16.093 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.961830+0000", 2024-02-05T04:54:16.093 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.2798e-05 2024-02-05T04:54:16.093 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.093 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.093 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:16.093 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.969072+0000", 2024-02-05T04:54:16.093 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0072421960000000002 2024-02-05T04:54:16.093 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.093 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.093 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:16.094 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.969119+0000", 2024-02-05T04:54:16.094 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.6798000000000002e-05 2024-02-05T04:54:16.094 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:16.094 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:16.094 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:16.094 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.094 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.094 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:641 3.2s0 3:41121c54:::smithi19639362-23:head [copy-from ver 29] snapc 16=[16,15,12,11,f,b,8,7] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-02-05T04:54:16.094 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:54:00.998481+0000", 2024-02-05T04:54:16.094 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 15.035360449000001, 2024-02-05T04:54:16.094 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.024160300999999999, 2024-02-05T04:54:16.094 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:16.094 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:16.094 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:54:16.094 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:54:16.095 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:16.095 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 641 2024-02-05T04:54:16.095 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.095 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:16.095 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.095 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:16.095 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.998481+0000", 2024-02-05T04:54:16.095 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:16.095 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.095 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.095 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:16.095 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.998481+0000", 2024-02-05T04:54:16.095 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:16.096 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.096 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.096 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:16.096 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.998483+0000", 2024-02-05T04:54:16.096 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.728e-06 2024-02-05T04:54:16.096 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.096 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.096 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:16.096 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.998488+0000", 2024-02-05T04:54:16.096 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.4519999999999996e-06 2024-02-05T04:54:16.096 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.096 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.096 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:16.097 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.998501+0000", 2024-02-05T04:54:16.097 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.2867000000000001e-05 2024-02-05T04:54:16.097 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.097 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.097 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:16.097 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.998507+0000", 2024-02-05T04:54:16.097 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.3929999999999996e-06 2024-02-05T04:54:16.097 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.097 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.097 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:16.097 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.998570+0000", 2024-02-05T04:54:16.097 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.2575000000000003e-05 2024-02-05T04:54:16.097 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.097 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.097 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:54:16.098 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.998672+0000", 2024-02-05T04:54:16.098 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000102298 2024-02-05T04:54:16.098 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.098 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.098 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:16.098 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:01.014182+0000", 2024-02-05T04:54:16.098 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.015509573 2024-02-05T04:54:16.098 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.098 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.098 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:16.098 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:01.016579+0000", 2024-02-05T04:54:16.098 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0023966489999999998 2024-02-05T04:54:16.098 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.098 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.099 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:54:16.099 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:01.022562+0000", 2024-02-05T04:54:16.099 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0059837889999999998 2024-02-05T04:54:16.099 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.099 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.099 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:16.099 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:01.022641+0000", 2024-02-05T04:54:16.099 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.8977e-05 2024-02-05T04:54:16.099 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:16.099 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:16.099 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:16.099 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.099 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.099 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:642 3.2s0 3:41121c54:::smithi19639362-23:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e69)", 2024-02-05T04:54:16.099 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:54:00.998532+0000", 2024-02-05T04:54:16.100 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 15.035309659999999, 2024-02-05T04:54:16.100 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.017716162000000001, 2024-02-05T04:54:16.100 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:16.100 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "started", 2024-02-05T04:54:16.100 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:54:16.100 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:54:16.100 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:16.100 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 642 2024-02-05T04:54:16.100 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.100 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:16.100 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.100 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:16.100 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.998532+0000", 2024-02-05T04:54:16.101 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:16.101 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.101 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.101 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:16.101 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.998532+0000", 2024-02-05T04:54:16.101 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:16.101 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.101 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.101 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:16.101 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.998532+0000", 2024-02-05T04:54:16.101 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.39e-07 2024-02-05T04:54:16.101 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.101 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.101 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:16.102 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.998534+0000", 2024-02-05T04:54:16.102 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.17e-06 2024-02-05T04:54:16.102 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.102 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.102 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:16.102 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.998541+0000", 2024-02-05T04:54:16.102 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.4869999999999999e-06 2024-02-05T04:54:16.102 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.102 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.102 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:16.102 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.998545+0000", 2024-02-05T04:54:16.102 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.3449999999999999e-06 2024-02-05T04:54:16.102 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.102 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.102 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:16.103 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.998806+0000", 2024-02-05T04:54:16.103 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00026117600000000001 2024-02-05T04:54:16.103 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.103 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.103 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for blocked object", 2024-02-05T04:54:16.103 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.998867+0000", 2024-02-05T04:54:16.103 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.0637000000000002e-05 2024-02-05T04:54:16.103 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.103 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.103 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:16.103 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:01.016028+0000", 2024-02-05T04:54:16.103 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.017161201000000001 2024-02-05T04:54:16.103 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.103 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.103 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:54:16.103 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:01.016138+0000", 2024-02-05T04:54:16.104 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000109942 2024-02-05T04:54:16.104 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.104 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.104 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:16.104 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:01.016248+0000", 2024-02-05T04:54:16.104 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000109765 2024-02-05T04:54:16.104 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:16.104 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:16.104 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:16.104 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.104 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.104 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:701 3.10s0 3:0bdcbf39:::smithi19639362-7:head [append 0~573440] snapc 18=[18,17,16,12,8,7] ondisk+write+known_if_redirected+supports_pool_eio e75)", 2024-02-05T04:54:16.104 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:54:07.046260+0000", 2024-02-05T04:54:16.104 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 8.9875816159999999, 2024-02-05T04:54:16.105 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.010674739000000001, 2024-02-05T04:54:16.105 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:16.105 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:16.105 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:54:16.105 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:54:16.105 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:16.105 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 701 2024-02-05T04:54:16.105 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.105 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:16.105 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.105 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:16.105 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:07.046260+0000", 2024-02-05T04:54:16.105 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:16.105 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.106 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.106 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:16.106 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:07.046260+0000", 2024-02-05T04:54:16.106 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:16.106 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.106 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.106 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:16.106 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:07.046261+0000", 2024-02-05T04:54:16.106 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0589999999999999e-06 2024-02-05T04:54:16.106 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.106 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.106 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:16.106 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:07.046964+0000", 2024-02-05T04:54:16.106 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00070279800000000003 2024-02-05T04:54:16.106 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.107 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.107 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:16.107 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:07.046975+0000", 2024-02-05T04:54:16.107 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1666e-05 2024-02-05T04:54:16.107 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.107 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.107 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:16.107 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:07.046980+0000", 2024-02-05T04:54:16.107 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.9060000000000001e-06 2024-02-05T04:54:16.107 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.107 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.107 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:16.107 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:07.047022+0000", 2024-02-05T04:54:16.107 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.1863999999999998e-05 2024-02-05T04:54:16.107 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.107 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.108 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:54:16.108 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:07.047127+0000", 2024-02-05T04:54:16.108 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00010534 2024-02-05T04:54:16.108 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.108 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.108 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:16.108 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:07.047937+0000", 2024-02-05T04:54:16.108 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00080946399999999997 2024-02-05T04:54:16.108 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.108 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.108 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:16.108 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:07.052998+0000", 2024-02-05T04:54:16.108 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0050611279999999998 2024-02-05T04:54:16.108 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.109 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.109 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:54:16.109 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:07.056859+0000", 2024-02-05T04:54:16.109 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0038610699999999999 2024-02-05T04:54:16.109 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.109 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.109 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:16.109 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:07.056935+0000", 2024-02-05T04:54:16.109 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.5444000000000005e-05 2024-02-05T04:54:16.109 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:16.109 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:16.109 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:16.109 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.109 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.110 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:702 3.10s0 3:0bdcbf39:::smithi19639362-7:head [append 0~622592] snapc 18=[18,17,16,12,8,7] ondisk+write+known_if_redirected+supports_pool_eio e75)", 2024-02-05T04:54:16.110 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:54:07.050619+0000", 2024-02-05T04:54:16.110 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 8.9832227400000004, 2024-02-05T04:54:16.110 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.011124143, 2024-02-05T04:54:16.110 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:16.110 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:16.110 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:54:16.110 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:54:16.110 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:16.110 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 702 2024-02-05T04:54:16.110 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.110 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:16.110 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.110 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:16.110 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:07.050619+0000", 2024-02-05T04:54:16.111 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:16.111 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.111 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.111 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:16.111 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:07.050619+0000", 2024-02-05T04:54:16.111 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:16.111 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.111 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.111 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:16.111 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:07.050621+0000", 2024-02-05T04:54:16.111 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.8190000000000001e-06 2024-02-05T04:54:16.111 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.111 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.111 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:16.112 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:07.051303+0000", 2024-02-05T04:54:16.112 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000682531 2024-02-05T04:54:16.112 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.112 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.112 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:16.112 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:07.051315+0000", 2024-02-05T04:54:16.112 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1426e-05 2024-02-05T04:54:16.112 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.112 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.112 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:16.112 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:07.051320+0000", 2024-02-05T04:54:16.112 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.6779999999999997e-06 2024-02-05T04:54:16.112 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.112 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.113 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:16.113 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:07.051383+0000", 2024-02-05T04:54:16.113 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.2984000000000005e-05 2024-02-05T04:54:16.113 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.113 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.113 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:54:16.113 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:07.051502+0000", 2024-02-05T04:54:16.113 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00011898 2024-02-05T04:54:16.113 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.113 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.113 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:16.113 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:07.052330+0000", 2024-02-05T04:54:16.113 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000827672 2024-02-05T04:54:16.113 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.114 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.114 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:16.114 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:07.053496+0000", 2024-02-05T04:54:16.114 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.001166545 2024-02-05T04:54:16.114 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.114 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.114 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:54:16.114 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:07.061669+0000", 2024-02-05T04:54:16.114 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0081722630000000008 2024-02-05T04:54:16.114 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.114 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.114 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:16.114 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:07.061743+0000", 2024-02-05T04:54:16.114 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.4245000000000005e-05 2024-02-05T04:54:16.114 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:16.115 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:16.115 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:16.115 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.115 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.115 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:703 3.10s0 3:0bdcbf39:::smithi19639362-7:head [append 0~622592] snapc 18=[18,17,16,12,8,7] ondisk+write+known_if_redirected+supports_pool_eio e75)", 2024-02-05T04:54:16.115 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:54:07.054594+0000", 2024-02-05T04:54:16.115 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 8.9792475009999997, 2024-02-05T04:54:16.115 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.010248314999999999, 2024-02-05T04:54:16.115 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:16.115 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:16.115 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:54:16.115 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:54:16.115 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:16.115 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 703 2024-02-05T04:54:16.115 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.116 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:16.116 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.116 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:16.116 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:07.054594+0000", 2024-02-05T04:54:16.116 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:16.116 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.116 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.116 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:16.116 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:07.054594+0000", 2024-02-05T04:54:16.116 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:16.116 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.116 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.116 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:16.116 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:07.054596+0000", 2024-02-05T04:54:16.117 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.824e-06 2024-02-05T04:54:16.117 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.117 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.117 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:16.117 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:07.055314+0000", 2024-02-05T04:54:16.117 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00071812099999999995 2024-02-05T04:54:16.117 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.117 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.117 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:16.117 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:07.055325+0000", 2024-02-05T04:54:16.117 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0798000000000001e-05 2024-02-05T04:54:16.117 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.117 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.117 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:16.117 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:07.055332+0000", 2024-02-05T04:54:16.118 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.2099999999999996e-06 2024-02-05T04:54:16.118 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.118 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.118 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:16.118 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:07.055381+0000", 2024-02-05T04:54:16.118 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.8946999999999997e-05 2024-02-05T04:54:16.118 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.118 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.118 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:54:16.118 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:07.055486+0000", 2024-02-05T04:54:16.118 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00010498899999999999 2024-02-05T04:54:16.118 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.118 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.118 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:16.118 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:07.056303+0000", 2024-02-05T04:54:16.118 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00081756500000000002 2024-02-05T04:54:16.119 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.119 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.119 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:16.119 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:07.059758+0000", 2024-02-05T04:54:16.119 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0034542380000000001 2024-02-05T04:54:16.119 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.119 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.119 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:54:16.119 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:07.064797+0000", 2024-02-05T04:54:16.119 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0050393019999999998 2024-02-05T04:54:16.119 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.119 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.119 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:16.119 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:07.064842+0000", 2024-02-05T04:54:16.119 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.5321000000000002e-05 2024-02-05T04:54:16.120 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:16.120 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:16.120 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:16.120 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.120 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.120 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:704 3.10s0 3:0bdcbf39:::smithi19639362-7:head [append 0~311296] snapc 18=[18,17,16,12,8,7] ondisk+write+known_if_redirected+supports_pool_eio e75)", 2024-02-05T04:54:16.120 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:54:07.056750+0000", 2024-02-05T04:54:16.120 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 8.9770917309999998, 2024-02-05T04:54:16.120 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.019945488000000001, 2024-02-05T04:54:16.120 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:16.120 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:16.120 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:54:16.120 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:54:16.120 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:16.121 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 704 2024-02-05T04:54:16.121 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.121 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:16.121 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.121 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:16.121 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:07.056750+0000", 2024-02-05T04:54:16.121 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:16.121 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.121 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.121 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:16.121 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:07.056750+0000", 2024-02-05T04:54:16.121 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:16.121 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.121 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.122 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:16.122 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:07.056751+0000", 2024-02-05T04:54:16.122 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.3659999999999999e-06 2024-02-05T04:54:16.122 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.122 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.122 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:16.122 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:07.057080+0000", 2024-02-05T04:54:16.122 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00032860499999999999 2024-02-05T04:54:16.122 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.122 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.122 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:16.122 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:07.057091+0000", 2024-02-05T04:54:16.122 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1751e-05 2024-02-05T04:54:16.122 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.122 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.123 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:16.123 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:07.057106+0000", 2024-02-05T04:54:16.123 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.4450999999999999e-05 2024-02-05T04:54:16.123 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.123 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.123 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:16.123 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:07.057260+0000", 2024-02-05T04:54:16.123 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00015431 2024-02-05T04:54:16.123 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.123 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.123 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:54:16.123 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:07.057370+0000", 2024-02-05T04:54:16.123 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000110079 2024-02-05T04:54:16.123 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.123 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.124 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:16.124 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:07.058139+0000", 2024-02-05T04:54:16.124 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00076891899999999996 2024-02-05T04:54:16.124 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.124 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.124 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:16.124 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:07.061235+0000", 2024-02-05T04:54:16.124 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0030956009999999999 2024-02-05T04:54:16.124 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.124 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.124 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:54:16.124 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:07.076610+0000", 2024-02-05T04:54:16.124 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.015375381 2024-02-05T04:54:16.125 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.125 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.125 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:16.125 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:07.076695+0000", 2024-02-05T04:54:16.125 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.5025000000000007e-05 2024-02-05T04:54:16.125 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:16.125 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:16.125 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:16.125 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.125 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.125 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:705 3.10s0 3:0bdcbf39:::smithi19639362-7:head [setxattr _header (56) in=7b] snapc 18=[18,17,16,12,8,7] ondisk+write+known_if_redirected+supports_pool_eio e75)", 2024-02-05T04:54:16.125 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:54:07.057135+0000", 2024-02-05T04:54:16.125 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 8.9767065979999998, 2024-02-05T04:54:16.126 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.019877990000000002, 2024-02-05T04:54:16.126 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:16.126 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:16.126 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:54:16.126 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:54:16.126 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:16.126 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 705 2024-02-05T04:54:16.126 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.126 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:16.126 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.126 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:16.126 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:07.057135+0000", 2024-02-05T04:54:16.126 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:16.126 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.127 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.127 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:16.127 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:07.057135+0000", 2024-02-05T04:54:16.127 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:16.127 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.127 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.127 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:16.127 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:07.057136+0000", 2024-02-05T04:54:16.127 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.6899999999999996e-07 2024-02-05T04:54:16.127 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.127 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.127 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:16.127 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:07.057139+0000", 2024-02-05T04:54:16.127 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.3699999999999999e-06 2024-02-05T04:54:16.127 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.128 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.128 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:16.128 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:07.057147+0000", 2024-02-05T04:54:16.128 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.7419999999999997e-06 2024-02-05T04:54:16.128 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.128 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.128 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:16.128 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:07.057153+0000", 2024-02-05T04:54:16.128 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.0589999999999999e-06 2024-02-05T04:54:16.128 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.128 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.128 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:16.128 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:07.058609+0000", 2024-02-05T04:54:16.128 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.001456496 2024-02-05T04:54:16.129 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.129 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.129 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:54:16.129 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:07.058701+0000", 2024-02-05T04:54:16.129 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.1671999999999999e-05 2024-02-05T04:54:16.129 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.129 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.129 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:16.129 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:07.059133+0000", 2024-02-05T04:54:16.129 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000432342 2024-02-05T04:54:16.129 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.129 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.129 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:16.129 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:07.061237+0000", 2024-02-05T04:54:16.129 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.002103887 2024-02-05T04:54:16.129 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.130 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.130 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:54:16.130 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:07.076910+0000", 2024-02-05T04:54:16.130 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.015672537 2024-02-05T04:54:16.130 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.130 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.130 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:16.130 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:07.077013+0000", 2024-02-05T04:54:16.130 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00010311599999999999 2024-02-05T04:54:16.130 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:16.130 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:16.130 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:16.130 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.130 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.131 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:706 3.10s0 3:0bdcbf39:::smithi19639362-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e75)", 2024-02-05T04:54:16.131 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:54:07.057162+0000", 2024-02-05T04:54:16.131 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 8.9766790850000007, 2024-02-05T04:54:16.131 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.027181864, 2024-02-05T04:54:16.131 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:16.131 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "started", 2024-02-05T04:54:16.131 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:54:16.131 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:54:16.131 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:16.131 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 706 2024-02-05T04:54:16.131 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.131 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:16.131 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.131 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:16.131 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:07.057162+0000", 2024-02-05T04:54:16.132 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:16.132 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.132 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.132 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:16.132 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:07.057162+0000", 2024-02-05T04:54:16.132 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:16.132 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.132 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.132 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:16.132 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:07.057163+0000", 2024-02-05T04:54:16.132 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.0800000000000005e-07 2024-02-05T04:54:16.132 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.132 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.132 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:16.133 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:07.057175+0000", 2024-02-05T04:54:16.133 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.2565000000000001e-05 2024-02-05T04:54:16.133 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.133 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.133 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:16.133 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:07.057183+0000", 2024-02-05T04:54:16.133 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.7770000000000002e-06 2024-02-05T04:54:16.133 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.133 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.133 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:16.133 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:07.057189+0000", 2024-02-05T04:54:16.133 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.558e-06 2024-02-05T04:54:16.133 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.133 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.133 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:16.134 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:07.059664+0000", 2024-02-05T04:54:16.134 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0024755430000000002 2024-02-05T04:54:16.134 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.134 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.134 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for rw locks", 2024-02-05T04:54:16.134 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:07.059750+0000", 2024-02-05T04:54:16.134 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.5378999999999997e-05 2024-02-05T04:54:16.134 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.134 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.134 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:16.134 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:07.077425+0000", 2024-02-05T04:54:16.134 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.017675462999999999 2024-02-05T04:54:16.134 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.134 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.134 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:54:16.135 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:07.077520+0000", 2024-02-05T04:54:16.135 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.4522000000000001e-05 2024-02-05T04:54:16.135 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:16.135 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:16.135 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:16.135 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:07.084344+0000", 2024-02-05T04:54:16.135 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0068245490000000001 2024-02-05T04:54:16.135 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:16.135 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:16.135 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:16.135 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:16.135 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:16.135 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-02-05T04:54:16.137 INFO:tasks.ceph.osd.1.smithi091.stderr:2024-02-05T04:54:16.135+0000 7ff9dc283640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34275) UID: 0 2024-02-05T04:54:16.138 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-02-05T04:54:16.140 INFO:tasks.rados.rados.0.smithi196.stdout:338: done (3 left) 2024-02-05T04:54:16.140 INFO:tasks.rados.rados.0.smithi196.stdout:339: delete oid 49 current snap is 28 2024-02-05T04:54:16.140 INFO:tasks.rados.rados.0.smithi196.stdout:332: finishing rollback tid 1 to smithi19639362-38 2024-02-05T04:54:16.140 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 38 v 21 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-02-05T04:54:16.140 INFO:tasks.rados.rados.0.smithi196.stdout:335: finishing copy_from to smithi19639362-47 2024-02-05T04:54:16.140 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 47 v 6 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-02-05T04:54:16.140 INFO:tasks.rados.rados.0.smithi196.stdout:335: finishing copy_from racing read to smithi19639362-47 2024-02-05T04:54:16.140 INFO:tasks.rados.rados.0.smithi196.stdout:332: done (3 left) 2024-02-05T04:54:16.140 INFO:tasks.rados.rados.0.smithi196.stdout:335: done (2 left) 2024-02-05T04:54:16.141 INFO:tasks.rados.rados.0.smithi196.stdout:339: done (1 left) 2024-02-05T04:54:16.141 INFO:tasks.rados.rados.0.smithi196.stdout:340: copy_from oid 48 from oid 16 current snap is 28 2024-02-05T04:54:16.141 INFO:tasks.rados.rados.0.smithi196.stdout:341: read oid 6 snap -1 2024-02-05T04:54:16.141 INFO:tasks.rados.rados.0.smithi196.stdout:341: expect deleted 2024-02-05T04:54:16.141 INFO:tasks.rados.rados.0.smithi196.stdout:342: rollback oid 40 current snap is 28 2024-02-05T04:54:16.141 INFO:tasks.rados.rados.0.smithi196.stdout:rollback oid 40 to 27 2024-02-05T04:54:16.141 INFO:tasks.rados.rados.0.smithi196.stdout:append oid 13 current snap is 28 2024-02-05T04:54:16.141 INFO:tasks.rados.rados.0.smithi196.stdout:343: seq_num 143 ranges {1277952=458752,1736704=475136,2211840=655360,2867200=671744,3538944=147456} 2024-02-05T04:54:16.141 INFO:tasks.rados.rados.0.smithi196.stdout:343: writing smithi19639362-13 from 1277952 to 1736704 tid 1 2024-02-05T04:54:16.141 INFO:tasks.rados.rados.0.smithi196.stdout:343: writing smithi19639362-13 from 1736704 to 2211840 tid 2 2024-02-05T04:54:16.141 INFO:tasks.rados.rados.0.smithi196.stdout:343: writing smithi19639362-13 from 2211840 to 2867200 tid 3 2024-02-05T04:54:16.141 INFO:tasks.rados.rados.0.smithi196.stdout:343: writing smithi19639362-13 from 2867200 to 3538944 tid 4 2024-02-05T04:54:16.141 INFO:tasks.rados.rados.0.smithi196.stdout:343: writing smithi19639362-13 from 3538944 to 3686400 tid 5 2024-02-05T04:54:16.141 INFO:tasks.rados.rados.0.smithi196.stdout:344: delete oid 17 current snap is 28 2024-02-05T04:54:16.141 INFO:tasks.rados.rados.0.smithi196.stdout:342: finishing rollback tid 1 to smithi19639362-40 2024-02-05T04:54:16.142 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 40 v 56 (ObjNum 129 snap 0 seq_num 3191332608) dirty dne 2024-02-05T04:54:16.142 INFO:tasks.rados.rados.0.smithi196.stdout:340: finishing copy_from racing read to smithi19639362-48 2024-02-05T04:54:16.142 INFO:tasks.rados.rados.0.smithi196.stdout:336: finishing copy_from racing read to smithi19639362-39 2024-02-05T04:54:16.142 INFO:tasks.rados.rados.0.smithi196.stdout:343: finishing write tid 1 to smithi19639362-13 2024-02-05T04:54:16.142 INFO:tasks.rados.rados.0.smithi196.stdout:343: finishing write tid 2 to smithi19639362-13 2024-02-05T04:54:16.142 INFO:tasks.rados.rados.0.smithi196.stdout:340: finishing copy_from to smithi19639362-48 2024-02-05T04:54:16.142 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 48 v 50 (ObjNum 60 snap 2 seq_num 60) dirty exists 2024-02-05T04:54:16.142 INFO:tasks.rados.rados.0.smithi196.stdout:340: done (5 left) 2024-02-05T04:54:16.142 INFO:tasks.rados.rados.0.smithi196.stdout:341: done (4 left) 2024-02-05T04:54:16.142 INFO:tasks.rados.rados.0.smithi196.stdout:342: done (3 left) 2024-02-05T04:54:16.142 INFO:tasks.rados.rados.0.smithi196.stdout:344: done (2 left) 2024-02-05T04:54:16.142 INFO:tasks.rados.rados.0.smithi196.stdout:append oid 33 current snap is 28 2024-02-05T04:54:16.142 INFO:tasks.rados.rados.0.smithi196.stdout:345: seq_num 145 ranges {2129920=32768} 2024-02-05T04:54:16.144 INFO:tasks.rados.rados.0.smithi196.stdout:345: writing smithi19639362-33 from 2129920 to 2162688 tid 1 2024-02-05T04:54:16.144 INFO:tasks.rados.rados.0.smithi196.stdout:346: rollback oid 44 current snap is 28 2024-02-05T04:54:16.144 INFO:tasks.rados.rados.0.smithi196.stdout:rollback oid 44 to 16 2024-02-05T04:54:16.144 INFO:tasks.rados.rados.0.smithi196.stdout:336: finishing copy_from to smithi19639362-39 2024-02-05T04:54:16.144 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 39 v 74 (ObjNum 78 snap 4 seq_num 78) dirty exists 2024-02-05T04:54:16.144 INFO:tasks.rados.rados.0.smithi196.stdout:336: done (3 left) 2024-02-05T04:54:16.144 INFO:tasks.rados.rados.0.smithi196.stdout:347: rollback oid 10 current snap is 28 2024-02-05T04:54:16.144 INFO:tasks.rados.rados.0.smithi196.stdout:rollback oid 10 to 23 2024-02-05T04:54:16.144 INFO:tasks.rados.rados.0.smithi196.stdout:348: delete oid 49 current snap is 28 2024-02-05T04:54:16.144 INFO:tasks.rados.rados.0.smithi196.stdout:343: finishing write tid 3 to smithi19639362-13 2024-02-05T04:54:16.144 INFO:tasks.rados.rados.0.smithi196.stdout:343: finishing write tid 4 to smithi19639362-13 2024-02-05T04:54:16.144 INFO:tasks.rados.rados.0.smithi196.stdout:343: finishing write tid 5 to smithi19639362-13 2024-02-05T04:54:16.145 INFO:tasks.rados.rados.0.smithi196.stdout:343: finishing write tid 6 to smithi19639362-13 2024-02-05T04:54:16.145 INFO:tasks.rados.rados.0.smithi196.stdout:343: finishing write tid 7 to smithi19639362-13 2024-02-05T04:54:16.145 INFO:tasks.rados.rados.0.smithi196.stdout:343: oid 13 updating version 0 to 83 2024-02-05T04:54:16.145 INFO:tasks.rados.rados.0.smithi196.stdout:343: oid 13 version 83 is already newer than 82 2024-02-05T04:54:16.145 INFO:tasks.rados.rados.0.smithi196.stdout:343: oid 13 updating version 83 to 85 2024-02-05T04:54:16.145 INFO:tasks.rados.rados.0.smithi196.stdout:343: oid 13 updating version 85 to 86 2024-02-05T04:54:16.145 INFO:tasks.rados.rados.0.smithi196.stdout:343: oid 13 version 86 is already newer than 84 2024-02-05T04:54:16.145 INFO:tasks.rados.rados.0.smithi196.stdout:343: oid 13 version 86 is already newer than 81 2024-02-05T04:54:16.145 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 13 v 86 (ObjNum 143 snap 28 seq_num 143) dirty exists 2024-02-05T04:54:16.145 INFO:tasks.rados.rados.0.smithi196.stdout:343: left oid 13 (ObjNum 143 snap 28 seq_num 143) 2024-02-05T04:54:16.146 INFO:tasks.rados.rados.0.smithi196.stdout:343: done (4 left) 2024-02-05T04:54:16.146 INFO:tasks.rados.rados.0.smithi196.stdout:348: done (3 left) 2024-02-05T04:54:16.146 INFO:tasks.rados.rados.0.smithi196.stdout:349: delete oid 42 current snap is 28 2024-02-05T04:54:16.148 INFO:tasks.rados.rados.0.smithi196.stdout:347: finishing rollback tid 0 to smithi19639362-10 2024-02-05T04:54:16.148 INFO:tasks.rados.rados.0.smithi196.stdout:346: finishing rollback tid 0 to smithi19639362-44 2024-02-05T04:54:16.150 INFO:tasks.rados.rados.0.smithi196.stdout:347: finishing rollback tid 1 to smithi19639362-10 2024-02-05T04:54:16.150 INFO:tasks.rados.rados.0.smithi196.stdout:347: finishing rollback tid 2 to smithi19639362-10 2024-02-05T04:54:16.151 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 10 v 85 (ObjNum 53 snap 0 seq_num 53) dirty exists 2024-02-05T04:54:16.151 INFO:tasks.rados.rados.0.smithi196.stdout:345: finishing write tid 1 to smithi19639362-33 2024-02-05T04:54:16.151 INFO:tasks.rados.rados.0.smithi196.stdout:345: finishing write tid 2 to smithi19639362-33 2024-02-05T04:54:16.152 INFO:tasks.rados.rados.0.smithi196.stdout:346: finishing rollback tid 1 to smithi19639362-44 2024-02-05T04:54:16.152 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 44 v 62 (ObjNum 97 snap 0 seq_num 3191445840) dirty dne 2024-02-05T04:54:16.155 INFO:tasks.rados.rados.0.smithi196.stdout:346: done (3 left) 2024-02-05T04:54:16.155 INFO:tasks.rados.rados.0.smithi196.stdout:347: done (2 left) 2024-02-05T04:54:16.155 INFO:tasks.rados.rados.0.smithi196.stdout:349: done (1 left) 2024-02-05T04:54:16.155 INFO:tasks.rados.rados.0.smithi196.stdout:350: read oid 12 snap -1 2024-02-05T04:54:16.155 INFO:tasks.rados.rados.0.smithi196.stdout:350: expect deleted 2024-02-05T04:54:16.155 INFO:tasks.rados.rados.0.smithi196.stdout:351: copy_from oid 32 from oid 21 current snap is 28 2024-02-05T04:54:16.155 INFO:tasks.rados.rados.0.smithi196.stdout:352: read oid 49 snap 27 2024-02-05T04:54:16.155 INFO:tasks.rados.rados.0.smithi196.stdout:352: expect (ObjNum 48 snap 0 seq_num 48) 2024-02-05T04:54:16.155 INFO:tasks.rados.rados.0.smithi196.stdout:353: read oid 5 snap -1 2024-02-05T04:54:16.155 INFO:tasks.rados.rados.0.smithi196.stdout:353: expect (ObjNum 4 snap 0 seq_num 4) 2024-02-05T04:54:16.155 INFO:tasks.rados.rados.0.smithi196.stdout:append oid 8 current snap is 28 2024-02-05T04:54:16.155 INFO:tasks.rados.rados.0.smithi196.stdout:354: seq_num 148 ranges {1277952=81920} 2024-02-05T04:54:16.163 INFO:tasks.rados.rados.0.smithi196.stdout:354: writing smithi19639362-8 from 1277952 to 1359872 tid 1 2024-02-05T04:54:16.163 INFO:tasks.rados.rados.0.smithi196.stdout:355: copy_from oid 17 from oid 42 current snap is 28 2024-02-05T04:54:16.163 INFO:tasks.rados.rados.0.smithi196.stdout:345: finishing write tid 3 to smithi19639362-33 2024-02-05T04:54:16.163 INFO:tasks.rados.rados.0.smithi196.stdout:345: oid 33 updating version 0 to 135 2024-02-05T04:54:16.163 INFO:tasks.rados.rados.0.smithi196.stdout:345: oid 33 updating version 135 to 136 2024-02-05T04:54:16.163 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 33 v 136 (ObjNum 145 snap 28 seq_num 145) dirty exists 2024-02-05T04:54:16.163 INFO:tasks.rados.rados.0.smithi196.stdout:345: left oid 33 (ObjNum 145 snap 28 seq_num 145) 2024-02-05T04:54:16.163 INFO:tasks.rados.rados.0.smithi196.stdout:353: expect (ObjNum 4 snap 0 seq_num 4) 2024-02-05T04:54:16.166 INFO:tasks.rados.rados.0.smithi196.stdout:352: expect (ObjNum 48 snap 0 seq_num 48) 2024-02-05T04:54:16.192 INFO:tasks.rados.rados.0.smithi196.stdout:351: finishing copy_from racing read to smithi19639362-32 2024-02-05T04:54:16.192 INFO:tasks.rados.rados.0.smithi196.stdout:355: finishing copy_from to smithi19639362-17 2024-02-05T04:54:16.192 INFO:tasks.rados.rados.0.smithi196.stdout:355: got expected ENOENT (src dne) 2024-02-05T04:54:16.192 INFO:tasks.rados.rados.0.smithi196.stdout:355: finishing copy_from racing read to smithi19639362-17 2024-02-05T04:54:16.192 INFO:tasks.rados.rados.0.smithi196.stdout:351: finishing copy_from to smithi19639362-32 2024-02-05T04:54:16.192 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 32 v 82 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-02-05T04:54:16.192 INFO:tasks.rados.rados.0.smithi196.stdout:354: finishing write tid 1 to smithi19639362-8 2024-02-05T04:54:16.192 INFO:tasks.rados.rados.0.smithi196.stdout:354: finishing write tid 2 to smithi19639362-8 2024-02-05T04:54:16.192 INFO:tasks.rados.rados.0.smithi196.stdout:345: done (6 left) 2024-02-05T04:54:16.192 INFO:tasks.rados.rados.0.smithi196.stdout:350: done (5 left) 2024-02-05T04:54:16.192 INFO:tasks.rados.rados.0.smithi196.stdout:351: done (4 left) 2024-02-05T04:54:16.192 INFO:tasks.rados.rados.0.smithi196.stdout:352: done (3 left) 2024-02-05T04:54:16.192 INFO:tasks.rados.rados.0.smithi196.stdout:353: done (2 left) 2024-02-05T04:54:16.192 INFO:tasks.rados.rados.0.smithi196.stdout:355: done (1 left) 2024-02-05T04:54:16.193 INFO:tasks.rados.rados.0.smithi196.stdout:append oid 49 current snap is 28 2024-02-05T04:54:16.193 INFO:tasks.rados.rados.0.smithi196.stdout:354: finishing write tid 3 to smithi19639362-8 2024-02-05T04:54:16.193 INFO:tasks.rados.rados.0.smithi196.stdout:354: oid 8 updating version 0 to 89 2024-02-05T04:54:16.193 INFO:tasks.rados.rados.0.smithi196.stdout:354: oid 8 version 89 is already newer than 88 2024-02-05T04:54:16.193 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 8 v 89 (ObjNum 148 snap 28 seq_num 148) dirty exists 2024-02-05T04:54:16.193 INFO:tasks.rados.rados.0.smithi196.stdout:354: left oid 8 (ObjNum 148 snap 28 seq_num 148) 2024-02-05T04:54:16.193 INFO:tasks.rados.rados.0.smithi196.stdout:356: seq_num 149 ranges {0=98304} 2024-02-05T04:54:16.193 INFO:tasks.rados.rados.0.smithi196.stdout:356: writing smithi19639362-49 from 0 to 98304 tid 1 2024-02-05T04:54:16.193 INFO:tasks.rados.rados.0.smithi196.stdout:354: done (1 left) 2024-02-05T04:54:16.193 INFO:tasks.rados.rados.0.smithi196.stdout:357: rollback oid 10 current snap is 28 2024-02-05T04:54:16.193 INFO:tasks.rados.rados.0.smithi196.stdout:rollback oid 10 to 21 2024-02-05T04:54:16.193 INFO:tasks.rados.rados.0.smithi196.stdout:append oid 37 current snap is 28 2024-02-05T04:54:16.193 INFO:tasks.rados.rados.0.smithi196.stdout:358: seq_num 150 ranges {0=114688} 2024-02-05T04:54:16.193 INFO:tasks.rados.rados.0.smithi196.stdout:358: writing smithi19639362-37 from 0 to 114688 tid 1 2024-02-05T04:54:16.193 INFO:tasks.rados.rados.0.smithi196.stdout:359: read oid 28 snap 23 2024-02-05T04:54:16.194 INFO:tasks.rados.rados.0.smithi196.stdout:359: expect (ObjNum 60 snap 2 seq_num 60) 2024-02-05T04:54:16.194 INFO:tasks.rados.rados.0.smithi196.stdout:360: rollback oid 38 current snap is 28 2024-02-05T04:54:16.194 INFO:tasks.rados.rados.0.smithi196.stdout:rollback oid 38 to 26 2024-02-05T04:54:16.194 INFO:tasks.rados.rados.0.smithi196.stdout:361: copy_from oid 3 from oid 30 current snap is 28 2024-02-05T04:54:16.194 INFO:tasks.rados.rados.0.smithi196.stdout:362: read oid 25 snap -1 2024-02-05T04:54:16.194 INFO:tasks.rados.rados.0.smithi196.stdout:362: expect (ObjNum 38 snap 0 seq_num 38) 2024-02-05T04:54:16.194 INFO:tasks.rados.rados.0.smithi196.stdout:append oid 19 current snap is 28 2024-02-05T04:54:16.194 INFO:tasks.rados.rados.0.smithi196.stdout:363: seq_num 151 ranges {393216=131072} 2024-02-05T04:54:16.196 INFO:tasks.rados.rados.0.smithi196.stdout:363: writing smithi19639362-19 from 393216 to 524288 tid 1 2024-02-05T04:54:16.196 INFO:tasks.rados.rados.0.smithi196.stdout:364: rollback oid 20 current snap is 28 2024-02-05T04:54:16.196 INFO:tasks.rados.rados.0.smithi196.stdout:rollback oid 20 to 23 2024-02-05T04:54:16.196 INFO:tasks.rados.rados.0.smithi196.stdout:365: delete oid 36 current snap is 28 2024-02-05T04:54:16.196 INFO:tasks.rados.rados.0.smithi196.stdout:357: finishing rollback tid 0 to smithi19639362-10 2024-02-05T04:54:16.196 INFO:tasks.rados.rados.0.smithi196.stdout:356: finishing write tid 1 to smithi19639362-49 2024-02-05T04:54:16.196 INFO:tasks.rados.rados.0.smithi196.stdout:356: finishing write tid 2 to smithi19639362-49 2024-02-05T04:54:16.197 INFO:tasks.rados.rados.0.smithi196.stdout:358: finishing write tid 1 to smithi19639362-37 2024-02-05T04:54:16.197 INFO:tasks.rados.rados.0.smithi196.stdout:357: finishing rollback tid 1 to smithi19639362-10 2024-02-05T04:54:16.197 INFO:tasks.rados.rados.0.smithi196.stdout:357: finishing rollback tid 2 to smithi19639362-10 2024-02-05T04:54:16.197 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 10 v 88 (ObjNum 53 snap 0 seq_num 53) dirty exists 2024-02-05T04:54:16.198 INFO:tasks.rados.rados.0.smithi196.stdout:358: finishing write tid 2 to smithi19639362-37 2024-02-05T04:54:16.199 INFO:tasks.rados.rados.0.smithi196.stdout:356: finishing write tid 3 to smithi19639362-49 2024-02-05T04:54:16.200 INFO:tasks.rados.rados.0.smithi196.stdout:356: oid 49 updating version 0 to 117 2024-02-05T04:54:16.200 INFO:tasks.rados.rados.0.smithi196.stdout:356: oid 49 updating version 117 to 118 2024-02-05T04:54:16.200 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 49 v 118 (ObjNum 149 snap 28 seq_num 149) dirty exists 2024-02-05T04:54:16.200 INFO:tasks.rados.rados.0.smithi196.stdout:356: left oid 49 (ObjNum 149 snap 28 seq_num 149) 2024-02-05T04:54:16.201 INFO:tasks.rados.rados.0.smithi196.stdout:364: finishing rollback tid 0 to smithi19639362-20 2024-02-05T04:54:16.202 INFO:tasks.rados.rados.0.smithi196.stdout:360: finishing rollback tid 1 to smithi19639362-38 2024-02-05T04:54:16.202 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 38 v 21 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-02-05T04:54:16.202 INFO:tasks.rados.rados.0.smithi196.stdout:364: finishing rollback tid 1 to smithi19639362-20 2024-02-05T04:54:16.202 INFO:tasks.rados.rados.0.smithi196.stdout:359: expect (ObjNum 60 snap 2 seq_num 60) 2024-02-05T04:54:16.237 INFO:tasks.ceph.osd.6.smithi150.stderr:2024-02-05T04:54:16.236+0000 7f6bea9f0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34015) UID: 0 2024-02-05T04:54:16.250 INFO:tasks.rados.rados.0.smithi196.stdout:362: expect (ObjNum 38 snap 0 seq_num 38) 2024-02-05T04:54:16.275 INFO:tasks.rados.rados.0.smithi196.stdout:358: finishing write tid 3 to smithi19639362-37 2024-02-05T04:54:16.275 INFO:tasks.rados.rados.0.smithi196.stdout:358: oid 37 updating version 0 to 69 2024-02-05T04:54:16.275 INFO:tasks.rados.rados.0.smithi196.stdout:358: oid 37 updating version 69 to 70 2024-02-05T04:54:16.275 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 37 v 70 (ObjNum 150 snap 28 seq_num 150) dirty exists 2024-02-05T04:54:16.276 INFO:tasks.rados.rados.0.smithi196.stdout:358: left oid 37 (ObjNum 150 snap 28 seq_num 150) 2024-02-05T04:54:16.276 INFO:tasks.rados.rados.0.smithi196.stdout:364: finishing rollback tid 2 to smithi19639362-20 2024-02-05T04:54:16.276 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 20 v 13 (ObjNum 112 snap 18 seq_num 112) dirty exists 2024-02-05T04:54:16.276 INFO:tasks.rados.rados.0.smithi196.stdout:363: finishing write tid 1 to smithi19639362-19 2024-02-05T04:54:16.276 INFO:tasks.rados.rados.0.smithi196.stdout:363: finishing write tid 2 to smithi19639362-19 2024-02-05T04:54:16.276 INFO:tasks.rados.rados.0.smithi196.stdout:356: done (9 left) 2024-02-05T04:54:16.276 INFO:tasks.rados.rados.0.smithi196.stdout:357: done (8 left) 2024-02-05T04:54:16.276 INFO:tasks.rados.rados.0.smithi196.stdout:358: done (7 left) 2024-02-05T04:54:16.276 INFO:tasks.rados.rados.0.smithi196.stdout:359: done (6 left) 2024-02-05T04:54:16.276 INFO:tasks.rados.rados.0.smithi196.stdout:360: done (5 left) 2024-02-05T04:54:16.276 INFO:tasks.rados.rados.0.smithi196.stdout:362: done (4 left) 2024-02-05T04:54:16.276 INFO:tasks.rados.rados.0.smithi196.stdout:364: done (3 left) 2024-02-05T04:54:16.276 INFO:tasks.rados.rados.0.smithi196.stdout:365: done (2 left) 2024-02-05T04:54:16.276 INFO:tasks.rados.rados.0.smithi196.stdout:append oid 18 current snap is 28 2024-02-05T04:54:16.277 INFO:tasks.rados.rados.0.smithi196.stdout:361: finishing copy_from racing read to smithi19639362-3 2024-02-05T04:54:16.277 INFO:tasks.rados.rados.0.smithi196.stdout:361: finishing copy_from to smithi19639362-3 2024-02-05T04:54:16.277 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 3 v 27 (ObjNum 60 snap 2 seq_num 60) dirty exists 2024-02-05T04:54:16.277 INFO:tasks.rados.rados.0.smithi196.stdout:363: finishing write tid 3 to smithi19639362-19 2024-02-05T04:54:16.277 INFO:tasks.rados.rados.0.smithi196.stdout:363: oid 19 updating version 0 to 79 2024-02-05T04:54:16.277 INFO:tasks.rados.rados.0.smithi196.stdout:363: oid 19 version 79 is already newer than 78 2024-02-05T04:54:16.277 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 19 v 79 (ObjNum 151 snap 28 seq_num 151) dirty exists 2024-02-05T04:54:16.277 INFO:tasks.rados.rados.0.smithi196.stdout:363: left oid 19 (ObjNum 151 snap 28 seq_num 151) 2024-02-05T04:54:16.277 INFO:tasks.rados.rados.0.smithi196.stdout:366: seq_num 153 ranges {0=163840} 2024-02-05T04:54:16.277 INFO:tasks.rados.rados.0.smithi196.stdout:366: writing smithi19639362-18 from 0 to 163840 tid 1 2024-02-05T04:54:16.277 INFO:tasks.rados.rados.0.smithi196.stdout:361: done (2 left) 2024-02-05T04:54:16.277 INFO:tasks.rados.rados.0.smithi196.stdout:363: done (1 left) 2024-02-05T04:54:16.277 INFO:tasks.rados.rados.0.smithi196.stdout:append oid 33 current snap is 28 2024-02-05T04:54:16.277 INFO:tasks.rados.rados.0.smithi196.stdout:367: seq_num 154 ranges {2162688=180224} 2024-02-05T04:54:16.283 INFO:teuthology.orchestra.run.smithi091.stdout:{ 2024-02-05T04:54:16.283 INFO:teuthology.orchestra.run.smithi091.stdout: "ops": [ 2024-02-05T04:54:16.283 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.283 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:918 3.4s0 3:25248d28:::smithi19639362-18:head [setxattr _header (57) in=64b] snapc 1d=[1d,1c,19,18,17,16,12,8] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-02-05T04:54:16.283 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:54:16.275503+0000", 2024-02-05T04:54:16.283 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 0.007130473, 2024-02-05T04:54:16.283 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0071497699999999997, 2024-02-05T04:54:16.283 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:16.283 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "started", 2024-02-05T04:54:16.283 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:54:16.283 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:54:16.284 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:16.284 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 918 2024-02-05T04:54:16.284 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.284 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:16.284 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.284 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:16.284 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.275503+0000", 2024-02-05T04:54:16.284 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:16.284 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.284 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.284 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:16.284 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.275503+0000", 2024-02-05T04:54:16.284 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:16.285 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.285 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.285 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:16.285 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.275504+0000", 2024-02-05T04:54:16.285 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.3000000000000001e-07 2024-02-05T04:54:16.285 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.285 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.285 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:16.285 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.275507+0000", 2024-02-05T04:54:16.285 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.5760000000000002e-06 2024-02-05T04:54:16.285 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.285 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.285 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:16.285 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.275514+0000", 2024-02-05T04:54:16.286 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.1369999999999996e-06 2024-02-05T04:54:16.286 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.286 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.286 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:16.286 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.275520+0000", 2024-02-05T04:54:16.286 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.5929999999999996e-06 2024-02-05T04:54:16.286 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.286 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.286 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:16.286 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.276523+0000", 2024-02-05T04:54:16.286 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.001003351 2024-02-05T04:54:16.286 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.286 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.287 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:54:16.287 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.276657+0000", 2024-02-05T04:54:16.287 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00013371100000000001 2024-02-05T04:54:16.287 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.287 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.287 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:16.287 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.277242+0000", 2024-02-05T04:54:16.287 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00058546499999999997 2024-02-05T04:54:16.287 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.287 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.287 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:16.287 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.278669+0000", 2024-02-05T04:54:16.287 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0014270070000000001 2024-02-05T04:54:16.287 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:16.288 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:16.288 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:16.288 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.288 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.288 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:919 3.4s0 3:25248d28:::smithi19639362-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e84)", 2024-02-05T04:54:16.288 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:54:16.275539+0000", 2024-02-05T04:54:16.288 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 0.0070945640000000003, 2024-02-05T04:54:16.288 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0071669489999999997, 2024-02-05T04:54:16.288 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:16.288 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "waiting for rw locks", 2024-02-05T04:54:16.288 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:54:16.288 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:54:16.288 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:16.288 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 919 2024-02-05T04:54:16.289 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.289 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:16.289 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.289 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:16.289 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.275539+0000", 2024-02-05T04:54:16.289 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:16.289 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.289 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.289 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:16.289 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.275539+0000", 2024-02-05T04:54:16.289 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:16.289 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.289 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.289 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:16.289 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.275539+0000", 2024-02-05T04:54:16.290 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.5999999999999999e-07 2024-02-05T04:54:16.290 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.290 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.290 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:16.290 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.275547+0000", 2024-02-05T04:54:16.290 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.7400000000000004e-06 2024-02-05T04:54:16.290 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.290 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.290 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:16.290 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.275555+0000", 2024-02-05T04:54:16.290 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.7309999999999993e-06 2024-02-05T04:54:16.290 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.290 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.290 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:16.290 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.275560+0000", 2024-02-05T04:54:16.291 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.5659999999999997e-06 2024-02-05T04:54:16.291 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.291 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.291 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:16.291 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.277516+0000", 2024-02-05T04:54:16.291 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0019556600000000001 2024-02-05T04:54:16.291 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.291 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.291 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "waiting for rw locks", 2024-02-05T04:54:16.291 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.277637+0000", 2024-02-05T04:54:16.291 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00012056 2024-02-05T04:54:16.291 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:16.291 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:16.291 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:16.292 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.292 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.292 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpRead(3.5s5 84/83 ECSubRead(tid=102, to_read={3:a78fa099:::smithi19639362-42:c=262144,262144,0}, subchunks={3:a78fa099:::smithi19639362-42:c=[0,1]}, attrs_to_read=))", 2024-02-05T04:54:16.292 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:54:16.281450+0000", 2024-02-05T04:54:16.292 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 0.001183826, 2024-02-05T04:54:16.292 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0012967460000000001, 2024-02-05T04:54:16.292 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:16.292 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:54:16.292 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:16.293 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.293 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:16.293 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.281450+0000", 2024-02-05T04:54:16.293 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:16.293 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.293 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.293 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:16.293 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.281450+0000", 2024-02-05T04:54:16.293 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:16.293 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.293 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.293 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:16.293 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.281450+0000", 2024-02-05T04:54:16.293 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.8400000000000004e-07 2024-02-05T04:54:16.294 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.294 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.294 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:16.294 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.281455+0000", 2024-02-05T04:54:16.294 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.1889999999999998e-06 2024-02-05T04:54:16.294 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.294 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.294 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:16.294 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.281467+0000", 2024-02-05T04:54:16.294 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.1487e-05 2024-02-05T04:54:16.294 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.294 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.294 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:16.294 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.281484+0000", 2024-02-05T04:54:16.295 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.6654e-05 2024-02-05T04:54:16.295 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.295 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.295 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:16.295 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.281528+0000", 2024-02-05T04:54:16.295 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.4508999999999997e-05 2024-02-05T04:54:16.295 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:16.295 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:16.295 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:16.295 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:16.295 INFO:teuthology.orchestra.run.smithi091.stdout: ], 2024-02-05T04:54:16.295 INFO:teuthology.orchestra.run.smithi091.stdout: "num_ops": 3 2024-02-05T04:54:16.295 INFO:teuthology.orchestra.run.smithi091.stdout:} 2024-02-05T04:54:16.296 INFO:tasks.rados.rados.0.smithi196.stdout:367: writing smithi19639362-33 from 2162688 to 2342912 tid 1 2024-02-05T04:54:16.296 INFO:tasks.rados.rados.0.smithi196.stdout:append oid 2 current snap is 28 2024-02-05T04:54:16.297 INFO:tasks.rados.rados.0.smithi196.stdout:368: seq_num 155 ranges {2818048=212992} 2024-02-05T04:54:16.297 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-02-05T04:54:16.306 INFO:tasks.rados.rados.0.smithi196.stdout:368: writing smithi19639362-2 from 2818048 to 3031040 tid 1 2024-02-05T04:54:16.306 INFO:tasks.rados.rados.0.smithi196.stdout:369: snap_create 2024-02-05T04:54:16.306 INFO:tasks.rados.rados.0.smithi196.stdout:366: finishing write tid 1 to smithi19639362-18 2024-02-05T04:54:16.306 INFO:tasks.rados.rados.0.smithi196.stdout:366: finishing write tid 2 to smithi19639362-18 2024-02-05T04:54:16.306 INFO:tasks.rados.rados.0.smithi196.stdout:366: finishing write tid 3 to smithi19639362-18 2024-02-05T04:54:16.307 INFO:tasks.rados.rados.0.smithi196.stdout:366: oid 18 updating version 0 to 86 2024-02-05T04:54:16.307 INFO:tasks.rados.rados.0.smithi196.stdout:366: oid 18 version 86 is already newer than 85 2024-02-05T04:54:16.307 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 18 v 86 (ObjNum 153 snap 28 seq_num 153) dirty exists 2024-02-05T04:54:16.307 INFO:tasks.rados.rados.0.smithi196.stdout:366: left oid 18 (ObjNum 153 snap 28 seq_num 153) 2024-02-05T04:54:16.307 INFO:tasks.rados.rados.0.smithi196.stdout:367: finishing write tid 1 to smithi19639362-33 2024-02-05T04:54:16.307 INFO:tasks.rados.rados.0.smithi196.stdout:367: finishing write tid 2 to smithi19639362-33 2024-02-05T04:54:16.307 INFO:tasks.rados.rados.0.smithi196.stdout:367: finishing write tid 3 to smithi19639362-33 2024-02-05T04:54:16.307 INFO:tasks.rados.rados.0.smithi196.stdout:367: oid 33 updating version 0 to 137 2024-02-05T04:54:16.307 INFO:tasks.rados.rados.0.smithi196.stdout:367: oid 33 updating version 137 to 138 2024-02-05T04:54:16.307 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 33 v 138 (ObjNum 154 snap 28 seq_num 154) dirty exists 2024-02-05T04:54:16.307 INFO:tasks.rados.rados.0.smithi196.stdout:367: left oid 33 (ObjNum 154 snap 28 seq_num 154) 2024-02-05T04:54:16.309 INFO:tasks.rados.rados.0.smithi196.stdout:368: finishing write tid 1 to smithi19639362-2 2024-02-05T04:54:16.310 INFO:tasks.rados.rados.0.smithi196.stdout:368: finishing write tid 2 to smithi19639362-2 2024-02-05T04:54:16.315 INFO:tasks.rados.rados.0.smithi196.stdout:368: finishing write tid 3 to smithi19639362-2 2024-02-05T04:54:16.315 INFO:tasks.rados.rados.0.smithi196.stdout:368: oid 2 updating version 0 to 141 2024-02-05T04:54:16.315 INFO:tasks.rados.rados.0.smithi196.stdout:368: oid 2 version 141 is already newer than 140 2024-02-05T04:54:16.315 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 2 v 141 (ObjNum 155 snap 28 seq_num 155) dirty exists 2024-02-05T04:54:16.315 INFO:tasks.rados.rados.0.smithi196.stdout:368: left oid 2 (ObjNum 155 snap 28 seq_num 155) 2024-02-05T04:54:16.339 INFO:tasks.ceph.osd.8.smithi062.stderr:2024-02-05T04:54:16.338+0000 7f9385bf2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 34693) UID: 0 2024-02-05T04:54:16.404 INFO:teuthology.orchestra.run.smithi091.stdout:{ 2024-02-05T04:54:16.404 INFO:teuthology.orchestra.run.smithi091.stdout: "ops": [], 2024-02-05T04:54:16.404 INFO:teuthology.orchestra.run.smithi091.stdout: "complaint_time": 30, 2024-02-05T04:54:16.404 INFO:teuthology.orchestra.run.smithi091.stdout: "num_blocked_ops": 0 2024-02-05T04:54:16.404 INFO:teuthology.orchestra.run.smithi091.stdout:} 2024-02-05T04:54:16.413 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-02-05T04:54:16.440 INFO:tasks.ceph.osd.8.smithi062.stderr:2024-02-05T04:54:16.439+0000 7f9385bf2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 34693) UID: 0 2024-02-05T04:54:16.520 INFO:teuthology.orchestra.run.smithi091.stdout:{ 2024-02-05T04:54:16.520 INFO:teuthology.orchestra.run.smithi091.stdout: "size": 20, 2024-02-05T04:54:16.520 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 600, 2024-02-05T04:54:16.520 INFO:teuthology.orchestra.run.smithi091.stdout: "ops": [ 2024-02-05T04:54:16.520 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.520 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:285 3.4s0 3:264f0b4b:::smithi19639362-35:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-02-05T04:54:16.520 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:30.205648+0000", 2024-02-05T04:54:16.520 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 46.311912155999998, 2024-02-05T04:54:16.520 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0099691669999999993, 2024-02-05T04:54:16.520 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:16.520 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "started", 2024-02-05T04:54:16.520 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:54:16.521 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:54:16.521 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:16.521 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 285 2024-02-05T04:54:16.521 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.521 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:16.521 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.521 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:16.521 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.205648+0000", 2024-02-05T04:54:16.521 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:16.521 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.521 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.521 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:16.521 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.205648+0000", 2024-02-05T04:54:16.521 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:16.521 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.522 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.522 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:16.522 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.205648+0000", 2024-02-05T04:54:16.522 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.7100000000000002e-07 2024-02-05T04:54:16.522 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.522 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.522 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:16.522 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.205651+0000", 2024-02-05T04:54:16.522 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.277e-06 2024-02-05T04:54:16.522 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.522 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.522 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:16.522 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.205658+0000", 2024-02-05T04:54:16.522 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.1709999999999996e-06 2024-02-05T04:54:16.522 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.523 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.523 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:16.523 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.205663+0000", 2024-02-05T04:54:16.523 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.4009999999999996e-06 2024-02-05T04:54:16.523 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.523 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.523 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:16.523 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.208044+0000", 2024-02-05T04:54:16.523 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.002380957 2024-02-05T04:54:16.523 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.523 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.523 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "waiting for rw locks", 2024-02-05T04:54:16.523 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.208122+0000", 2024-02-05T04:54:16.524 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.7688999999999994e-05 2024-02-05T04:54:16.524 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.524 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.524 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:16.524 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.212367+0000", 2024-02-05T04:54:16.524 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0042449569999999997 2024-02-05T04:54:16.524 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.524 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.524 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:54:16.524 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.212502+0000", 2024-02-05T04:54:16.524 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00013503199999999999 2024-02-05T04:54:16.524 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.524 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.524 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:16.525 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:30.215617+0000", 2024-02-05T04:54:16.525 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0031151120000000002 2024-02-05T04:54:16.525 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:16.525 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:16.525 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:16.525 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.525 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.525 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:500 3.4s0 3:2c15cb2e:::smithi19639362-6:head [append 0~524288] snapc d=[d,b,9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-02-05T04:54:16.525 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:44.963298+0000", 2024-02-05T04:54:16.525 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 31.554261874000002, 2024-02-05T04:54:16.525 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.010488153, 2024-02-05T04:54:16.525 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:16.525 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:16.525 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:54:16.525 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:54:16.526 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:16.526 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 500 2024-02-05T04:54:16.526 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.526 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:16.526 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.526 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:16.526 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.963298+0000", 2024-02-05T04:54:16.526 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:16.526 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.526 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.526 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:16.526 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.963298+0000", 2024-02-05T04:54:16.526 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:16.527 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.527 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.527 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:16.527 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.963301+0000", 2024-02-05T04:54:16.527 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.6769999999999999e-06 2024-02-05T04:54:16.527 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.527 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.527 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:16.527 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.965096+0000", 2024-02-05T04:54:16.527 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.001794618 2024-02-05T04:54:16.527 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.527 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.527 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:16.527 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.965120+0000", 2024-02-05T04:54:16.527 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.4161000000000001e-05 2024-02-05T04:54:16.528 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.528 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.528 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:16.528 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.965132+0000", 2024-02-05T04:54:16.528 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.2658e-05 2024-02-05T04:54:16.528 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.528 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.528 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:16.528 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.965206+0000", 2024-02-05T04:54:16.528 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.3632000000000002e-05 2024-02-05T04:54:16.528 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.528 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.528 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "op must wait for map", 2024-02-05T04:54:16.528 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.965213+0000", 2024-02-05T04:54:16.529 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.8399999999999997e-06 2024-02-05T04:54:16.529 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.529 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.529 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:16.529 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.967118+0000", 2024-02-05T04:54:16.529 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.001905688 2024-02-05T04:54:16.529 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.529 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.529 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:54:16.529 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.967190+0000", 2024-02-05T04:54:16.529 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.161e-05 2024-02-05T04:54:16.529 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.529 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.529 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:16.529 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.968120+0000", 2024-02-05T04:54:16.530 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00092962600000000002 2024-02-05T04:54:16.530 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.530 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.530 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:16.530 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.973623+0000", 2024-02-05T04:54:16.530 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0055030770000000003 2024-02-05T04:54:16.530 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.530 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.530 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "commit_sent", 2024-02-05T04:54:16.530 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.973717+0000", 2024-02-05T04:54:16.530 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.3845999999999996e-05 2024-02-05T04:54:16.530 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.530 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.530 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:16.530 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.973786+0000", 2024-02-05T04:54:16.531 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.9720000000000003e-05 2024-02-05T04:54:16.531 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:16.531 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:16.531 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:16.531 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.531 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.531 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:501 3.4s0 3:2c15cb2e:::smithi19639362-6:head [append 0~753664] snapc d=[d,b,9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-02-05T04:54:16.531 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:44.967519+0000", 2024-02-05T04:54:16.531 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 31.55004125, 2024-02-05T04:54:16.531 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.010834552000000001, 2024-02-05T04:54:16.531 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:16.531 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:16.531 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:54:16.531 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:54:16.531 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:16.531 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 501 2024-02-05T04:54:16.532 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.532 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:16.532 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.532 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:16.532 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.967519+0000", 2024-02-05T04:54:16.532 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:16.532 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.532 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.532 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:16.532 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.967519+0000", 2024-02-05T04:54:16.532 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:16.532 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.532 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.532 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:16.532 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.967520+0000", 2024-02-05T04:54:16.533 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.142e-06 2024-02-05T04:54:16.533 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.533 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.533 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:16.533 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.969864+0000", 2024-02-05T04:54:16.533 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0023435349999999999 2024-02-05T04:54:16.533 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.533 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.533 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:16.533 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.969881+0000", 2024-02-05T04:54:16.533 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.7592999999999999e-05 2024-02-05T04:54:16.533 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.533 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.533 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:16.533 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.969890+0000", 2024-02-05T04:54:16.534 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.2609999999999999e-06 2024-02-05T04:54:16.534 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.534 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.534 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:16.534 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.969975+0000", 2024-02-05T04:54:16.534 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.4382999999999995e-05 2024-02-05T04:54:16.534 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.534 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.534 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:54:16.534 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.970100+0000", 2024-02-05T04:54:16.534 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00012555199999999999 2024-02-05T04:54:16.534 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.534 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.534 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:16.534 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.971116+0000", 2024-02-05T04:54:16.535 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.001015665 2024-02-05T04:54:16.535 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.535 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.535 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:16.535 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.977653+0000", 2024-02-05T04:54:16.535 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0065369429999999999 2024-02-05T04:54:16.535 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.535 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.535 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "commit_sent", 2024-02-05T04:54:16.535 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.978281+0000", 2024-02-05T04:54:16.535 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00062775600000000004 2024-02-05T04:54:16.535 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.535 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.535 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:16.536 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.978353+0000", 2024-02-05T04:54:16.536 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.2721999999999999e-05 2024-02-05T04:54:16.536 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:16.536 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:16.536 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:16.536 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.536 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.536 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:502 3.4s0 3:2c15cb2e:::smithi19639362-6:head [append 0~409600] snapc d=[d,b,9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-02-05T04:54:16.536 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:44.970001+0000", 2024-02-05T04:54:16.536 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 31.547558631000001, 2024-02-05T04:54:16.536 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0088644669999999991, 2024-02-05T04:54:16.536 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:16.536 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:16.536 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:54:16.536 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:54:16.537 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:16.537 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 502 2024-02-05T04:54:16.537 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.537 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:16.537 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.537 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:16.537 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.970001+0000", 2024-02-05T04:54:16.537 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:16.537 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.537 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.537 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:16.537 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.970001+0000", 2024-02-05T04:54:16.537 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:16.537 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.537 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.538 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:16.538 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.970003+0000", 2024-02-05T04:54:16.538 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.7120000000000001e-06 2024-02-05T04:54:16.538 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.538 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.538 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:16.538 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.971437+0000", 2024-02-05T04:54:16.538 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.001433911 2024-02-05T04:54:16.538 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.538 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.538 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:16.538 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.971460+0000", 2024-02-05T04:54:16.538 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.3394e-05 2024-02-05T04:54:16.538 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.539 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.539 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:16.539 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.971482+0000", 2024-02-05T04:54:16.539 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.1137000000000001e-05 2024-02-05T04:54:16.539 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.539 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.539 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:16.539 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.972029+0000", 2024-02-05T04:54:16.539 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00054736499999999996 2024-02-05T04:54:16.539 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.539 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.539 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:54:16.539 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.972141+0000", 2024-02-05T04:54:16.539 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.000111742 2024-02-05T04:54:16.539 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.539 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.540 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:16.540 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.972917+0000", 2024-02-05T04:54:16.540 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00077584400000000003 2024-02-05T04:54:16.540 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.540 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.540 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:16.540 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.978062+0000", 2024-02-05T04:54:16.540 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0051453000000000002 2024-02-05T04:54:16.540 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.540 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.540 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "commit_sent", 2024-02-05T04:54:16.540 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.978801+0000", 2024-02-05T04:54:16.540 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00073907499999999995 2024-02-05T04:54:16.540 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.541 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.541 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:16.541 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.978866+0000", 2024-02-05T04:54:16.541 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.4987000000000006e-05 2024-02-05T04:54:16.541 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:16.541 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:16.541 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:16.541 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.541 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.541 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:504 3.4s0 3:2c15cb2e:::smithi19639362-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-02-05T04:54:16.541 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:53:44.971539+0000", 2024-02-05T04:54:16.541 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 31.546021240999998, 2024-02-05T04:54:16.541 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0087845130000000007, 2024-02-05T04:54:16.541 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:16.541 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "started", 2024-02-05T04:54:16.542 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:54:16.542 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:54:16.542 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:16.542 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 504 2024-02-05T04:54:16.542 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.542 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:16.542 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.542 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:16.542 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.971539+0000", 2024-02-05T04:54:16.542 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:16.542 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.542 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.542 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:16.542 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.971539+0000", 2024-02-05T04:54:16.542 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:16.543 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.543 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.543 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:16.543 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.971539+0000", 2024-02-05T04:54:16.543 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.0500000000000003e-07 2024-02-05T04:54:16.543 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.543 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.543 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:16.543 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.971542+0000", 2024-02-05T04:54:16.543 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.6970000000000002e-06 2024-02-05T04:54:16.543 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.543 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.543 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:16.543 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.971549+0000", 2024-02-05T04:54:16.543 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.9890000000000003e-06 2024-02-05T04:54:16.543 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.544 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.544 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:16.544 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.971555+0000", 2024-02-05T04:54:16.544 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.6690000000000003e-06 2024-02-05T04:54:16.544 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.544 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.544 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:16.544 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.974686+0000", 2024-02-05T04:54:16.544 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0031309939999999998 2024-02-05T04:54:16.544 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.544 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.544 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "waiting for rw locks", 2024-02-05T04:54:16.544 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.974749+0000", 2024-02-05T04:54:16.544 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.3167e-05 2024-02-05T04:54:16.544 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.545 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.545 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:16.545 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.979371+0000", 2024-02-05T04:54:16.545 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0046216349999999998 2024-02-05T04:54:16.545 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.545 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.545 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:54:16.545 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.979458+0000", 2024-02-05T04:54:16.545 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.7661e-05 2024-02-05T04:54:16.545 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.545 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.545 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:16.545 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:53:44.980323+0000", 2024-02-05T04:54:16.545 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00086509599999999999 2024-02-05T04:54:16.545 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:16.546 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:16.546 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:16.546 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.546 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.546 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.19s1 74/67 ECSubWrite(tid=9, reqid=osd.9.0:9, at_version=74'29, trim_to=0'0, roll_forward_to=72'25))", 2024-02-05T04:54:16.546 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:54:06.044912+0000", 2024-02-05T04:54:16.546 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 10.472647768, 2024-02-05T04:54:16.546 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0086155429999999998, 2024-02-05T04:54:16.546 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:16.546 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:54:16.546 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:16.546 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.546 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:16.546 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.044912+0000", 2024-02-05T04:54:16.546 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:16.547 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.547 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.547 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:16.547 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.044912+0000", 2024-02-05T04:54:16.547 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:16.547 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.547 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.547 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:16.547 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.044913+0000", 2024-02-05T04:54:16.547 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.47e-07 2024-02-05T04:54:16.547 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.547 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.547 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:16.547 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.044936+0000", 2024-02-05T04:54:16.547 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.2282000000000001e-05 2024-02-05T04:54:16.548 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.548 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.548 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:16.548 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.044946+0000", 2024-02-05T04:54:16.548 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.0438000000000001e-05 2024-02-05T04:54:16.548 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.548 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.548 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:16.548 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.044954+0000", 2024-02-05T04:54:16.548 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.1680000000000004e-06 2024-02-05T04:54:16.548 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.548 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.548 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:16.548 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.045511+0000", 2024-02-05T04:54:16.548 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00055711599999999997 2024-02-05T04:54:16.549 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.549 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.549 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:16.549 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.045522+0000", 2024-02-05T04:54:16.549 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.0601999999999999e-05 2024-02-05T04:54:16.549 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.549 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.549 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:16.549 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.053477+0000", 2024-02-05T04:54:16.549 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0079552640000000001 2024-02-05T04:54:16.549 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.549 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.549 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:16.549 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.053528+0000", 2024-02-05T04:54:16.549 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.0726000000000002e-05 2024-02-05T04:54:16.550 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:16.550 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:16.550 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:16.550 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.550 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.550 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.13s3 74/67 ECSubWrite(tid=153, reqid=osd.11.0:153, at_version=74'48, trim_to=0'0, roll_forward_to=72'46))", 2024-02-05T04:54:16.550 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:54:06.049780+0000", 2024-02-05T04:54:16.550 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 10.467780362999999, 2024-02-05T04:54:16.550 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0098180690000000004, 2024-02-05T04:54:16.550 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:16.550 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:54:16.550 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:16.550 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.550 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:16.550 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.049780+0000", 2024-02-05T04:54:16.551 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:16.551 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.551 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.551 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:16.551 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.049780+0000", 2024-02-05T04:54:16.551 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:16.551 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.551 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.551 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:16.551 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.049780+0000", 2024-02-05T04:54:16.551 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.3e-07 2024-02-05T04:54:16.551 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.551 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.551 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:16.551 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.049803+0000", 2024-02-05T04:54:16.551 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.2160000000000001e-05 2024-02-05T04:54:16.552 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.552 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.552 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:16.552 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.049814+0000", 2024-02-05T04:54:16.552 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.1778e-05 2024-02-05T04:54:16.552 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.552 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.552 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:16.552 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.049823+0000", 2024-02-05T04:54:16.552 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.4929999999999998e-06 2024-02-05T04:54:16.552 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.552 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.552 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:16.552 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.050001+0000", 2024-02-05T04:54:16.553 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00017782299999999999 2024-02-05T04:54:16.553 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.553 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.553 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:16.553 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.050011+0000", 2024-02-05T04:54:16.553 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.046e-05 2024-02-05T04:54:16.553 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.553 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.553 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:16.553 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.059563+0000", 2024-02-05T04:54:16.553 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0095519530000000002 2024-02-05T04:54:16.553 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.553 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.553 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:16.553 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.059598+0000", 2024-02-05T04:54:16.553 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.4672000000000003e-05 2024-02-05T04:54:16.554 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:16.554 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:16.554 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:16.554 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.554 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.554 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.3s2 74/62 ECSubWrite(tid=106, reqid=osd.4.0:105, at_version=0'0, trim_to=0'0, roll_forward_to=74'47))", 2024-02-05T04:54:16.554 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:54:06.050407+0000", 2024-02-05T04:54:16.554 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 10.467152773, 2024-02-05T04:54:16.554 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0091517449999999993, 2024-02-05T04:54:16.554 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:16.554 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:54:16.554 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:16.554 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.554 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:16.554 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.050407+0000", 2024-02-05T04:54:16.554 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:16.555 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.555 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.555 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:16.555 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.050407+0000", 2024-02-05T04:54:16.555 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:16.555 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.555 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.555 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:16.555 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.050408+0000", 2024-02-05T04:54:16.555 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.8400000000000003e-07 2024-02-05T04:54:16.555 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.555 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.555 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:16.555 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.050420+0000", 2024-02-05T04:54:16.555 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.1819e-05 2024-02-05T04:54:16.555 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.556 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.556 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:16.556 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.050433+0000", 2024-02-05T04:54:16.556 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.269e-05 2024-02-05T04:54:16.556 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.556 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.556 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:16.556 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.050441+0000", 2024-02-05T04:54:16.556 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.5150000000000006e-06 2024-02-05T04:54:16.556 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.556 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.556 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:16.556 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.050517+0000", 2024-02-05T04:54:16.556 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.6243999999999997e-05 2024-02-05T04:54:16.556 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.557 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.557 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:16.557 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.050527+0000", 2024-02-05T04:54:16.557 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.3810000000000005e-06 2024-02-05T04:54:16.557 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.557 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.557 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:16.557 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.059527+0000", 2024-02-05T04:54:16.557 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0089998509999999997 2024-02-05T04:54:16.557 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.557 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.557 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:16.557 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.059559+0000", 2024-02-05T04:54:16.557 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.2360999999999997e-05 2024-02-05T04:54:16.557 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:16.557 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:16.558 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:16.558 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.558 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.558 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.8s1 74/62 ECSubWrite(tid=107, reqid=osd.4.0:104, at_version=0'0, trim_to=0'0, roll_forward_to=74'37))", 2024-02-05T04:54:16.558 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:54:06.050958+0000", 2024-02-05T04:54:16.558 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 10.466602052000001, 2024-02-05T04:54:16.558 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0085881529999999994, 2024-02-05T04:54:16.558 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:16.558 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:54:16.558 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:16.558 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.558 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:16.558 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.050958+0000", 2024-02-05T04:54:16.558 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:16.558 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.558 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.559 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:16.559 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.050958+0000", 2024-02-05T04:54:16.559 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:16.559 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.559 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.559 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:16.559 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.050959+0000", 2024-02-05T04:54:16.559 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.2399999999999997e-07 2024-02-05T04:54:16.559 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.559 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.559 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:16.559 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.050971+0000", 2024-02-05T04:54:16.559 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.1916000000000001e-05 2024-02-05T04:54:16.559 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.560 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.560 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:16.560 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.050997+0000", 2024-02-05T04:54:16.560 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.5860999999999999e-05 2024-02-05T04:54:16.560 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.560 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.560 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:16.560 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.051005+0000", 2024-02-05T04:54:16.560 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.6209999999999993e-06 2024-02-05T04:54:16.560 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.560 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.560 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:16.560 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.051090+0000", 2024-02-05T04:54:16.560 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.4838000000000003e-05 2024-02-05T04:54:16.560 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.561 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.561 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:16.561 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.051101+0000", 2024-02-05T04:54:16.561 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.0607e-05 2024-02-05T04:54:16.561 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.561 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.561 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:16.561 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.059503+0000", 2024-02-05T04:54:16.561 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0084025599999999999 2024-02-05T04:54:16.561 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.561 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.561 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:16.561 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.059546+0000", 2024-02-05T04:54:16.561 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.2926000000000002e-05 2024-02-05T04:54:16.561 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:16.561 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:16.562 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:16.562 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.562 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.562 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.5s5 74/62 ECSubWrite(tid=79, reqid=osd.15.0:79, at_version=74'59, trim_to=0'0, roll_forward_to=72'57))", 2024-02-05T04:54:16.562 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:54:06.054419+0000", 2024-02-05T04:54:16.562 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 10.463140928, 2024-02-05T04:54:16.562 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.011377302000000001, 2024-02-05T04:54:16.562 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:16.562 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:54:16.562 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:16.562 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.562 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:16.562 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.054419+0000", 2024-02-05T04:54:16.562 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:16.562 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.563 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.563 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:16.563 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.054419+0000", 2024-02-05T04:54:16.563 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:16.563 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.563 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.563 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:16.563 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.054419+0000", 2024-02-05T04:54:16.563 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.27e-07 2024-02-05T04:54:16.563 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.563 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.563 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:16.563 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.054444+0000", 2024-02-05T04:54:16.563 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.4992999999999999e-05 2024-02-05T04:54:16.563 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.563 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.564 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:16.564 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.054461+0000", 2024-02-05T04:54:16.564 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.6169999999999999e-05 2024-02-05T04:54:16.564 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.564 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.564 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:16.564 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.054480+0000", 2024-02-05T04:54:16.564 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.8967999999999999e-05 2024-02-05T04:54:16.564 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.564 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.564 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:16.564 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.054571+0000", 2024-02-05T04:54:16.564 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.1335999999999994e-05 2024-02-05T04:54:16.564 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.564 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.565 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:16.565 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.054581+0000", 2024-02-05T04:54:16.565 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.0343000000000001e-05 2024-02-05T04:54:16.565 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.565 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.565 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:16.565 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.065755+0000", 2024-02-05T04:54:16.565 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.011174132 2024-02-05T04:54:16.565 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.565 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.565 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:16.565 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.065796+0000", 2024-02-05T04:54:16.565 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.1032999999999998e-05 2024-02-05T04:54:16.565 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:16.565 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:16.566 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:16.566 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.566 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.566 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.6s4 74/62 ECSubWrite(tid=341, reqid=osd.13.0:337, at_version=0'0, trim_to=0'0, roll_forward_to=74'38))", 2024-02-05T04:54:16.566 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:54:06.054605+0000", 2024-02-05T04:54:16.566 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 10.462954592999999, 2024-02-05T04:54:16.566 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0084369000000000006, 2024-02-05T04:54:16.566 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:16.566 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:54:16.566 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:16.566 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.566 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:16.566 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.054605+0000", 2024-02-05T04:54:16.566 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:16.566 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.567 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.567 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:16.567 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.054605+0000", 2024-02-05T04:54:16.567 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:16.567 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.567 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.567 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:16.567 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.054606+0000", 2024-02-05T04:54:16.567 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.5300000000000003e-07 2024-02-05T04:54:16.567 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.567 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.567 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:16.567 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.054619+0000", 2024-02-05T04:54:16.567 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.2401e-05 2024-02-05T04:54:16.567 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.568 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.568 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:16.568 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.054631+0000", 2024-02-05T04:54:16.568 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.2018000000000001e-05 2024-02-05T04:54:16.568 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.568 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.568 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:16.568 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.054640+0000", 2024-02-05T04:54:16.568 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.8969999999999997e-06 2024-02-05T04:54:16.568 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.568 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.568 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:16.568 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.054713+0000", 2024-02-05T04:54:16.568 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.3683000000000006e-05 2024-02-05T04:54:16.568 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.569 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.569 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:16.569 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.054727+0000", 2024-02-05T04:54:16.569 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.3633e-05 2024-02-05T04:54:16.569 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.569 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.569 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:16.569 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.063005+0000", 2024-02-05T04:54:16.569 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0082776210000000006 2024-02-05T04:54:16.569 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.569 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.569 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:16.569 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.063042+0000", 2024-02-05T04:54:16.569 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.7793999999999998e-05 2024-02-05T04:54:16.569 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:16.570 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:16.570 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:16.570 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.570 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.570 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.bs1 74/57 ECSubWrite(tid=340, reqid=client.4634.0:687, at_version=74'92, trim_to=0'0, roll_forward_to=74'88))", 2024-02-05T04:54:16.570 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:54:06.054677+0000", 2024-02-05T04:54:16.570 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 10.462882585999999, 2024-02-05T04:54:16.570 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.014039972, 2024-02-05T04:54:16.570 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:16.570 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:54:16.570 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:16.570 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.570 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:16.570 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.054677+0000", 2024-02-05T04:54:16.570 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:16.570 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.571 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.571 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:16.571 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.054677+0000", 2024-02-05T04:54:16.571 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:16.571 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.571 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.571 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:16.571 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.054678+0000", 2024-02-05T04:54:16.571 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.46e-07 2024-02-05T04:54:16.571 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.571 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.571 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:16.571 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.054796+0000", 2024-02-05T04:54:16.571 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.000118152 2024-02-05T04:54:16.571 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.572 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.572 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:16.572 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.054808+0000", 2024-02-05T04:54:16.572 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.145e-05 2024-02-05T04:54:16.572 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.572 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.572 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:16.572 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.054815+0000", 2024-02-05T04:54:16.572 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.7980000000000006e-06 2024-02-05T04:54:16.572 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.572 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.572 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:16.572 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.054950+0000", 2024-02-05T04:54:16.572 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.000134456 2024-02-05T04:54:16.572 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.573 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.573 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:16.573 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.054968+0000", 2024-02-05T04:54:16.573 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.861e-05 2024-02-05T04:54:16.573 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.573 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.573 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:16.573 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.068684+0000", 2024-02-05T04:54:16.573 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.013715364000000001 2024-02-05T04:54:16.573 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.573 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.573 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:16.573 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.068717+0000", 2024-02-05T04:54:16.573 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.3695999999999997e-05 2024-02-05T04:54:16.573 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:16.574 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:16.574 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:16.574 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.574 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.574 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "MOSDECSubOpWrite(3.5s5 74/62 ECSubWrite(tid=80, reqid=osd.15.0:80, at_version=74'61, trim_to=0'0, roll_forward_to=72'57))", 2024-02-05T04:54:16.574 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:54:06.058060+0000", 2024-02-05T04:54:16.574 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 10.459500022, 2024-02-05T04:54:16.574 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.010553674000000001, 2024-02-05T04:54:16.574 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:16.574 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "reached pg", 2024-02-05T04:54:16.574 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:16.574 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.574 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:16.574 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.058060+0000", 2024-02-05T04:54:16.574 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:16.574 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.575 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.575 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:16.575 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.058060+0000", 2024-02-05T04:54:16.575 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:16.575 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.575 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.575 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:16.575 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.058061+0000", 2024-02-05T04:54:16.575 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.7199999999999997e-07 2024-02-05T04:54:16.575 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.575 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.575 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:16.575 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.058084+0000", 2024-02-05T04:54:16.575 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.2657999999999999e-05 2024-02-05T04:54:16.575 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.575 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.576 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:16.576 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.058096+0000", 2024-02-05T04:54:16.576 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.2611e-05 2024-02-05T04:54:16.576 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.576 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.576 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:16.576 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.058106+0000", 2024-02-05T04:54:16.576 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.2629999999999992e-06 2024-02-05T04:54:16.576 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.576 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.576 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:16.576 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.060169+0000", 2024-02-05T04:54:16.576 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0020636539999999998 2024-02-05T04:54:16.576 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.576 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.576 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:16.577 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.060179+0000", 2024-02-05T04:54:16.577 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.6719999999999999e-06 2024-02-05T04:54:16.577 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.577 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.577 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:16.577 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.068579+0000", 2024-02-05T04:54:16.577 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0083999480000000008 2024-02-05T04:54:16.577 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.577 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.577 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:16.577 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:06.068614+0000", 2024-02-05T04:54:16.577 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.4895999999999999e-05 2024-02-05T04:54:16.577 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:16.577 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:16.577 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:16.577 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.578 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.578 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:810 3.4s0 3:22a8c9dd:::smithi19639362-45:head [read 0~2260992,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-02-05T04:54:16.578 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:54:11.099399+0000", 2024-02-05T04:54:16.578 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 5.4181606950000001, 2024-02-05T04:54:16.578 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.012351948, 2024-02-05T04:54:16.578 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:16.578 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "started", 2024-02-05T04:54:16.578 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:54:16.578 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:54:16.578 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:16.578 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 810 2024-02-05T04:54:16.578 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.578 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:16.578 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.578 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:16.578 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:11.099399+0000", 2024-02-05T04:54:16.579 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:16.579 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.579 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.579 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:16.579 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:11.099399+0000", 2024-02-05T04:54:16.579 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:16.579 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.579 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.579 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:16.579 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:11.099400+0000", 2024-02-05T04:54:16.579 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.6800000000000001e-07 2024-02-05T04:54:16.579 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.579 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.579 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:16.579 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:11.099404+0000", 2024-02-05T04:54:16.579 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.5430000000000004e-06 2024-02-05T04:54:16.580 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.580 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.580 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:16.580 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:11.099410+0000", 2024-02-05T04:54:16.580 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.0990000000000004e-06 2024-02-05T04:54:16.580 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.580 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.580 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:16.580 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:11.099416+0000", 2024-02-05T04:54:16.580 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.3669999999999996e-06 2024-02-05T04:54:16.580 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.580 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.580 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:16.580 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:11.107843+0000", 2024-02-05T04:54:16.581 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0084272989999999992 2024-02-05T04:54:16.581 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.581 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.581 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:54:16.581 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:11.107960+0000", 2024-02-05T04:54:16.581 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.000116449 2024-02-05T04:54:16.581 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.581 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.581 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:16.581 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:11.111751+0000", 2024-02-05T04:54:16.581 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0037917229999999999 2024-02-05T04:54:16.581 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:16.581 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:16.581 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:16.581 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.581 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.581 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:845 3.bs1 3:d0b920f1:::smithi19639362-47:head [copy-from ver 6] snapc 1d=[1d,1c,1b,19,18,17,16,12,8] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e83)", 2024-02-05T04:54:16.582 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:54:15.077293+0000", 2024-02-05T04:54:16.582 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 1.440267159, 2024-02-05T04:54:16.582 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.0229032659999999, 2024-02-05T04:54:16.582 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:16.582 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:16.582 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:54:16.582 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:54:16.582 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:16.582 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 845 2024-02-05T04:54:16.582 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.582 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:16.582 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.582 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:16.582 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:15.077293+0000", 2024-02-05T04:54:16.582 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:16.583 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.583 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.583 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:16.583 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:15.077293+0000", 2024-02-05T04:54:16.583 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:16.583 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.583 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.583 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:16.583 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:15.077294+0000", 2024-02-05T04:54:16.583 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.2619999999999999e-06 2024-02-05T04:54:16.583 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.583 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.583 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:16.583 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:15.077299+0000", 2024-02-05T04:54:16.583 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.5900000000000001e-06 2024-02-05T04:54:16.584 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.584 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.584 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:16.584 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:15.077307+0000", 2024-02-05T04:54:16.584 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 8.1969999999999996e-06 2024-02-05T04:54:16.584 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.584 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.584 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:16.584 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:15.077314+0000", 2024-02-05T04:54:16.584 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.7630000000000003e-06 2024-02-05T04:54:16.584 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.584 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.584 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:16.584 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:15.077381+0000", 2024-02-05T04:54:16.585 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.7568000000000001e-05 2024-02-05T04:54:16.585 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.585 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.585 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "waiting for peered", 2024-02-05T04:54:16.585 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:15.077385+0000", 2024-02-05T04:54:16.585 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.4589999999999998e-06 2024-02-05T04:54:16.585 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.585 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.585 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:16.585 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.085687+0000", 2024-02-05T04:54:16.585 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.0083026559999999 2024-02-05T04:54:16.585 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.585 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.585 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:54:16.585 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.085988+0000", 2024-02-05T04:54:16.586 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00030026199999999998 2024-02-05T04:54:16.586 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.586 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.586 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:16.586 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.095325+0000", 2024-02-05T04:54:16.586 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.009337204 2024-02-05T04:54:16.586 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.586 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.586 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:16.586 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.097870+0000", 2024-02-05T04:54:16.586 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0025448430000000002 2024-02-05T04:54:16.586 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.586 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.586 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "commit_sent", 2024-02-05T04:54:16.586 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.099885+0000", 2024-02-05T04:54:16.587 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0020150760000000002 2024-02-05T04:54:16.587 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.587 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.587 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:16.587 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.100196+0000", 2024-02-05T04:54:16.587 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00031138599999999998 2024-02-05T04:54:16.587 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:16.587 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:16.587 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:16.587 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.587 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.587 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:846 3.bs1 3:d0b920f1:::smithi19639362-47:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e83)", 2024-02-05T04:54:16.587 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:54:15.077325+0000", 2024-02-05T04:54:16.587 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 1.4402350529999999, 2024-02-05T04:54:16.587 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.0238196959999999, 2024-02-05T04:54:16.588 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:16.588 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "started", 2024-02-05T04:54:16.588 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:54:16.588 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:54:16.588 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:16.588 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 846 2024-02-05T04:54:16.588 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.588 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:16.588 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.588 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:16.588 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:15.077325+0000", 2024-02-05T04:54:16.588 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:16.588 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.588 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.588 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:16.589 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:15.077325+0000", 2024-02-05T04:54:16.589 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:16.589 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.589 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.589 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:16.589 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:15.077325+0000", 2024-02-05T04:54:16.589 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.2000000000000001e-07 2024-02-05T04:54:16.589 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.589 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.589 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:16.589 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:15.077327+0000", 2024-02-05T04:54:16.589 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.5710000000000001e-06 2024-02-05T04:54:16.589 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.589 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.589 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:16.589 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:15.077331+0000", 2024-02-05T04:54:16.590 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.3050000000000003e-06 2024-02-05T04:54:16.590 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.590 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.590 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:16.590 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:15.077334+0000", 2024-02-05T04:54:16.590 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.2399999999999999e-06 2024-02-05T04:54:16.590 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.590 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.590 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:16.590 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:15.077414+0000", 2024-02-05T04:54:16.590 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.9085999999999995e-05 2024-02-05T04:54:16.590 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.590 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.590 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "waiting for peered", 2024-02-05T04:54:16.590 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:15.077416+0000", 2024-02-05T04:54:16.590 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.978e-06 2024-02-05T04:54:16.591 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.591 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.591 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:16.591 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.086761+0000", 2024-02-05T04:54:16.591 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.009344979 2024-02-05T04:54:16.591 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.591 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.591 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "waiting for blocked object", 2024-02-05T04:54:16.591 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.086832+0000", 2024-02-05T04:54:16.591 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.1778000000000003e-05 2024-02-05T04:54:16.591 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.591 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.591 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:16.591 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.096409+0000", 2024-02-05T04:54:16.591 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0095764549999999993 2024-02-05T04:54:16.591 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.591 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.592 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "waiting for rw locks", 2024-02-05T04:54:16.592 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.096530+0000", 2024-02-05T04:54:16.592 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00012171399999999999 2024-02-05T04:54:16.592 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.592 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.592 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:16.592 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.100682+0000", 2024-02-05T04:54:16.592 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0041513260000000003 2024-02-05T04:54:16.592 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.592 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.592 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:54:16.592 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.100842+0000", 2024-02-05T04:54:16.592 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00016020699999999999 2024-02-05T04:54:16.592 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.592 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.592 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:16.593 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.101145+0000", 2024-02-05T04:54:16.593 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00030273700000000003 2024-02-05T04:54:16.593 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:16.593 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:16.593 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:16.593 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.593 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.593 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:866 3.bs1 3:df6528f1:::smithi19639362-33:head [append 0~32768] snapc 1d=[1d,1c,19,18,17,16,12,8] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-02-05T04:54:16.593 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:54:16.142613+0000", 2024-02-05T04:54:16.593 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 0.37494666500000001, 2024-02-05T04:54:16.593 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0089806929999999997, 2024-02-05T04:54:16.593 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:16.593 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:16.593 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:54:16.593 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:54:16.593 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:16.594 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 866 2024-02-05T04:54:16.594 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.594 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:16.594 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.594 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:16.594 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.142613+0000", 2024-02-05T04:54:16.594 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:16.594 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.594 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.594 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:16.594 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.142613+0000", 2024-02-05T04:54:16.594 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:16.594 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.594 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.594 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:16.594 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.142615+0000", 2024-02-05T04:54:16.594 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.5030000000000001e-06 2024-02-05T04:54:16.595 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.595 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.595 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:16.595 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.142706+0000", 2024-02-05T04:54:16.595 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.0939999999999993e-05 2024-02-05T04:54:16.595 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.595 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.595 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:16.595 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.142720+0000", 2024-02-05T04:54:16.595 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 1.4243999999999999e-05 2024-02-05T04:54:16.595 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.595 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.595 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:16.595 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.142729+0000", 2024-02-05T04:54:16.595 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.3330000000000003e-06 2024-02-05T04:54:16.595 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.595 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.596 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:16.596 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.142782+0000", 2024-02-05T04:54:16.596 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.2429999999999998e-05 2024-02-05T04:54:16.596 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.596 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.596 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:54:16.596 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.143040+0000", 2024-02-05T04:54:16.596 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.000258053 2024-02-05T04:54:16.596 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.596 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.596 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:16.596 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.144146+0000", 2024-02-05T04:54:16.596 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.001106023 2024-02-05T04:54:16.596 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.596 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.597 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:16.597 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.150942+0000", 2024-02-05T04:54:16.597 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0067963839999999999 2024-02-05T04:54:16.597 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.597 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.597 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "commit_sent", 2024-02-05T04:54:16.597 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.151203+0000", 2024-02-05T04:54:16.597 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00026041299999999997 2024-02-05T04:54:16.597 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.597 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.597 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:16.597 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.151594+0000", 2024-02-05T04:54:16.597 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00039136999999999999 2024-02-05T04:54:16.597 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:16.597 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:16.597 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:16.597 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.598 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.598 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:867 3.bs1 3:df6528f1:::smithi19639362-33:head [setxattr _header (57) in=7b] snapc 1d=[1d,1c,19,18,17,16,12,8] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-02-05T04:54:16.598 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:54:16.142745+0000", 2024-02-05T04:54:16.598 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 0.37481542400000001, 2024-02-05T04:54:16.598 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0096244650000000005, 2024-02-05T04:54:16.598 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:16.598 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:16.598 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:54:16.598 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:54:16.598 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:16.598 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 867 2024-02-05T04:54:16.598 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.598 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:16.598 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.598 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:16.598 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.142745+0000", 2024-02-05T04:54:16.598 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:16.599 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.599 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.599 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:16.599 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.142745+0000", 2024-02-05T04:54:16.599 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:16.599 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.599 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.599 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:16.599 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.142745+0000", 2024-02-05T04:54:16.599 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.75e-07 2024-02-05T04:54:16.599 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.599 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.599 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:16.599 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.142748+0000", 2024-02-05T04:54:16.599 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 3.2059999999999999e-06 2024-02-05T04:54:16.600 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.600 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.600 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:16.600 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.142755+0000", 2024-02-05T04:54:16.600 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 6.6749999999999996e-06 2024-02-05T04:54:16.600 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.600 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.600 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:16.600 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.142761+0000", 2024-02-05T04:54:16.600 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.5539999999999997e-06 2024-02-05T04:54:16.600 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.600 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.600 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:16.600 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.145266+0000", 2024-02-05T04:54:16.600 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0025051219999999998 2024-02-05T04:54:16.601 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.601 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.601 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:54:16.601 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.145496+0000", 2024-02-05T04:54:16.601 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00023002999999999999 2024-02-05T04:54:16.601 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.601 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.601 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_started", 2024-02-05T04:54:16.601 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.146149+0000", 2024-02-05T04:54:16.601 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00065332699999999999 2024-02-05T04:54:16.601 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.601 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.601 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "sub_op_committed", 2024-02-05T04:54:16.601 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.151599+0000", 2024-02-05T04:54:16.601 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0054495419999999999 2024-02-05T04:54:16.601 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.602 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.602 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "commit_sent", 2024-02-05T04:54:16.602 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.151695+0000", 2024-02-05T04:54:16.602 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 9.5947999999999997e-05 2024-02-05T04:54:16.602 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.602 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.602 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:16.602 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.152369+0000", 2024-02-05T04:54:16.602 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00067448599999999997 2024-02-05T04:54:16.602 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:16.602 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:16.602 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:16.602 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.602 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.602 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:868 3.bs1 3:df6528f1:::smithi19639362-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e84)", 2024-02-05T04:54:16.602 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:54:16.142768+0000", 2024-02-05T04:54:16.603 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 0.37479175799999997, 2024-02-05T04:54:16.603 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.011967857, 2024-02-05T04:54:16.603 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:16.603 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "started", 2024-02-05T04:54:16.603 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:54:16.603 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:54:16.603 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:16.603 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 868 2024-02-05T04:54:16.603 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.603 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:16.603 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.603 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:16.603 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.142768+0000", 2024-02-05T04:54:16.603 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:16.604 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.604 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.604 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:16.604 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.142768+0000", 2024-02-05T04:54:16.604 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:16.604 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.604 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.604 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:16.604 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.142769+0000", 2024-02-05T04:54:16.604 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.34e-07 2024-02-05T04:54:16.604 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.604 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.604 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:16.604 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.142771+0000", 2024-02-05T04:54:16.604 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 2.4260000000000002e-06 2024-02-05T04:54:16.605 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.605 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.605 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:16.605 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.142779+0000", 2024-02-05T04:54:16.605 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.391e-06 2024-02-05T04:54:16.605 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.605 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.605 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:16.605 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.142784+0000", 2024-02-05T04:54:16.605 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.0509999999999996e-06 2024-02-05T04:54:16.605 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.605 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.605 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:16.605 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.147206+0000", 2024-02-05T04:54:16.605 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0044224809999999998 2024-02-05T04:54:16.605 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.605 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.606 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "waiting for rw locks", 2024-02-05T04:54:16.606 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.147374+0000", 2024-02-05T04:54:16.606 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00016807000000000001 2024-02-05T04:54:16.606 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.606 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.606 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:16.606 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.152407+0000", 2024-02-05T04:54:16.606 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0050331880000000001 2024-02-05T04:54:16.606 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.606 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.606 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:54:16.606 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.152561+0000", 2024-02-05T04:54:16.606 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00015354 2024-02-05T04:54:16.606 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.606 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.606 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:16.606 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.154736+0000", 2024-02-05T04:54:16.606 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0021752759999999999 2024-02-05T04:54:16.607 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:16.607 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:16.607 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:16.607 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.607 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.607 INFO:teuthology.orchestra.run.smithi091.stdout: "description": "osd_op(client.4634.0:919 3.4s0 3:25248d28:::smithi19639362-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e84)", 2024-02-05T04:54:16.607 INFO:teuthology.orchestra.run.smithi091.stdout: "initiated_at": "2024-02-05T04:54:16.275539+0000", 2024-02-05T04:54:16.607 INFO:teuthology.orchestra.run.smithi091.stdout: "age": 0.242021191, 2024-02-05T04:54:16.607 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0094218419999999997, 2024-02-05T04:54:16.607 INFO:teuthology.orchestra.run.smithi091.stdout: "type_data": { 2024-02-05T04:54:16.607 INFO:teuthology.orchestra.run.smithi091.stdout: "flag_point": "started", 2024-02-05T04:54:16.607 INFO:teuthology.orchestra.run.smithi091.stdout: "client_info": { 2024-02-05T04:54:16.607 INFO:teuthology.orchestra.run.smithi091.stdout: "client": "client.4634", 2024-02-05T04:54:16.607 INFO:teuthology.orchestra.run.smithi091.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:16.607 INFO:teuthology.orchestra.run.smithi091.stdout: "tid": 919 2024-02-05T04:54:16.607 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.608 INFO:teuthology.orchestra.run.smithi091.stdout: "events": [ 2024-02-05T04:54:16.608 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.608 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "initiated", 2024-02-05T04:54:16.608 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.275539+0000", 2024-02-05T04:54:16.608 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:16.608 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.608 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.608 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "header_read", 2024-02-05T04:54:16.608 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.275539+0000", 2024-02-05T04:54:16.608 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0 2024-02-05T04:54:16.608 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.608 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.608 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "throttled", 2024-02-05T04:54:16.608 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.275539+0000", 2024-02-05T04:54:16.608 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 4.5999999999999999e-07 2024-02-05T04:54:16.609 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.609 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.609 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "all_read", 2024-02-05T04:54:16.609 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.275547+0000", 2024-02-05T04:54:16.609 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.7400000000000004e-06 2024-02-05T04:54:16.609 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.609 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.609 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "dispatched", 2024-02-05T04:54:16.609 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.275555+0000", 2024-02-05T04:54:16.609 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 7.7309999999999993e-06 2024-02-05T04:54:16.609 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.609 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.609 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "queued_for_pg", 2024-02-05T04:54:16.609 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.275560+0000", 2024-02-05T04:54:16.610 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 5.5659999999999997e-06 2024-02-05T04:54:16.610 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.610 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.610 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:16.610 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.277516+0000", 2024-02-05T04:54:16.610 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0019556600000000001 2024-02-05T04:54:16.610 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.610 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.610 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "waiting for rw locks", 2024-02-05T04:54:16.610 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.277637+0000", 2024-02-05T04:54:16.610 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.00012056 2024-02-05T04:54:16.610 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.610 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.610 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "reached_pg", 2024-02-05T04:54:16.610 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.283561+0000", 2024-02-05T04:54:16.610 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0059244789999999999 2024-02-05T04:54:16.611 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.611 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.611 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "started", 2024-02-05T04:54:16.611 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.283677+0000", 2024-02-05T04:54:16.611 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.000116346 2024-02-05T04:54:16.611 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2024-02-05T04:54:16.611 INFO:teuthology.orchestra.run.smithi091.stdout: { 2024-02-05T04:54:16.611 INFO:teuthology.orchestra.run.smithi091.stdout: "event": "done", 2024-02-05T04:54:16.611 INFO:teuthology.orchestra.run.smithi091.stdout: "time": "2024-02-05T04:54:16.284961+0000", 2024-02-05T04:54:16.611 INFO:teuthology.orchestra.run.smithi091.stdout: "duration": 0.0012833 2024-02-05T04:54:16.611 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:16.611 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:16.611 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:16.611 INFO:teuthology.orchestra.run.smithi091.stdout: } 2024-02-05T04:54:16.611 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2024-02-05T04:54:16.611 INFO:teuthology.orchestra.run.smithi091.stdout:} 2024-02-05T04:54:16.613 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-02-05T04:54:16.614 INFO:tasks.ceph.osd.7.smithi196.stderr:2024-02-05T04:54:16.612+0000 7fb281419640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34606) UID: 0 2024-02-05T04:54:16.671 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T04:54:16.714 INFO:tasks.ceph.osd.2.smithi150.stderr:2024-02-05T04:54:16.714+0000 7f7337bf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 34014) UID: 0 2024-02-05T04:54:16.721 INFO:teuthology.orchestra.run.smithi091.stdout:{ 2024-02-05T04:54:16.722 INFO:teuthology.orchestra.run.smithi091.stdout: "ops": [], 2024-02-05T04:54:16.722 INFO:teuthology.orchestra.run.smithi091.stdout: "num_ops": 0 2024-02-05T04:54:16.722 INFO:teuthology.orchestra.run.smithi091.stdout:} 2024-02-05T04:54:16.730 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-02-05T04:54:16.800 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0: {} 2024-02-05T04:54:16.800 INFO:teuthology.orchestra.run.smithi062.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T04:54:16.815 INFO:tasks.ceph.osd.14.smithi150.stderr:2024-02-05T04:54:16.813+0000 7f5f81083640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 34020) UID: 0 2024-02-05T04:54:16.818 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1: {} 2024-02-05T04:54:16.818 INFO:teuthology.orchestra.run.smithi062.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T04:54:16.836 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2: {} 2024-02-05T04:54:16.836 INFO:teuthology.orchestra.run.smithi062.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T04:54:16.837 INFO:teuthology.orchestra.run.smithi091.stdout:{ 2024-02-05T04:54:16.837 INFO:teuthology.orchestra.run.smithi091.stdout: "ops": [], 2024-02-05T04:54:16.837 INFO:teuthology.orchestra.run.smithi091.stdout: "complaint_time": 30, 2024-02-05T04:54:16.837 INFO:teuthology.orchestra.run.smithi091.stdout: "num_blocked_ops": 0 2024-02-05T04:54:16.837 INFO:teuthology.orchestra.run.smithi091.stdout:} 2024-02-05T04:54:16.845 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-02-05T04:54:16.854 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3: {} 2024-02-05T04:54:16.854 INFO:teuthology.orchestra.run.smithi062.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T04:54:16.871 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4: {} 2024-02-05T04:54:16.871 INFO:teuthology.orchestra.run.smithi062.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T04:54:16.888 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5: {} 2024-02-05T04:54:16.888 INFO:teuthology.orchestra.run.smithi062.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T04:54:16.905 INFO:teuthology.orchestra.run.smithi062.stdout:osd.6: {} 2024-02-05T04:54:16.906 INFO:teuthology.orchestra.run.smithi062.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T04:54:16.915 INFO:tasks.ceph.osd.6.smithi150.stderr:2024-02-05T04:54:16.915+0000 7f6bea9f0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34015) UID: 0 2024-02-05T04:54:16.923 INFO:teuthology.orchestra.run.smithi062.stdout:osd.7: {} 2024-02-05T04:54:16.923 INFO:teuthology.orchestra.run.smithi062.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T04:54:16.940 INFO:teuthology.orchestra.run.smithi062.stdout:osd.8: {} 2024-02-05T04:54:16.940 INFO:teuthology.orchestra.run.smithi062.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-05T04:54:16.952 INFO:teuthology.orchestra.run.smithi091.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:16.952 INFO:teuthology.orchestra.run.smithi091.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:16.958 INFO:teuthology.orchestra.run.smithi062.stdout:osd.9: {} 2024-02-05T04:54:16.958 INFO:teuthology.orchestra.run.smithi062.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-05T04:54:16.961 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-02-05T04:54:16.975 INFO:teuthology.orchestra.run.smithi091.stderr:Exporting 3.7s0 info 3.7s0( v 79'38 (0'0,79'38] local-lis/les=62/63 n=3 ec=31/31 lis/c=62/62 les/c/f=63/63/0 sis=62) 2024-02-05T04:54:16.976 INFO:teuthology.orchestra.run.smithi062.stdout:osd.10: {} 2024-02-05T04:54:16.976 INFO:teuthology.orchestra.run.smithi062.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-05T04:54:16.978 INFO:teuthology.orchestra.run.smithi091.stderr:Read 0#3:e54c8e71:::smithi19639362-27:c# 2024-02-05T04:54:16.978 INFO:teuthology.orchestra.run.smithi091.stderr:Read 0#3:e54c8e71:::smithi19639362-27:1b# 2024-02-05T04:54:16.980 INFO:teuthology.orchestra.run.smithi091.stderr:Read 0#3:e54c8e71:::smithi19639362-27:head# 2024-02-05T04:54:16.983 INFO:teuthology.orchestra.run.smithi091.stderr:Export successful 2024-02-05T04:54:16.993 INFO:teuthology.orchestra.run.smithi062.stdout:osd.11: {} 2024-02-05T04:54:16.994 INFO:teuthology.orchestra.run.smithi062.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-05T04:54:17.011 INFO:teuthology.orchestra.run.smithi062.stdout:osd.12: {} 2024-02-05T04:54:17.011 INFO:teuthology.orchestra.run.smithi062.stderr:osd.12: osd_enable_op_tracker = '' 2024-02-05T04:54:17.011 INFO:teuthology.orchestra.run.smithi062.stderr:Error ENXIO: problem getting command descriptions from osd.13 2024-02-05T04:54:17.016 INFO:tasks.ceph.osd.0.smithi062.stderr:2024-02-05T04:54:17.014+0000 7fece8e03640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34688) UID: 0 2024-02-05T04:54:17.031 INFO:teuthology.orchestra.run.smithi062.stdout:osd.14: {} 2024-02-05T04:54:17.031 INFO:teuthology.orchestra.run.smithi062.stderr:osd.14: osd_enable_op_tracker = '' 2024-02-05T04:54:17.047 INFO:teuthology.orchestra.run.smithi062.stdout:osd.15: {} 2024-02-05T04:54:17.047 INFO:teuthology.orchestra.run.smithi062.stderr:osd.15: osd_enable_op_tracker = '' 2024-02-05T04:54:17.050 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-05T04:54:17.050 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-05T04:54:17.069 INFO:teuthology.orchestra.run.smithi091.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:17.069 INFO:teuthology.orchestra.run.smithi091.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:17.078 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-02-05T04:54:17.079 INFO:tasks.rados.rados.0.smithi196.stdout:366: done (3 left) 2024-02-05T04:54:17.079 INFO:tasks.rados.rados.0.smithi196.stdout:367: done (2 left) 2024-02-05T04:54:17.079 INFO:tasks.rados.rados.0.smithi196.stdout:368: done (1 left) 2024-02-05T04:54:17.079 INFO:tasks.rados.rados.0.smithi196.stdout:369: done (0 left) 2024-02-05T04:54:17.079 INFO:tasks.rados.rados.0.smithi196.stdout:370: snap_create 2024-02-05T04:54:17.118 INFO:tasks.ceph.osd.11.smithi196.stderr:2024-02-05T04:54:17.116+0000 7f85a8d86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 34609) UID: 0 2024-02-05T04:54:17.121 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-02-05T04:54:17.218 INFO:tasks.ceph.osd.4.smithi062.stderr:2024-02-05T04:54:17.216+0000 7f708c308640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34689) UID: 0 2024-02-05T04:54:17.223 INFO:teuthology.orchestra.run.smithi091.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:17.223 INFO:teuthology.orchestra.run.smithi091.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:17.232 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-02-05T04:54:17.319 INFO:tasks.ceph.osd.9.smithi091.stderr:2024-02-05T04:54:17.317+0000 7efc18827640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 34279) UID: 0 2024-02-05T04:54:17.337 INFO:teuthology.orchestra.run.smithi091.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:17.338 INFO:teuthology.orchestra.run.smithi091.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:17.346 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-02-05T04:54:17.420 INFO:tasks.ceph.osd.11.smithi196.stderr:2024-02-05T04:54:17.420+0000 7f85a8d86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 34609) UID: 0 2024-02-05T04:54:17.454 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:17.454 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:17.463 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-02-05T04:54:17.502 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-13 --journal-path /var/lib/ceph/osd/ceph-13/journal --force --op remove --pgid 3.7s0 2024-02-05T04:54:17.521 INFO:tasks.ceph.osd.8.smithi062.stderr:2024-02-05T04:54:17.519+0000 7f9385bf2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 34693) UID: 0 2024-02-05T04:54:17.570 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:17.570 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:17.579 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-02-05T04:54:17.623 INFO:tasks.ceph.osd.6.smithi150.stderr:2024-02-05T04:54:17.622+0000 7f6bea9f0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34015) UID: 0 2024-02-05T04:54:17.684 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:17.684 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:17.693 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-02-05T04:54:17.723 INFO:tasks.ceph.osd.5.smithi091.stderr:2024-02-05T04:54:17.721+0000 7f63983b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34277) UID: 0 2024-02-05T04:54:17.823 INFO:tasks.ceph.osd.0.smithi062.stderr:2024-02-05T04:54:17.822+0000 7fece8e03640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34688) UID: 0 2024-02-05T04:54:17.839 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:17.839 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:17.849 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-02-05T04:54:17.924 INFO:tasks.ceph.osd.1.smithi091.stderr:2024-02-05T04:54:17.923+0000 7ff9dc283640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34275) UID: 0 2024-02-05T04:54:17.955 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:17.955 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:17.964 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-02-05T04:54:18.024 INFO:tasks.ceph.osd.8.smithi062.stderr:2024-02-05T04:54:18.023+0000 7f9385bf2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 34693) UID: 0 2024-02-05T04:54:18.069 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:18.069 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:18.078 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-02-05T04:54:18.081 INFO:teuthology.orchestra.run.smithi062.stderr:nodeep-scrub is unset 2024-02-05T04:54:18.085 INFO:tasks.rados.rados.0.smithi196.stdout:370: done (0 left) 2024-02-05T04:54:18.085 INFO:tasks.rados.rados.0.smithi196.stdout:371: setattr oid 6 current snap is 30 2024-02-05T04:54:18.089 INFO:tasks.rados.rados.0.smithi196.stdout:372: read oid 41 snap 16 2024-02-05T04:54:18.089 INFO:tasks.rados.rados.0.smithi196.stdout:372: expect deleted 2024-02-05T04:54:18.089 INFO:tasks.rados.rados.0.smithi196.stdout:373: read oid 40 snap -1 2024-02-05T04:54:18.089 INFO:tasks.rados.rados.0.smithi196.stdout:373: expect deleted 2024-02-05T04:54:18.090 INFO:tasks.rados.rados.0.smithi196.stdout:374: snap_remove snap 20 2024-02-05T04:54:18.144 INFO:tasks.ceph.osd.12.smithi062.stderr:2024-02-05T04:54:18.143+0000 7f5e5847e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 34695) UID: 0 2024-02-05T04:54:18.200 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:18.200 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:18.209 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-02-05T04:54:18.226 INFO:tasks.ceph.osd.8.smithi062.stderr:2024-02-05T04:54:18.225+0000 7f9385bf2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 34693) UID: 0 2024-02-05T04:54:18.314 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:18.315 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:18.323 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-02-05T04:54:18.327 INFO:tasks.ceph.osd.11.smithi196.stderr:2024-02-05T04:54:18.326+0000 7f85a8d86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 34609) UID: 0 2024-02-05T04:54:18.428 INFO:tasks.ceph.osd.2.smithi150.stderr:2024-02-05T04:54:18.427+0000 7f7337bf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 34014) UID: 0 2024-02-05T04:54:18.429 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:18.429 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:18.437 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-02-05T04:54:18.529 INFO:tasks.ceph.osd.8.smithi062.stderr:2024-02-05T04:54:18.528+0000 7f9385bf2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 34693) UID: 0 2024-02-05T04:54:18.583 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:18.583 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:18.592 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-02-05T04:54:18.598 INFO:teuthology.orchestra.run.smithi091.stdout: marking collection for removal 2024-02-05T04:54:18.598 INFO:teuthology.orchestra.run.smithi091.stdout:setting '_remove' omap key 2024-02-05T04:54:18.598 INFO:teuthology.orchestra.run.smithi091.stdout:finish_remove_pgs 3.7s0_head removing 3.7s0 2024-02-05T04:54:18.599 INFO:teuthology.orchestra.run.smithi091.stdout:Remove successful 2024-02-05T04:54:18.630 INFO:tasks.ceph.osd.0.smithi062.stderr:2024-02-05T04:54:18.629+0000 7fece8e03640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34688) UID: 0 2024-02-05T04:54:18.696 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:18.697 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:18.705 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-02-05T04:54:18.731 INFO:tasks.ceph.osd.5.smithi091.stderr:2024-02-05T04:54:18.730+0000 7f63983b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34277) UID: 0 2024-02-05T04:54:18.811 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:18.811 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:18.820 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-02-05T04:54:18.865 INFO:tasks.ceph.osd.3.smithi196.stderr:2024-02-05T04:54:18.865+0000 7fc8817c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34605) UID: 0 2024-02-05T04:54:18.933 INFO:tasks.ceph.osd.2.smithi150.stderr:2024-02-05T04:54:18.933+0000 7f7337bf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 34014) UID: 0 2024-02-05T04:54:18.938 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:18.938 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:18.947 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-02-05T04:54:18.952 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 6 v 90 (ObjNum 33 snap 0 seq_num 3191112320) dirty exists 2024-02-05T04:54:18.952 INFO:tasks.rados.rados.0.smithi196.stdout:371: done (3 left) 2024-02-05T04:54:18.952 INFO:tasks.rados.rados.0.smithi196.stdout:372: done (2 left) 2024-02-05T04:54:18.952 INFO:tasks.rados.rados.0.smithi196.stdout:373: done (1 left) 2024-02-05T04:54:18.952 INFO:tasks.rados.rados.0.smithi196.stdout:374: done (0 left) 2024-02-05T04:54:18.952 INFO:tasks.rados.rados.0.smithi196.stdout:375: read oid 27 snap 27 2024-02-05T04:54:18.952 INFO:tasks.rados.rados.0.smithi196.stdout:375: expect (ObjNum 138 snap 26 seq_num 138) 2024-02-05T04:54:18.952 INFO:tasks.rados.rados.0.smithi196.stdout:376: read oid 6 snap -1 2024-02-05T04:54:18.952 INFO:tasks.rados.rados.0.smithi196.stdout:376: expect (ObjNum 0 snap 0 seq_num 0) 2024-02-05T04:54:18.952 INFO:tasks.rados.rados.0.smithi196.stdout:append oid 11 current snap is 30 2024-02-05T04:54:18.952 INFO:tasks.rados.rados.0.smithi196.stdout:377: seq_num 156 ranges {0=229376} 2024-02-05T04:54:18.953 INFO:tasks.rados.rados.0.smithi196.stdout:377: writing smithi19639362-11 from 0 to 229376 tid 1 2024-02-05T04:54:18.953 INFO:tasks.rados.rados.0.smithi196.stdout:378: read oid 5 snap -1 2024-02-05T04:54:18.953 INFO:tasks.rados.rados.0.smithi196.stdout:378: expect (ObjNum 4 snap 0 seq_num 4) 2024-02-05T04:54:18.953 INFO:tasks.rados.rados.0.smithi196.stdout:append oid 15 current snap is 30 2024-02-05T04:54:18.953 INFO:tasks.rados.rados.0.smithi196.stdout:379: seq_num 157 ranges {4210688=245760} 2024-02-05T04:54:18.981 INFO:tasks.rados.rados.0.smithi196.stdout:379: writing smithi19639362-15 from 4210688 to 4456448 tid 1 2024-02-05T04:54:18.981 INFO:tasks.rados.rados.0.smithi196.stdout:376: expect (ObjNum 0 snap 0 seq_num 0) 2024-02-05T04:54:18.984 INFO:tasks.rados.rados.0.smithi196.stdout:378: expect (ObjNum 4 snap 0 seq_num 4) 2024-02-05T04:54:18.987 INFO:tasks.ceph.osd.14.smithi150.stderr:2024-02-05T04:54:18.986+0000 7f5f62650640 -1 snap_mapper.get_next_objects_to_trim::New Clone-Objects were added to Snap 16 after trimming was started 2024-02-05T04:54:18.989 INFO:tasks.rados.rados.0.smithi196.stdout:377: finishing write tid 1 to smithi19639362-11 2024-02-05T04:54:18.989 INFO:tasks.rados.rados.0.smithi196.stdout:377: finishing write tid 2 to smithi19639362-11 2024-02-05T04:54:18.989 INFO:tasks.rados.rados.0.smithi196.stdout:376: done (4 left) 2024-02-05T04:54:18.989 INFO:tasks.rados.rados.0.smithi196.stdout:378: done (3 left) 2024-02-05T04:54:18.990 INFO:tasks.rados.rados.0.smithi196.stdout:380: delete oid 6 current snap is 30 2024-02-05T04:54:18.990 INFO:tasks.rados.rados.0.smithi196.stdout:377: finishing write tid 3 to smithi19639362-11 2024-02-05T04:54:18.990 INFO:tasks.rados.rados.0.smithi196.stdout:377: oid 11 updating version 0 to 42 2024-02-05T04:54:18.990 INFO:tasks.rados.rados.0.smithi196.stdout:377: oid 11 version 42 is already newer than 41 2024-02-05T04:54:18.990 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 11 v 42 (ObjNum 156 snap 30 seq_num 156) dirty exists 2024-02-05T04:54:18.990 INFO:tasks.rados.rados.0.smithi196.stdout:377: left oid 11 (ObjNum 156 snap 30 seq_num 156) 2024-02-05T04:54:18.993 INFO:tasks.rados.rados.0.smithi196.stdout:379: finishing write tid 1 to smithi19639362-15 2024-02-05T04:54:18.993 INFO:tasks.rados.rados.0.smithi196.stdout:379: finishing write tid 2 to smithi19639362-15 2024-02-05T04:54:18.994 INFO:tasks.rados.rados.0.smithi196.stdout:379: finishing write tid 3 to smithi19639362-15 2024-02-05T04:54:18.994 INFO:tasks.rados.rados.0.smithi196.stdout:379: oid 15 updating version 0 to 76 2024-02-05T04:54:18.994 INFO:tasks.rados.rados.0.smithi196.stdout:379: oid 15 version 76 is already newer than 75 2024-02-05T04:54:18.994 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 15 v 76 (ObjNum 157 snap 30 seq_num 157) dirty exists 2024-02-05T04:54:18.994 INFO:tasks.rados.rados.0.smithi196.stdout:379: left oid 15 (ObjNum 157 snap 30 seq_num 157) 2024-02-05T04:54:18.995 INFO:tasks.rados.rados.0.smithi196.stdout:377: done (3 left) 2024-02-05T04:54:18.995 INFO:tasks.rados.rados.0.smithi196.stdout:379: done (2 left) 2024-02-05T04:54:18.996 INFO:tasks.rados.rados.0.smithi196.stdout:380: done (1 left) 2024-02-05T04:54:18.996 INFO:tasks.rados.rados.0.smithi196.stdout:append oid 36 current snap is 30 2024-02-05T04:54:18.996 INFO:tasks.rados.rados.0.smithi196.stdout:381: seq_num 159 ranges {0=278528} 2024-02-05T04:54:18.997 INFO:tasks.rados.rados.0.smithi196.stdout:381: writing smithi19639362-36 from 0 to 278528 tid 1 2024-02-05T04:54:18.998 INFO:tasks.rados.rados.0.smithi196.stdout:382: rollback oid 30 current snap is 30 2024-02-05T04:54:18.998 INFO:tasks.rados.rados.0.smithi196.stdout:rollback oid 30 to 22 2024-02-05T04:54:18.998 INFO:tasks.rados.rados.0.smithi196.stdout:383: rollback oid 28 current snap is 30 2024-02-05T04:54:18.998 INFO:tasks.rados.rados.0.smithi196.stdout:rollback oid 28 to 16 2024-02-05T04:54:18.998 INFO:tasks.rados.rados.0.smithi196.stdout:append oid 39 current snap is 30 2024-02-05T04:54:18.998 INFO:tasks.rados.rados.0.smithi196.stdout:384: seq_num 160 ranges {2752512=294912} 2024-02-05T04:54:19.014 INFO:tasks.rados.rados.0.smithi196.stdout:384: writing smithi19639362-39 from 2752512 to 3047424 tid 1 2024-02-05T04:54:19.014 INFO:tasks.rados.rados.0.smithi196.stdout:append oid 38 current snap is 30 2024-02-05T04:54:19.015 INFO:tasks.rados.rados.0.smithi196.stdout:385: seq_num 161 ranges {0=311296} 2024-02-05T04:54:19.016 INFO:tasks.rados.rados.0.smithi196.stdout:385: writing smithi19639362-38 from 0 to 311296 tid 1 2024-02-05T04:54:19.016 INFO:tasks.rados.rados.0.smithi196.stdout:append oid 23 current snap is 30 2024-02-05T04:54:19.016 INFO:tasks.rados.rados.0.smithi196.stdout:386: seq_num 162 ranges {4702208=327680} 2024-02-05T04:54:19.034 INFO:tasks.ceph.osd.2.smithi150.stderr:2024-02-05T04:54:19.033+0000 7f7337bf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 34014) UID: 0 2024-02-05T04:54:19.044 INFO:tasks.rados.rados.0.smithi196.stdout:386: writing smithi19639362-23 from 4702208 to 5029888 tid 1 2024-02-05T04:54:19.044 INFO:tasks.rados.rados.0.smithi196.stdout:387: rollback oid 43 current snap is 30 2024-02-05T04:54:19.044 INFO:tasks.rados.rados.0.smithi196.stdout:rollback oid 43 to 28 2024-02-05T04:54:19.045 INFO:tasks.rados.rados.0.smithi196.stdout:375: expect (ObjNum 138 snap 26 seq_num 138) 2024-02-05T04:54:19.051 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T04:54:19.082 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:19.082 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:19.091 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-02-05T04:54:19.112 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-13 --journal-path /var/lib/ceph/osd/ceph-13/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.3.7s0.13 2024-02-05T04:54:19.135 INFO:tasks.ceph.osd.0.smithi062.stderr:2024-02-05T04:54:19.133+0000 7fece8e03640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34688) UID: 0 2024-02-05T04:54:19.178 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0: {} 2024-02-05T04:54:19.179 INFO:teuthology.orchestra.run.smithi062.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T04:54:19.199 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1: {} 2024-02-05T04:54:19.199 INFO:teuthology.orchestra.run.smithi062.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T04:54:19.216 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2: {} 2024-02-05T04:54:19.216 INFO:teuthology.orchestra.run.smithi062.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T04:54:19.234 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3: {} 2024-02-05T04:54:19.234 INFO:teuthology.orchestra.run.smithi062.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T04:54:19.235 INFO:tasks.ceph.osd.6.smithi150.stderr:2024-02-05T04:54:19.234+0000 7f6bea9f0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34015) UID: 0 2024-02-05T04:54:19.240 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T04:54:19.240 INFO:teuthology.orchestra.run.smithi196.stdout: "size": 20, 2024-02-05T04:54:19.240 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 600, 2024-02-05T04:54:19.240 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-02-05T04:54:19.240 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.240 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.ds1 57/31 ECSubWrite(tid=54, reqid=client.4634.0:525, at_version=57'34, trim_to=0'0, roll_forward_to=57'32))", 2024-02-05T04:54:19.240 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:48.959354+0000", 2024-02-05T04:54:19.240 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 30.279087893, 2024-02-05T04:54:19.241 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0066299729999999999, 2024-02-05T04:54:19.241 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:19.241 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:19.241 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:19.241 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.241 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:19.241 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.959354+0000", 2024-02-05T04:54:19.241 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:19.241 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.241 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.241 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:19.241 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.959354+0000", 2024-02-05T04:54:19.241 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:19.241 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.242 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.242 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:19.242 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.959355+0000", 2024-02-05T04:54:19.242 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5600000000000005e-07 2024-02-05T04:54:19.242 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.242 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.242 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:19.242 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.959379+0000", 2024-02-05T04:54:19.242 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3366e-05 2024-02-05T04:54:19.242 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.242 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.242 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:19.243 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.959391+0000", 2024-02-05T04:54:19.243 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2672e-05 2024-02-05T04:54:19.243 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.243 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.243 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:19.243 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.959398+0000", 2024-02-05T04:54:19.243 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3450000000000002e-06 2024-02-05T04:54:19.243 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.243 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.243 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:19.243 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.959467+0000", 2024-02-05T04:54:19.243 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9665000000000005e-05 2024-02-05T04:54:19.244 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.244 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.244 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:19.244 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.959477+0000", 2024-02-05T04:54:19.244 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6930000000000002e-06 2024-02-05T04:54:19.244 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.244 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.244 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:19.244 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.965940+0000", 2024-02-05T04:54:19.244 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.006463112 2024-02-05T04:54:19.244 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.244 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.244 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:19.245 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.965984+0000", 2024-02-05T04:54:19.245 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4363999999999997e-05 2024-02-05T04:54:19.245 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.245 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:19.245 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.245 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.245 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.245 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.3s6 57/31 ECSubWrite(tid=77, reqid=osd.4.0:76, at_version=0'0, trim_to=0'0, roll_forward_to=57'35))", 2024-02-05T04:54:19.245 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:48.960703+0000", 2024-02-05T04:54:19.245 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 30.277739560000001, 2024-02-05T04:54:19.245 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.014641594000000001, 2024-02-05T04:54:19.245 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:19.245 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:19.245 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:19.246 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.246 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:19.246 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.960703+0000", 2024-02-05T04:54:19.246 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:19.246 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.246 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.246 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:19.246 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.960703+0000", 2024-02-05T04:54:19.246 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:19.246 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.246 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.246 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:19.246 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.960704+0000", 2024-02-05T04:54:19.247 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1399999999999995e-07 2024-02-05T04:54:19.247 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.247 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.247 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:19.247 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.960715+0000", 2024-02-05T04:54:19.247 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1534e-05 2024-02-05T04:54:19.247 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.247 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.247 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:19.247 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.960734+0000", 2024-02-05T04:54:19.247 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9264999999999999e-05 2024-02-05T04:54:19.247 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.247 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.247 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:19.247 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.960744+0000", 2024-02-05T04:54:19.248 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.7610000000000002e-06 2024-02-05T04:54:19.248 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.248 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.248 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:19.248 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.960808+0000", 2024-02-05T04:54:19.248 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4257000000000005e-05 2024-02-05T04:54:19.248 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.248 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.248 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:19.248 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.960819+0000", 2024-02-05T04:54:19.248 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0158e-05 2024-02-05T04:54:19.248 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.248 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.249 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:19.249 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.975292+0000", 2024-02-05T04:54:19.249 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.014473803 2024-02-05T04:54:19.249 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.249 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.249 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:19.249 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.975344+0000", 2024-02-05T04:54:19.249 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1901999999999999e-05 2024-02-05T04:54:19.249 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.249 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:19.250 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.250 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.250 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.250 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.ds1 57/31 ECSubWrite(tid=55, reqid=client.4634.0:525, at_version=0'0, trim_to=0'0, roll_forward_to=57'34))", 2024-02-05T04:54:19.250 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:48.966615+0000", 2024-02-05T04:54:19.250 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 30.271827328000001, 2024-02-05T04:54:19.250 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0093695359999999995, 2024-02-05T04:54:19.250 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:19.250 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:19.250 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:19.250 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.251 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:19.251 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.966615+0000", 2024-02-05T04:54:19.251 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:19.251 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.251 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.251 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:19.251 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.966615+0000", 2024-02-05T04:54:19.251 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:19.251 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.251 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.251 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:19.251 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.966616+0000", 2024-02-05T04:54:19.251 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.7000000000000003e-07 2024-02-05T04:54:19.251 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.252 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.252 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:19.252 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.966638+0000", 2024-02-05T04:54:19.252 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1980999999999998e-05 2024-02-05T04:54:19.252 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.252 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.252 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:19.252 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.966656+0000", 2024-02-05T04:54:19.252 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8550999999999999e-05 2024-02-05T04:54:19.252 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.252 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.252 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:19.252 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.966674+0000", 2024-02-05T04:54:19.252 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7510999999999999e-05 2024-02-05T04:54:19.253 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.253 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.253 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:19.253 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.966728+0000", 2024-02-05T04:54:19.253 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4001e-05 2024-02-05T04:54:19.253 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.253 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.253 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:19.253 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.966737+0000", 2024-02-05T04:54:19.253 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.6759999999999996e-06 2024-02-05T04:54:19.253 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.253 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.253 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:19.253 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.975954+0000", 2024-02-05T04:54:19.253 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0092173919999999996 2024-02-05T04:54:19.253 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.254 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.254 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:19.254 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.975984+0000", 2024-02-05T04:54:19.254 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.0454e-05 2024-02-05T04:54:19.254 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.254 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:19.254 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.254 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.254 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.254 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.5s3 58/57 ECSubWrite(tid=57, reqid=osd.15.0:57, at_version=58'36, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:19.254 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:49.955234+0000", 2024-02-05T04:54:19.254 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 29.283208712, 2024-02-05T04:54:19.254 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0090458080000000007, 2024-02-05T04:54:19.254 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:19.254 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:19.255 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:19.255 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.255 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:19.255 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.955234+0000", 2024-02-05T04:54:19.255 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:19.255 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.255 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.255 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:19.255 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.955234+0000", 2024-02-05T04:54:19.255 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:19.255 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.255 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.255 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:19.255 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.955234+0000", 2024-02-05T04:54:19.255 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1699999999999997e-07 2024-02-05T04:54:19.256 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.256 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.256 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:19.256 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.955259+0000", 2024-02-05T04:54:19.256 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.4264e-05 2024-02-05T04:54:19.256 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.256 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.256 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:19.256 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.955275+0000", 2024-02-05T04:54:19.256 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5896000000000002e-05 2024-02-05T04:54:19.256 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.256 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.256 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:19.256 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.955285+0000", 2024-02-05T04:54:19.257 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0643e-05 2024-02-05T04:54:19.257 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.257 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.257 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:19.257 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.955388+0000", 2024-02-05T04:54:19.257 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010303599999999999 2024-02-05T04:54:19.257 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.257 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.257 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:19.257 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.955400+0000", 2024-02-05T04:54:19.257 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1582e-05 2024-02-05T04:54:19.257 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.257 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.257 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:19.257 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.964224+0000", 2024-02-05T04:54:19.258 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0088238989999999996 2024-02-05T04:54:19.258 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.258 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.258 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:19.258 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.964279+0000", 2024-02-05T04:54:19.258 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5571e-05 2024-02-05T04:54:19.258 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.258 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:19.258 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.258 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.258 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.258 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.5s3 58/57 ECSubWrite(tid=58, reqid=osd.15.0:58, at_version=58'38, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:19.258 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:49.956373+0000", 2024-02-05T04:54:19.258 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 29.282068989999999, 2024-02-05T04:54:19.258 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.010807686, 2024-02-05T04:54:19.259 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:19.259 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:19.259 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:19.259 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.259 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:19.259 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.956373+0000", 2024-02-05T04:54:19.259 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:19.259 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.259 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.259 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:19.259 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.956373+0000", 2024-02-05T04:54:19.259 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:19.259 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.260 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.260 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:19.260 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.956374+0000", 2024-02-05T04:54:19.260 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.9899999999999999e-07 2024-02-05T04:54:19.260 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.260 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.260 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:19.260 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.956401+0000", 2024-02-05T04:54:19.260 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.6404e-05 2024-02-05T04:54:19.260 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.260 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.260 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:19.260 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.956417+0000", 2024-02-05T04:54:19.260 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.613e-05 2024-02-05T04:54:19.261 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.261 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.261 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:19.261 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.956441+0000", 2024-02-05T04:54:19.261 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3836e-05 2024-02-05T04:54:19.261 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.261 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.261 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:19.261 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.956524+0000", 2024-02-05T04:54:19.261 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3902999999999999e-05 2024-02-05T04:54:19.261 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.261 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.261 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:19.261 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.956536+0000", 2024-02-05T04:54:19.262 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.118e-05 2024-02-05T04:54:19.262 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.262 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.262 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:19.262 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.967117+0000", 2024-02-05T04:54:19.262 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.010581620999999999 2024-02-05T04:54:19.262 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.262 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.262 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:19.262 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.967181+0000", 2024-02-05T04:54:19.262 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3713000000000005e-05 2024-02-05T04:54:19.262 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.262 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:19.262 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.263 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.263 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.263 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.5s3 58/57 ECSubWrite(tid=60, reqid=osd.15.0:60, at_version=58'40, trim_to=0'0, roll_forward_to=58'38))", 2024-02-05T04:54:19.263 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:49.969331+0000", 2024-02-05T04:54:19.263 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 29.269111293000002, 2024-02-05T04:54:19.263 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0094108439999999998, 2024-02-05T04:54:19.263 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:19.263 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:19.263 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:19.263 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.263 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:19.263 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.969331+0000", 2024-02-05T04:54:19.263 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:19.263 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.263 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.264 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:19.264 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.969331+0000", 2024-02-05T04:54:19.264 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:19.264 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.264 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.264 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:19.264 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.969332+0000", 2024-02-05T04:54:19.264 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2419999999999999e-06 2024-02-05T04:54:19.264 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.264 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.264 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:19.264 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.969362+0000", 2024-02-05T04:54:19.265 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.9696000000000002e-05 2024-02-05T04:54:19.265 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.265 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.265 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:19.265 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.969382+0000", 2024-02-05T04:54:19.265 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9945999999999998e-05 2024-02-05T04:54:19.265 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.265 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.265 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:19.265 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.969393+0000", 2024-02-05T04:54:19.265 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1015999999999999e-05 2024-02-05T04:54:19.265 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.265 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.265 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:19.266 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.969485+0000", 2024-02-05T04:54:19.266 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2377e-05 2024-02-05T04:54:19.266 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.266 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.266 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:19.266 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.969496+0000", 2024-02-05T04:54:19.266 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.065e-05 2024-02-05T04:54:19.266 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.266 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.266 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:19.266 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.978687+0000", 2024-02-05T04:54:19.266 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0091914130000000007 2024-02-05T04:54:19.266 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.266 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.266 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:19.266 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.978742+0000", 2024-02-05T04:54:19.267 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4503999999999999e-05 2024-02-05T04:54:19.267 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.267 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:19.267 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.267 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.267 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.267 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.ds1 58/31 ECSubWrite(tid=56, reqid=client.4634.0:526, at_version=58'35, trim_to=0'0, roll_forward_to=57'34))", 2024-02-05T04:54:19.267 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:49.970064+0000", 2024-02-05T04:54:19.267 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 29.268378528, 2024-02-05T04:54:19.267 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.011474428, 2024-02-05T04:54:19.267 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:19.267 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:19.267 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:19.267 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.267 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:19.268 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.970064+0000", 2024-02-05T04:54:19.268 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:19.268 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.268 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.268 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:19.268 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.970064+0000", 2024-02-05T04:54:19.268 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:19.268 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.268 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.268 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:19.268 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.970064+0000", 2024-02-05T04:54:19.268 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4099999999999998e-07 2024-02-05T04:54:19.268 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.269 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.269 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:19.269 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.970616+0000", 2024-02-05T04:54:19.269 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00055171500000000004 2024-02-05T04:54:19.269 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.269 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.269 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:19.269 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.970636+0000", 2024-02-05T04:54:19.269 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0244000000000001e-05 2024-02-05T04:54:19.269 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.269 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.269 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:19.269 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.970648+0000", 2024-02-05T04:54:19.270 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1678000000000001e-05 2024-02-05T04:54:19.270 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.270 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.270 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:19.270 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.970755+0000", 2024-02-05T04:54:19.270 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000107188 2024-02-05T04:54:19.270 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.270 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.270 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:19.270 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.970769+0000", 2024-02-05T04:54:19.270 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3346000000000001e-05 2024-02-05T04:54:19.270 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.270 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.270 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:19.271 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.981495+0000", 2024-02-05T04:54:19.271 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.010726352999999999 2024-02-05T04:54:19.271 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.271 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.271 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:19.271 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.981538+0000", 2024-02-05T04:54:19.271 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3163e-05 2024-02-05T04:54:19.271 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.271 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:19.271 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.271 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.271 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.271 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.12s5 68/67 ECSubWrite(tid=85, reqid=osd.2.0:85, at_version=68'57, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:19.271 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:59.961270+0000", 2024-02-05T04:54:19.272 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 19.277172611000001, 2024-02-05T04:54:19.272 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0062076639999999999, 2024-02-05T04:54:19.272 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:19.272 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:19.272 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:19.272 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.272 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:19.272 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.961270+0000", 2024-02-05T04:54:19.272 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:19.272 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.272 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.272 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:19.272 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.961270+0000", 2024-02-05T04:54:19.272 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:19.273 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.273 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.273 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:19.273 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.961270+0000", 2024-02-05T04:54:19.273 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3099999999999997e-07 2024-02-05T04:54:19.273 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.273 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.273 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:19.273 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.961288+0000", 2024-02-05T04:54:19.274 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7640000000000001e-05 2024-02-05T04:54:19.274 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.274 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.274 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:19.274 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.961300+0000", 2024-02-05T04:54:19.274 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2245999999999999e-05 2024-02-05T04:54:19.274 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.274 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.274 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:19.274 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.961309+0000", 2024-02-05T04:54:19.274 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.738e-06 2024-02-05T04:54:19.274 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.274 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.274 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:19.274 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.961373+0000", 2024-02-05T04:54:19.274 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3801999999999997e-05 2024-02-05T04:54:19.275 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.275 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.275 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:19.275 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.961383+0000", 2024-02-05T04:54:19.275 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0037999999999999e-05 2024-02-05T04:54:19.275 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.275 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.275 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:19.275 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.967435+0000", 2024-02-05T04:54:19.275 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0060525559999999997 2024-02-05T04:54:19.275 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.275 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.275 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:19.275 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:59.967477+0000", 2024-02-05T04:54:19.275 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2012999999999999e-05 2024-02-05T04:54:19.276 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.276 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:19.276 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.276 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.276 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.276 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.5s3 69/62 ECSubWrite(tid=71, reqid=client.4634.0:638, at_version=0'0, trim_to=0'0, roll_forward_to=69'50))", 2024-02-05T04:54:19.276 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:54:00.999685+0000", 2024-02-05T04:54:19.276 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 18.238757669000002, 2024-02-05T04:54:19.276 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0060564049999999999, 2024-02-05T04:54:19.276 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:19.276 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:19.276 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:19.276 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.276 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:19.277 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:00.999685+0000", 2024-02-05T04:54:19.277 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:19.277 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.277 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.277 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:19.277 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:00.999685+0000", 2024-02-05T04:54:19.277 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:19.277 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.277 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.277 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:19.277 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:00.999685+0000", 2024-02-05T04:54:19.277 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2799999999999995e-07 2024-02-05T04:54:19.277 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.277 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.277 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:19.278 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:00.999695+0000", 2024-02-05T04:54:19.278 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.9550000000000004e-06 2024-02-05T04:54:19.278 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.278 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.278 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:19.278 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:00.999708+0000", 2024-02-05T04:54:19.278 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3095000000000001e-05 2024-02-05T04:54:19.278 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.278 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.278 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:19.278 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:00.999719+0000", 2024-02-05T04:54:19.278 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0643e-05 2024-02-05T04:54:19.278 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.278 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.278 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:19.278 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:00.999773+0000", 2024-02-05T04:54:19.279 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3596e-05 2024-02-05T04:54:19.279 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.279 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.279 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:19.279 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:00.999781+0000", 2024-02-05T04:54:19.279 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.6049999999999998e-06 2024-02-05T04:54:19.279 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.279 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.279 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:19.279 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:01.005720+0000", 2024-02-05T04:54:19.279 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0059392409999999996 2024-02-05T04:54:19.279 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.279 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.279 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:19.279 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:01.005741+0000", 2024-02-05T04:54:19.280 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0542000000000001e-05 2024-02-05T04:54:19.280 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.280 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:19.280 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.280 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.280 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.280 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.2s2 69/62 ECSubWrite(tid=75, reqid=client.4634.0:641, at_version=69'55, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:19.280 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:54:01.014043+0000", 2024-02-05T04:54:19.280 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 18.224399049999999, 2024-02-05T04:54:19.280 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0081968869999999999, 2024-02-05T04:54:19.280 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:19.280 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:19.280 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:19.280 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.281 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:19.281 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:01.014043+0000", 2024-02-05T04:54:19.281 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:19.281 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.283 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.283 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:19.283 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:01.014043+0000", 2024-02-05T04:54:19.283 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:19.283 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.283 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.283 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:19.283 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:01.014043+0000", 2024-02-05T04:54:19.283 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.16e-07 2024-02-05T04:54:19.283 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.283 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.283 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:19.283 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:01.017527+0000", 2024-02-05T04:54:19.284 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0034836089999999999 2024-02-05T04:54:19.284 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.284 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.284 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:19.284 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:01.017541+0000", 2024-02-05T04:54:19.284 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3816e-05 2024-02-05T04:54:19.284 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.284 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.284 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:19.284 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:01.017549+0000", 2024-02-05T04:54:19.284 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8110000000000003e-06 2024-02-05T04:54:19.284 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.285 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.285 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:19.285 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:01.017615+0000", 2024-02-05T04:54:19.285 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6582999999999996e-05 2024-02-05T04:54:19.285 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.285 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.285 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:19.285 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:01.017626+0000", 2024-02-05T04:54:19.285 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1165000000000001e-05 2024-02-05T04:54:19.285 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.285 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.285 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:19.285 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:01.022186+0000", 2024-02-05T04:54:19.285 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0045597789999999999 2024-02-05T04:54:19.286 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.286 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.286 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:19.286 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:01.022240+0000", 2024-02-05T04:54:19.286 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3908e-05 2024-02-05T04:54:19.286 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.286 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:19.286 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.286 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.286 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.286 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.3s6 74/62 ECSubWrite(tid=105, reqid=osd.4.0:105, at_version=74'47, trim_to=0'0, roll_forward_to=72'45))", 2024-02-05T04:54:19.286 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:54:06.042188+0000", 2024-02-05T04:54:19.286 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 13.196253941, 2024-02-05T04:54:19.287 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0074863799999999999, 2024-02-05T04:54:19.287 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:19.287 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:19.287 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:19.287 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.287 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:19.287 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.042188+0000", 2024-02-05T04:54:19.287 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:19.287 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.287 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.287 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:19.287 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.042188+0000", 2024-02-05T04:54:19.287 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:19.287 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.287 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.288 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:19.288 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.042189+0000", 2024-02-05T04:54:19.288 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0100000000000005e-07 2024-02-05T04:54:19.288 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.288 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.288 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:19.288 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.042217+0000", 2024-02-05T04:54:19.288 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.7657e-05 2024-02-05T04:54:19.288 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.288 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.288 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:19.288 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.042226+0000", 2024-02-05T04:54:19.288 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5710000000000004e-06 2024-02-05T04:54:19.288 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.289 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.289 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:19.289 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.042234+0000", 2024-02-05T04:54:19.289 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9279999999999993e-06 2024-02-05T04:54:19.289 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.289 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.289 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:19.289 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.042323+0000", 2024-02-05T04:54:19.289 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8428000000000001e-05 2024-02-05T04:54:19.289 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.289 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.289 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:19.289 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.042339+0000", 2024-02-05T04:54:19.290 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5994e-05 2024-02-05T04:54:19.290 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.290 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.290 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:19.290 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.049625+0000", 2024-02-05T04:54:19.290 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0072861799999999997 2024-02-05T04:54:19.290 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.290 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.290 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:19.290 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.049675+0000", 2024-02-05T04:54:19.290 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0021000000000001e-05 2024-02-05T04:54:19.290 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.290 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:19.290 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.290 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.291 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.291 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.17s1 74/67 ECSubWrite(tid=4, reqid=osd.0.0:4, at_version=74'32, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:19.291 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:54:06.045938+0000", 2024-02-05T04:54:19.291 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 13.19250422, 2024-02-05T04:54:19.291 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0070776980000000003, 2024-02-05T04:54:19.291 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:19.291 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:19.291 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:19.291 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.291 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:19.291 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.045938+0000", 2024-02-05T04:54:19.291 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:19.291 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.291 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.291 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:19.291 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.045938+0000", 2024-02-05T04:54:19.292 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:19.292 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.292 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.292 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:19.292 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.045938+0000", 2024-02-05T04:54:19.292 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.27e-07 2024-02-05T04:54:19.292 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.292 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.293 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:19.293 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.045963+0000", 2024-02-05T04:54:19.293 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.4834999999999999e-05 2024-02-05T04:54:19.293 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.293 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.293 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:19.293 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.045973+0000", 2024-02-05T04:54:19.293 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0059e-05 2024-02-05T04:54:19.293 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.293 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.293 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:19.293 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.045982+0000", 2024-02-05T04:54:19.293 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.7860000000000002e-06 2024-02-05T04:54:19.293 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.294 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.294 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:19.294 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.046058+0000", 2024-02-05T04:54:19.294 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5958999999999996e-05 2024-02-05T04:54:19.294 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.294 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.294 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:19.294 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.046068+0000", 2024-02-05T04:54:19.294 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0407e-05 2024-02-05T04:54:19.294 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.294 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.294 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:19.294 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.052966+0000", 2024-02-05T04:54:19.294 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.006897668 2024-02-05T04:54:19.295 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.295 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.295 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:19.295 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.053016+0000", 2024-02-05T04:54:19.295 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9657000000000002e-05 2024-02-05T04:54:19.295 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.295 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:19.295 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.295 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.295 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.295 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWriteReply(3.14s0 74/67 ECSubWriteReply(tid=22, last_complete=74'83, committed=1, applied=1))", 2024-02-05T04:54:19.295 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:54:06.048543+0000", 2024-02-05T04:54:19.295 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 13.189898946, 2024-02-05T04:54:19.295 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0075604620000000004, 2024-02-05T04:54:19.296 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:19.296 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:19.296 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:19.296 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.296 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:19.296 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.048543+0000", 2024-02-05T04:54:19.296 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:19.296 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.296 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.296 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:19.296 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.048543+0000", 2024-02-05T04:54:19.296 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:19.296 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.297 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.297 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:19.297 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.048544+0000", 2024-02-05T04:54:19.297 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.06e-07 2024-02-05T04:54:19.297 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.297 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.297 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:19.297 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.048559+0000", 2024-02-05T04:54:19.297 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4912999999999999e-05 2024-02-05T04:54:19.297 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.297 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.297 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:19.297 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.048569+0000", 2024-02-05T04:54:19.297 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0186999999999999e-05 2024-02-05T04:54:19.298 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.298 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.298 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:19.298 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.048577+0000", 2024-02-05T04:54:19.298 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7689999999999996e-06 2024-02-05T04:54:19.298 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.298 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.298 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:19.298 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.056078+0000", 2024-02-05T04:54:19.298 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0075011210000000004 2024-02-05T04:54:19.298 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.298 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.298 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:19.298 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.056104+0000", 2024-02-05T04:54:19.299 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5965999999999998e-05 2024-02-05T04:54:19.299 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.299 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:19.299 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.299 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.299 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.299 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWriteReply(3.14s0 74/67 ECSubWriteReply(tid=22, last_complete=74'83, committed=1, applied=1))", 2024-02-05T04:54:19.299 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:54:06.049955+0000", 2024-02-05T04:54:19.299 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 13.188487185, 2024-02-05T04:54:19.299 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0061941080000000003, 2024-02-05T04:54:19.299 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:19.299 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:19.299 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:19.299 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.300 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:19.300 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.049955+0000", 2024-02-05T04:54:19.300 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:19.300 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.300 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.300 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:19.300 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.049955+0000", 2024-02-05T04:54:19.300 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:19.300 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.300 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.300 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:19.300 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.049956+0000", 2024-02-05T04:54:19.300 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.4e-07 2024-02-05T04:54:19.301 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.301 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.301 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:19.301 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.049963+0000", 2024-02-05T04:54:19.301 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.199e-06 2024-02-05T04:54:19.301 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.301 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.301 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:19.301 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.049976+0000", 2024-02-05T04:54:19.301 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2806000000000001e-05 2024-02-05T04:54:19.301 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.301 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.301 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:19.301 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.049984+0000", 2024-02-05T04:54:19.301 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8040000000000002e-06 2024-02-05T04:54:19.302 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.302 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.302 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:19.302 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.056132+0000", 2024-02-05T04:54:19.302 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0061485259999999996 2024-02-05T04:54:19.302 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.302 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.302 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:19.302 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.056149+0000", 2024-02-05T04:54:19.302 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6833e-05 2024-02-05T04:54:19.302 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.302 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:19.302 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.302 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.303 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.303 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.2s2 74/62 ECSubWrite(tid=89, reqid=osd.12.0:89, at_version=74'69, trim_to=0'0, roll_forward_to=72'67))", 2024-02-05T04:54:19.303 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:54:06.050676+0000", 2024-02-05T04:54:19.303 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 13.187765868, 2024-02-05T04:54:19.303 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0079223910000000008, 2024-02-05T04:54:19.303 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:19.304 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:19.304 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:19.304 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.304 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:19.304 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.050676+0000", 2024-02-05T04:54:19.304 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:19.304 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.304 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.304 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:19.304 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.050676+0000", 2024-02-05T04:54:19.304 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:19.304 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.304 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.305 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:19.305 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.050677+0000", 2024-02-05T04:54:19.305 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6899999999999996e-07 2024-02-05T04:54:19.305 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.305 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.305 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:19.305 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.050702+0000", 2024-02-05T04:54:19.305 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.4457e-05 2024-02-05T04:54:19.305 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.305 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.305 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:19.305 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.050717+0000", 2024-02-05T04:54:19.305 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4701e-05 2024-02-05T04:54:19.305 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.306 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.306 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:19.306 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.050726+0000", 2024-02-05T04:54:19.306 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8260000000000005e-06 2024-02-05T04:54:19.306 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.306 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.306 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:19.306 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.050807+0000", 2024-02-05T04:54:19.306 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0669999999999998e-05 2024-02-05T04:54:19.306 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.306 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.306 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:19.306 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.050817+0000", 2024-02-05T04:54:19.306 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0383999999999999e-05 2024-02-05T04:54:19.306 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.307 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.307 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:19.307 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.058551+0000", 2024-02-05T04:54:19.307 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0077333640000000004 2024-02-05T04:54:19.307 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.307 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.307 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:19.307 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.058599+0000", 2024-02-05T04:54:19.307 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8019999999999998e-05 2024-02-05T04:54:19.307 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.307 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:19.307 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.307 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.308 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.308 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.2s2 74/62 ECSubWrite(tid=90, reqid=osd.12.0:90, at_version=74'71, trim_to=0'0, roll_forward_to=72'67))", 2024-02-05T04:54:19.308 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:54:06.051210+0000", 2024-02-05T04:54:19.308 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 13.187231842999999, 2024-02-05T04:54:19.308 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.010150836, 2024-02-05T04:54:19.308 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:19.308 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:19.308 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:19.308 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.308 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:19.308 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.051210+0000", 2024-02-05T04:54:19.308 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:19.308 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.308 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.309 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:19.309 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.051210+0000", 2024-02-05T04:54:19.309 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:19.309 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.309 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.309 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:19.309 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.051211+0000", 2024-02-05T04:54:19.309 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.1299999999999999e-07 2024-02-05T04:54:19.309 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.309 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.309 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:19.309 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.051237+0000", 2024-02-05T04:54:19.309 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5746e-05 2024-02-05T04:54:19.309 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.309 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.310 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:19.310 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.051249+0000", 2024-02-05T04:54:19.310 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2398999999999999e-05 2024-02-05T04:54:19.310 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.310 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.310 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:19.310 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.051260+0000", 2024-02-05T04:54:19.310 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0473e-05 2024-02-05T04:54:19.310 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.310 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.310 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:19.310 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.053539+0000", 2024-02-05T04:54:19.310 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0022794960000000002 2024-02-05T04:54:19.310 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.310 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.310 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:19.311 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.053549+0000", 2024-02-05T04:54:19.311 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.7510000000000003e-06 2024-02-05T04:54:19.311 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.311 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.311 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:19.311 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.061320+0000", 2024-02-05T04:54:19.311 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0077706099999999998 2024-02-05T04:54:19.311 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.311 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.311 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:19.311 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.061361+0000", 2024-02-05T04:54:19.311 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1548000000000003e-05 2024-02-05T04:54:19.311 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.311 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:19.311 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.312 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.312 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.312 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.5s3 74/62 ECSubWrite(tid=79, reqid=osd.15.0:79, at_version=74'59, trim_to=0'0, roll_forward_to=72'57))", 2024-02-05T04:54:19.312 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:54:06.054480+0000", 2024-02-05T04:54:19.312 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 13.183962472999999, 2024-02-05T04:54:19.312 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0068285220000000001, 2024-02-05T04:54:19.312 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:19.312 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:19.312 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:19.312 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.312 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:19.312 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.054480+0000", 2024-02-05T04:54:19.312 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:19.313 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.313 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.313 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:19.313 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.054480+0000", 2024-02-05T04:54:19.313 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:19.313 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.313 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.313 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:19.313 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.054481+0000", 2024-02-05T04:54:19.313 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1310000000000001e-06 2024-02-05T04:54:19.313 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.313 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.313 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:19.313 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.054506+0000", 2024-02-05T04:54:19.313 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5406e-05 2024-02-05T04:54:19.314 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.314 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.314 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:19.314 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.054528+0000", 2024-02-05T04:54:19.314 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1211000000000001e-05 2024-02-05T04:54:19.314 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.314 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.314 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:19.314 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.054547+0000", 2024-02-05T04:54:19.314 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9548e-05 2024-02-05T04:54:19.314 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.314 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.314 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:19.314 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.054649+0000", 2024-02-05T04:54:19.314 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000102268 2024-02-05T04:54:19.315 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.315 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.315 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:19.315 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.054659+0000", 2024-02-05T04:54:19.315 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.008e-05 2024-02-05T04:54:19.315 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.315 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.315 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:19.315 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.061263+0000", 2024-02-05T04:54:19.315 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0066037329999999997 2024-02-05T04:54:19.315 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.315 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.315 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:19.315 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.061308+0000", 2024-02-05T04:54:19.315 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5145000000000002e-05 2024-02-05T04:54:19.316 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.316 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:19.316 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.316 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.316 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.316 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:793 3.14s0 3:2c242b83:::smithi19639362-9:head [stat,delete] snapc 1b=[1b,19,18,17,16,12,8,7] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-02-05T04:54:19.316 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:54:11.059753+0000", 2024-02-05T04:54:19.316 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 8.1786892990000002, 2024-02-05T04:54:19.316 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0086766699999999992, 2024-02-05T04:54:19.316 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:19.316 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:19.316 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:54:19.316 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:54:19.316 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:19.317 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 793 2024-02-05T04:54:19.317 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.317 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:19.317 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.317 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:19.317 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:11.059753+0000", 2024-02-05T04:54:19.317 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:19.317 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.317 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.317 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:19.317 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:11.059753+0000", 2024-02-05T04:54:19.317 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:19.317 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.317 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.317 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:19.318 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:11.059755+0000", 2024-02-05T04:54:19.318 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7430000000000001e-06 2024-02-05T04:54:19.318 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.318 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.318 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:19.318 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:11.059760+0000", 2024-02-05T04:54:19.318 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1610000000000002e-06 2024-02-05T04:54:19.318 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.318 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.318 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:19.318 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:11.059769+0000", 2024-02-05T04:54:19.318 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.285e-06 2024-02-05T04:54:19.318 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.318 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.318 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:19.318 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:11.059778+0000", 2024-02-05T04:54:19.319 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1190000000000003e-06 2024-02-05T04:54:19.319 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.319 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.319 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:19.319 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:11.059839+0000", 2024-02-05T04:54:19.319 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0878999999999998e-05 2024-02-05T04:54:19.319 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.319 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.319 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op must wait for map", 2024-02-05T04:54:19.319 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:11.059846+0000", 2024-02-05T04:54:19.319 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9840000000000004e-06 2024-02-05T04:54:19.319 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.319 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.319 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:19.319 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:11.063040+0000", 2024-02-05T04:54:19.320 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0031938420000000001 2024-02-05T04:54:19.320 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.320 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.320 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:54:19.320 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:11.063118+0000", 2024-02-05T04:54:19.320 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8027000000000004e-05 2024-02-05T04:54:19.320 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.320 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.320 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:19.320 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:11.064059+0000", 2024-02-05T04:54:19.320 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00094127900000000005 2024-02-05T04:54:19.320 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.320 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.320 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:19.320 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:11.065577+0000", 2024-02-05T04:54:19.321 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0015174019999999999 2024-02-05T04:54:19.321 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.321 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.321 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T04:54:19.321 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:11.068377+0000", 2024-02-05T04:54:19.321 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0028003709999999998 2024-02-05T04:54:19.321 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.321 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.321 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:19.321 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:11.068430+0000", 2024-02-05T04:54:19.321 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2577999999999997e-05 2024-02-05T04:54:19.321 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.321 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:19.321 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.321 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.322 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.322 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:850 3.14s0 3:2c15cb2e:::smithi19639362-6:head [stat,delete] snapc 1d=[1d,1c,19,18,17,16,12,8] ondisk+write+known_if_redirected+supports_pool_eio e83)", 2024-02-05T04:54:19.322 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:54:15.077366+0000", 2024-02-05T04:54:19.322 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 4.1610766950000002, 2024-02-05T04:54:19.322 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0202030150000001, 2024-02-05T04:54:19.322 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:19.322 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:19.322 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:54:19.322 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:54:19.322 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:19.322 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 850 2024-02-05T04:54:19.322 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.322 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:19.322 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.323 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:19.323 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:15.077366+0000", 2024-02-05T04:54:19.323 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:19.323 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.323 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.323 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:19.323 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:15.077366+0000", 2024-02-05T04:54:19.323 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:19.323 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.323 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.323 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:19.323 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:15.077367+0000", 2024-02-05T04:54:19.323 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8899999999999999e-06 2024-02-05T04:54:19.323 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.323 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.323 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:19.324 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:15.077375+0000", 2024-02-05T04:54:19.324 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.3830000000000003e-06 2024-02-05T04:54:19.324 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.324 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.324 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:19.324 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:15.077390+0000", 2024-02-05T04:54:19.324 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.482e-05 2024-02-05T04:54:19.324 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.324 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.324 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:19.324 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:15.077401+0000", 2024-02-05T04:54:19.324 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1637000000000001e-05 2024-02-05T04:54:19.324 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.325 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.325 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:19.325 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:15.077480+0000", 2024-02-05T04:54:19.325 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9103000000000005e-05 2024-02-05T04:54:19.325 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.325 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.325 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op must wait for map", 2024-02-05T04:54:19.325 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:15.077490+0000", 2024-02-05T04:54:19.325 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.0939999999999997e-06 2024-02-05T04:54:19.325 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.325 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.325 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:19.325 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:15.080589+0000", 2024-02-05T04:54:19.326 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.003099171 2024-02-05T04:54:19.326 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.326 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.326 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T04:54:19.326 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:15.080593+0000", 2024-02-05T04:54:19.326 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1459999999999998e-06 2024-02-05T04:54:19.326 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.326 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.326 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:19.326 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.087570+0000", 2024-02-05T04:54:19.326 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.006977247 2024-02-05T04:54:19.326 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.326 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.327 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:54:19.327 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.087871+0000", 2024-02-05T04:54:19.327 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00030117000000000002 2024-02-05T04:54:19.327 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.327 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.327 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:19.327 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.088959+0000", 2024-02-05T04:54:19.327 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001088173 2024-02-05T04:54:19.327 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.327 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.327 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:19.327 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.093574+0000", 2024-02-05T04:54:19.327 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0046144080000000004 2024-02-05T04:54:19.327 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.327 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.328 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T04:54:19.328 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.097183+0000", 2024-02-05T04:54:19.328 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0036090739999999999 2024-02-05T04:54:19.328 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.328 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.328 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:19.328 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.097569+0000", 2024-02-05T04:54:19.328 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00038569899999999998 2024-02-05T04:54:19.328 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.328 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:19.328 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.328 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.329 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.329 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.3s6 84/62 ECSubWrite(tid=123, reqid=client.4634.0:847, at_version=84'61, trim_to=0'0, roll_forward_to=83'59))", 2024-02-05T04:54:19.329 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:54:16.116614+0000", 2024-02-05T04:54:19.329 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 3.1218280859999998, 2024-02-05T04:54:19.329 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0073347739999999996, 2024-02-05T04:54:19.329 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:19.329 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:19.329 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:19.329 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.329 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:19.330 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.116614+0000", 2024-02-05T04:54:19.330 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:19.330 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.330 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.330 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:19.330 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.116614+0000", 2024-02-05T04:54:19.330 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:19.330 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.330 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.330 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:19.330 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.116615+0000", 2024-02-05T04:54:19.330 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.39e-07 2024-02-05T04:54:19.330 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.331 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.331 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:19.331 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.120742+0000", 2024-02-05T04:54:19.331 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0041273849999999999 2024-02-05T04:54:19.331 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.331 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.331 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:19.331 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.120757+0000", 2024-02-05T04:54:19.331 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4626e-05 2024-02-05T04:54:19.331 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.331 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.331 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:19.331 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.120767+0000", 2024-02-05T04:54:19.331 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.048e-05 2024-02-05T04:54:19.332 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.332 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.332 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:19.332 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.120864+0000", 2024-02-05T04:54:19.332 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6488000000000002e-05 2024-02-05T04:54:19.332 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.332 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.332 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:19.332 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.120875+0000", 2024-02-05T04:54:19.332 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1139e-05 2024-02-05T04:54:19.333 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.333 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.333 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:19.333 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.123871+0000", 2024-02-05T04:54:19.333 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0029964480000000001 2024-02-05T04:54:19.333 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.333 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.333 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:19.333 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.123949+0000", 2024-02-05T04:54:19.333 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7868999999999996e-05 2024-02-05T04:54:19.333 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.333 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:19.333 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.333 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.333 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:19.334 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T04:54:19.334 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4: {} 2024-02-05T04:54:19.335 INFO:teuthology.orchestra.run.smithi062.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T04:54:19.336 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5: {} 2024-02-05T04:54:19.336 INFO:teuthology.orchestra.run.smithi062.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T04:54:19.336 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-02-05T04:54:19.337 INFO:teuthology.orchestra.run.smithi062.stdout:osd.6: {} 2024-02-05T04:54:19.338 INFO:teuthology.orchestra.run.smithi062.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T04:54:19.338 INFO:teuthology.orchestra.run.smithi062.stdout:osd.7: {} 2024-02-05T04:54:19.339 INFO:teuthology.orchestra.run.smithi062.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T04:54:19.339 INFO:teuthology.orchestra.run.smithi062.stdout:osd.8: {} 2024-02-05T04:54:19.339 INFO:teuthology.orchestra.run.smithi062.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-05T04:54:19.341 INFO:tasks.rados.rados.0.smithi196.stdout:383: finishing rollback tid 0 to smithi19639362-28 2024-02-05T04:54:19.341 INFO:tasks.rados.rados.0.smithi196.stdout:382: finishing rollback tid 0 to smithi19639362-30 2024-02-05T04:54:19.341 INFO:tasks.rados.rados.0.smithi196.stdout:381: finishing write tid 1 to smithi19639362-36 2024-02-05T04:54:19.341 INFO:tasks.rados.rados.0.smithi196.stdout:382: finishing rollback tid 1 to smithi19639362-30 2024-02-05T04:54:19.341 INFO:tasks.rados.rados.0.smithi196.stdout:381: finishing write tid 2 to smithi19639362-36 2024-02-05T04:54:19.341 INFO:tasks.rados.rados.0.smithi196.stdout:383: finishing rollback tid 1 to smithi19639362-28 2024-02-05T04:54:19.341 INFO:tasks.rados.rados.0.smithi196.stdout:375: done (7 left) 2024-02-05T04:54:19.341 INFO:tasks.rados.rados.0.smithi196.stdout:388: rmattr oid 33 current snap is 30 2024-02-05T04:54:19.341 INFO:tasks.rados.rados.0.smithi196.stdout:388: done (7 left) 2024-02-05T04:54:19.342 INFO:tasks.rados.rados.0.smithi196.stdout:389: rollback oid 2 current snap is 30 2024-02-05T04:54:19.342 INFO:tasks.rados.rados.0.smithi196.stdout:rollback oid 2 to 29 2024-02-05T04:54:19.342 INFO:tasks.rados.rados.0.smithi196.stdout:382: finishing rollback tid 2 to smithi19639362-30 2024-02-05T04:54:19.342 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 30 v 43 (ObjNum 60 snap 2 seq_num 60) dirty exists 2024-02-05T04:54:19.342 INFO:tasks.rados.rados.0.smithi196.stdout:381: finishing write tid 3 to smithi19639362-36 2024-02-05T04:54:19.342 INFO:tasks.rados.rados.0.smithi196.stdout:381: oid 36 updating version 0 to 129 2024-02-05T04:54:19.342 INFO:tasks.rados.rados.0.smithi196.stdout:381: oid 36 updating version 129 to 130 2024-02-05T04:54:19.342 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 36 v 130 (ObjNum 159 snap 30 seq_num 159) dirty exists 2024-02-05T04:54:19.342 INFO:tasks.rados.rados.0.smithi196.stdout:381: left oid 36 (ObjNum 159 snap 30 seq_num 159) 2024-02-05T04:54:19.342 INFO:tasks.rados.rados.0.smithi196.stdout:384: finishing write tid 1 to smithi19639362-39 2024-02-05T04:54:19.342 INFO:tasks.rados.rados.0.smithi196.stdout:384: finishing write tid 2 to smithi19639362-39 2024-02-05T04:54:19.342 INFO:tasks.rados.rados.0.smithi196.stdout:385: finishing write tid 1 to smithi19639362-38 2024-02-05T04:54:19.342 INFO:tasks.rados.rados.0.smithi196.stdout:385: finishing write tid 2 to smithi19639362-38 2024-02-05T04:54:19.342 INFO:tasks.rados.rados.0.smithi196.stdout:384: finishing write tid 3 to smithi19639362-39 2024-02-05T04:54:19.343 INFO:tasks.rados.rados.0.smithi196.stdout:384: oid 39 updating version 0 to 77 2024-02-05T04:54:19.343 INFO:tasks.rados.rados.0.smithi196.stdout:384: oid 39 updating version 77 to 78 2024-02-05T04:54:19.343 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 39 v 78 (ObjNum 160 snap 30 seq_num 160) dirty exists 2024-02-05T04:54:19.343 INFO:tasks.rados.rados.0.smithi196.stdout:384: left oid 39 (ObjNum 160 snap 30 seq_num 160) 2024-02-05T04:54:19.343 INFO:tasks.rados.rados.0.smithi196.stdout:385: finishing write tid 3 to smithi19639362-38 2024-02-05T04:54:19.343 INFO:tasks.rados.rados.0.smithi196.stdout:385: oid 38 updating version 0 to 81 2024-02-05T04:54:19.343 INFO:tasks.rados.rados.0.smithi196.stdout:385: oid 38 version 81 is already newer than 80 2024-02-05T04:54:19.343 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 38 v 81 (ObjNum 161 snap 30 seq_num 161) dirty exists 2024-02-05T04:54:19.343 INFO:tasks.rados.rados.0.smithi196.stdout:385: left oid 38 (ObjNum 161 snap 30 seq_num 161) 2024-02-05T04:54:19.343 INFO:tasks.rados.rados.0.smithi196.stdout:387: finishing rollback tid 0 to smithi19639362-43 2024-02-05T04:54:19.343 INFO:tasks.rados.rados.0.smithi196.stdout:381: done (7 left) 2024-02-05T04:54:19.343 INFO:tasks.rados.rados.0.smithi196.stdout:382: done (6 left) 2024-02-05T04:54:19.343 INFO:tasks.rados.rados.0.smithi196.stdout:384: done (5 left) 2024-02-05T04:54:19.343 INFO:tasks.rados.rados.0.smithi196.stdout:385: done (4 left) 2024-02-05T04:54:19.343 INFO:tasks.rados.rados.0.smithi196.stdout:390: rmattr oid 47 current snap is 30 2024-02-05T04:54:19.343 INFO:tasks.rados.rados.0.smithi196.stdout:390: done (4 left) 2024-02-05T04:54:19.344 INFO:tasks.rados.rados.0.smithi196.stdout:391: rollback oid 36 current snap is 30 2024-02-05T04:54:19.344 INFO:tasks.rados.rados.0.smithi196.stdout:rollback oid 36 to 27 2024-02-05T04:54:19.344 INFO:tasks.rados.rados.0.smithi196.stdout:387: finishing rollback tid 1 to smithi19639362-43 2024-02-05T04:54:19.344 INFO:tasks.rados.rados.0.smithi196.stdout:386: finishing write tid 1 to smithi19639362-23 2024-02-05T04:54:19.344 INFO:tasks.rados.rados.0.smithi196.stdout:386: finishing write tid 2 to smithi19639362-23 2024-02-05T04:54:19.344 INFO:tasks.rados.rados.0.smithi196.stdout:386: finishing write tid 3 to smithi19639362-23 2024-02-05T04:54:19.344 INFO:tasks.rados.rados.0.smithi196.stdout:386: oid 23 updating version 0 to 92 2024-02-05T04:54:19.344 INFO:tasks.rados.rados.0.smithi196.stdout:386: oid 23 updating version 92 to 93 2024-02-05T04:54:19.344 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 23 v 93 (ObjNum 162 snap 30 seq_num 162) dirty exists 2024-02-05T04:54:19.344 INFO:tasks.rados.rados.0.smithi196.stdout:386: left oid 23 (ObjNum 162 snap 30 seq_num 162) 2024-02-05T04:54:19.344 INFO:tasks.rados.rados.0.smithi196.stdout:387: finishing rollback tid 2 to smithi19639362-43 2024-02-05T04:54:19.344 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 43 v 165 (ObjNum 129 snap 25 seq_num 129) dirty exists 2024-02-05T04:54:19.344 INFO:tasks.rados.rados.0.smithi196.stdout:386: done (4 left) 2024-02-05T04:54:19.345 INFO:tasks.rados.rados.0.smithi196.stdout:387: done (3 left) 2024-02-05T04:54:19.345 INFO:tasks.rados.rados.0.smithi196.stdout:append oid 48 current snap is 30 2024-02-05T04:54:19.345 INFO:tasks.rados.rados.0.smithi196.stdout:383: finishing rollback tid 2 to smithi19639362-28 2024-02-05T04:54:19.345 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 28 v 57 (ObjNum 60 snap 2 seq_num 60) dirty exists 2024-02-05T04:54:19.345 INFO:tasks.rados.rados.0.smithi196.stdout:392: seq_num 163 ranges {1277952=344064} 2024-02-05T04:54:19.345 INFO:tasks.rados.rados.0.smithi196.stdout:392: writing smithi19639362-48 from 1277952 to 1622016 tid 1 2024-02-05T04:54:19.345 INFO:tasks.rados.rados.0.smithi196.stdout:383: done (3 left) 2024-02-05T04:54:19.345 INFO:tasks.rados.rados.0.smithi196.stdout:393: rollback oid 14 current snap is 30 2024-02-05T04:54:19.345 INFO:tasks.rados.rados.0.smithi196.stdout:rollback oid 14 to 21 2024-02-05T04:54:19.345 INFO:tasks.rados.rados.0.smithi196.stdout:391: finishing rollback tid 0 to smithi19639362-36 2024-02-05T04:54:19.345 INFO:tasks.rados.rados.0.smithi196.stdout:389: finishing rollback tid 0 to smithi19639362-2 2024-02-05T04:54:19.345 INFO:tasks.rados.rados.0.smithi196.stdout:391: finishing rollback tid 1 to smithi19639362-36 2024-02-05T04:54:19.345 INFO:tasks.rados.rados.0.smithi196.stdout:391: finishing rollback tid 2 to smithi19639362-36 2024-02-05T04:54:19.345 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 36 v 133 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-02-05T04:54:19.345 INFO:tasks.rados.rados.0.smithi196.stdout:389: finishing rollback tid 1 to smithi19639362-2 2024-02-05T04:54:19.346 INFO:tasks.rados.rados.0.smithi196.stdout:389: finishing rollback tid 2 to smithi19639362-2 2024-02-05T04:54:19.346 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 2 v 169 (ObjNum 155 snap 28 seq_num 155) dirty exists 2024-02-05T04:54:19.346 INFO:tasks.rados.rados.0.smithi196.stdout:389: done (3 left) 2024-02-05T04:54:19.346 INFO:tasks.rados.rados.0.smithi196.stdout:391: done (2 left) 2024-02-05T04:54:19.346 INFO:tasks.rados.rados.0.smithi196.stdout:394: snap_remove snap 27 2024-02-05T04:54:19.347 INFO:teuthology.orchestra.run.smithi062.stdout:osd.9: {} 2024-02-05T04:54:19.347 INFO:teuthology.orchestra.run.smithi062.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-05T04:54:19.347 INFO:tasks.ceph.osd.8.smithi062.stderr:2024-02-05T04:54:19.346+0000 7f9385bf2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 34693) UID: 0 2024-02-05T04:54:19.359 INFO:teuthology.orchestra.run.smithi062.stdout:osd.10: {} 2024-02-05T04:54:19.359 INFO:teuthology.orchestra.run.smithi062.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-05T04:54:19.376 INFO:teuthology.orchestra.run.smithi062.stdout:osd.11: {} 2024-02-05T04:54:19.376 INFO:teuthology.orchestra.run.smithi062.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-05T04:54:19.393 INFO:teuthology.orchestra.run.smithi062.stdout:osd.12: {} 2024-02-05T04:54:19.393 INFO:teuthology.orchestra.run.smithi062.stderr:osd.12: osd_enable_op_tracker = '' 2024-02-05T04:54:19.393 INFO:teuthology.orchestra.run.smithi062.stderr:Error ENXIO: problem getting command descriptions from osd.13 2024-02-05T04:54:19.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-02-05T04:54:19.410+0000 7fc5d58cf640 0 -- 172.21.15.62:0/206083422 >> [v2:172.21.15.150:6812/2102158830,v1:172.21.15.150:6814/2102158830] conn(0x7fc5d021a240 msgr2=0x7fc5d021a6d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-02-05T04:54:19.412 INFO:teuthology.orchestra.run.smithi062.stdout:osd.14: {} 2024-02-05T04:54:19.430 INFO:teuthology.orchestra.run.smithi062.stdout:osd.15: {} 2024-02-05T04:54:19.431 INFO:teuthology.orchestra.run.smithi062.stderr:osd.15: osd_enable_op_tracker = '' 2024-02-05T04:54:19.434 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-05T04:54:19.434 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-05T04:54:19.448 INFO:tasks.ceph.osd.2.smithi150.stderr:2024-02-05T04:54:19.447+0000 7f7337bf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 34014) UID: 0 2024-02-05T04:54:19.453 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T04:54:19.453 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-02-05T04:54:19.453 INFO:teuthology.orchestra.run.smithi196.stdout: "num_ops": 0 2024-02-05T04:54:19.453 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T04:54:19.462 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-02-05T04:54:19.549 INFO:tasks.ceph.osd.9.smithi091.stderr:2024-02-05T04:54:19.548+0000 7efc18827640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 34279) UID: 0 2024-02-05T04:54:19.569 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T04:54:19.569 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-02-05T04:54:19.569 INFO:teuthology.orchestra.run.smithi196.stdout: "complaint_time": 30, 2024-02-05T04:54:19.569 INFO:teuthology.orchestra.run.smithi196.stdout: "num_blocked_ops": 0 2024-02-05T04:54:19.569 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T04:54:19.577 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-02-05T04:54:19.650 INFO:tasks.ceph.osd.8.smithi062.stderr:2024-02-05T04:54:19.649+0000 7f9385bf2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 34693) UID: 0 2024-02-05T04:54:19.685 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T04:54:19.685 INFO:teuthology.orchestra.run.smithi196.stdout: "size": 20, 2024-02-05T04:54:19.685 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 600, 2024-02-05T04:54:19.685 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-02-05T04:54:19.685 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.685 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:225 3.ds0 3:b4e8e82c:::smithi19639362-25:head [copy-from ver 6] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:54:19.685 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:29.135146+0000", 2024-02-05T04:54:19.685 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 50.548048397000002, 2024-02-05T04:54:19.685 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.016925534999999998, 2024-02-05T04:54:19.685 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:19.685 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:19.686 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:54:19.686 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:54:19.686 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:19.686 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 225 2024-02-05T04:54:19.686 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.686 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:19.686 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.686 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:19.686 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135146+0000", 2024-02-05T04:54:19.686 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:19.686 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.686 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.686 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:19.686 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135146+0000", 2024-02-05T04:54:19.687 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:19.687 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.687 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.687 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:19.687 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135148+0000", 2024-02-05T04:54:19.687 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5170000000000002e-06 2024-02-05T04:54:19.687 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.687 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.687 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:19.687 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135158+0000", 2024-02-05T04:54:19.687 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.3819999999999993e-06 2024-02-05T04:54:19.687 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.687 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.687 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:19.687 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135177+0000", 2024-02-05T04:54:19.688 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9513000000000001e-05 2024-02-05T04:54:19.688 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.688 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.688 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:19.688 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135187+0000", 2024-02-05T04:54:19.688 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0054000000000001e-05 2024-02-05T04:54:19.688 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.688 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.688 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:19.688 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135292+0000", 2024-02-05T04:54:19.689 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000104651 2024-02-05T04:54:19.689 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.689 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.689 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:54:19.689 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135405+0000", 2024-02-05T04:54:19.689 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011277099999999999 2024-02-05T04:54:19.689 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.689 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.689 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:19.690 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.146446+0000", 2024-02-05T04:54:19.690 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.011041535 2024-02-05T04:54:19.690 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.690 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.690 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:19.690 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.148233+0000", 2024-02-05T04:54:19.690 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001787089 2024-02-05T04:54:19.690 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.690 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.690 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T04:54:19.690 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.151960+0000", 2024-02-05T04:54:19.690 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0037264720000000002 2024-02-05T04:54:19.690 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.690 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.690 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:19.691 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.152071+0000", 2024-02-05T04:54:19.691 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000111551 2024-02-05T04:54:19.691 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.691 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:19.691 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.691 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.691 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.691 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:226 3.ds0 3:b4e8e82c:::smithi19639362-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:54:19.691 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:29.135222+0000", 2024-02-05T04:54:19.691 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 50.547971877000002, 2024-02-05T04:54:19.692 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.012833986, 2024-02-05T04:54:19.692 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:19.692 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-02-05T04:54:19.692 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:54:19.692 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:54:19.692 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:19.692 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 226 2024-02-05T04:54:19.692 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.692 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:19.692 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.692 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:19.695 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135222+0000", 2024-02-05T04:54:19.695 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:19.695 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.695 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.695 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:19.695 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135222+0000", 2024-02-05T04:54:19.695 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:19.695 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.696 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.696 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:19.696 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135223+0000", 2024-02-05T04:54:19.696 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4099999999999998e-07 2024-02-05T04:54:19.696 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.696 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.696 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:19.696 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135227+0000", 2024-02-05T04:54:19.696 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8399999999999997e-06 2024-02-05T04:54:19.696 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.696 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.696 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:19.696 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135237+0000", 2024-02-05T04:54:19.696 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8290000000000003e-06 2024-02-05T04:54:19.697 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.697 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.697 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:19.697 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135244+0000", 2024-02-05T04:54:19.697 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4989999999999997e-06 2024-02-05T04:54:19.697 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.697 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.697 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:19.697 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135624+0000", 2024-02-05T04:54:19.697 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00037992100000000002 2024-02-05T04:54:19.697 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.697 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.697 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for blocked object", 2024-02-05T04:54:19.697 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.135672+0000", 2024-02-05T04:54:19.697 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7861999999999997e-05 2024-02-05T04:54:19.697 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.697 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.698 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:19.698 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.147862+0000", 2024-02-05T04:54:19.698 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.012189847 2024-02-05T04:54:19.698 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.698 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.698 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:54:19.698 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.147947+0000", 2024-02-05T04:54:19.698 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4862000000000002e-05 2024-02-05T04:54:19.698 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.698 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.698 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:19.698 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:29.148056+0000", 2024-02-05T04:54:19.698 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000109585 2024-02-05T04:54:19.698 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.698 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:19.699 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.699 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.699 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.699 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:389 3.ds0 3:b4e8e82c:::smithi19639362-25:head [read 0~638976,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-02-05T04:54:19.699 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:34.171905+0000", 2024-02-05T04:54:19.699 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 45.511288977, 2024-02-05T04:54:19.699 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0072757830000000001, 2024-02-05T04:54:19.699 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:19.699 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-02-05T04:54:19.699 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:54:19.699 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:54:19.699 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:19.699 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 389 2024-02-05T04:54:19.699 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.699 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:19.699 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.700 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:19.700 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.171905+0000", 2024-02-05T04:54:19.700 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:19.700 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.700 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.700 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:19.700 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.171905+0000", 2024-02-05T04:54:19.700 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:19.700 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.700 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.700 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:19.700 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.171906+0000", 2024-02-05T04:54:19.700 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2499999999999995e-07 2024-02-05T04:54:19.700 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.701 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.701 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:19.701 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.171910+0000", 2024-02-05T04:54:19.701 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4490000000000001e-06 2024-02-05T04:54:19.701 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.701 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.701 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:19.701 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.171918+0000", 2024-02-05T04:54:19.701 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.2449999999999998e-06 2024-02-05T04:54:19.701 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.701 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.701 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:19.701 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.171924+0000", 2024-02-05T04:54:19.701 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6350000000000003e-06 2024-02-05T04:54:19.701 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.702 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.702 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:19.702 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.172022+0000", 2024-02-05T04:54:19.702 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8004000000000006e-05 2024-02-05T04:54:19.702 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.702 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.702 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting_for_map not empty", 2024-02-05T04:54:19.702 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.172026+0000", 2024-02-05T04:54:19.702 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8500000000000004e-06 2024-02-05T04:54:19.702 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.703 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.703 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:19.703 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.177805+0000", 2024-02-05T04:54:19.703 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0057789419999999996 2024-02-05T04:54:19.703 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.703 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.703 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:54:19.703 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.177900+0000", 2024-02-05T04:54:19.703 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5529e-05 2024-02-05T04:54:19.703 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.703 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.703 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:19.703 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:34.179181+0000", 2024-02-05T04:54:19.703 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0012806040000000001 2024-02-05T04:54:19.704 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.704 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:19.704 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.704 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.704 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.704 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.6s5 43/31 ECSubWrite(tid=138, reqid=osd.13.0:138, at_version=43'24, trim_to=0'0, roll_forward_to=40'22))", 2024-02-05T04:54:19.704 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:35.172477+0000", 2024-02-05T04:54:19.704 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 44.510717274000001, 2024-02-05T04:54:19.704 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0079411910000000002, 2024-02-05T04:54:19.704 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:19.704 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:19.704 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:19.704 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.705 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:19.705 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.172477+0000", 2024-02-05T04:54:19.705 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:19.705 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.705 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.705 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:19.705 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.172477+0000", 2024-02-05T04:54:19.705 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:19.705 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.705 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.705 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:19.705 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.172478+0000", 2024-02-05T04:54:19.705 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9299999999999997e-07 2024-02-05T04:54:19.705 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.706 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.706 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:19.706 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.172502+0000", 2024-02-05T04:54:19.706 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.4071e-05 2024-02-05T04:54:19.706 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.706 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.706 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:19.706 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.172525+0000", 2024-02-05T04:54:19.706 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3666e-05 2024-02-05T04:54:19.707 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.707 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.707 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:19.707 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.172534+0000", 2024-02-05T04:54:19.707 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.7749999999999998e-06 2024-02-05T04:54:19.707 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.707 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.707 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:19.707 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.172612+0000", 2024-02-05T04:54:19.707 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7486999999999999e-05 2024-02-05T04:54:19.707 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.707 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.707 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:19.708 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.172622+0000", 2024-02-05T04:54:19.708 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0040999999999999e-05 2024-02-05T04:54:19.708 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.708 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.708 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:19.708 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.180380+0000", 2024-02-05T04:54:19.708 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0077582739999999999 2024-02-05T04:54:19.708 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.708 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.708 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:19.708 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.180418+0000", 2024-02-05T04:54:19.708 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8183999999999999e-05 2024-02-05T04:54:19.708 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.709 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:19.709 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.709 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.709 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.709 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:401 3.ds0 3:b9c88094:::smithi19639362-50:head [delete] snapc 7=[7,6,4] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e43)", 2024-02-05T04:54:19.709 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:35.182518+0000", 2024-02-05T04:54:19.709 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 44.500676134000003, 2024-02-05T04:54:19.709 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0087143140000000008, 2024-02-05T04:54:19.709 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:19.709 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:19.709 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:54:19.709 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:54:19.709 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:19.709 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 401 2024-02-05T04:54:19.709 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.709 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:19.710 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.710 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:19.710 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.182518+0000", 2024-02-05T04:54:19.710 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:19.710 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.710 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.710 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:19.710 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.182518+0000", 2024-02-05T04:54:19.710 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:19.710 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.710 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.710 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:19.710 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.182519+0000", 2024-02-05T04:54:19.710 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.313e-06 2024-02-05T04:54:19.711 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.711 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.711 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:19.711 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.182524+0000", 2024-02-05T04:54:19.711 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9150000000000004e-06 2024-02-05T04:54:19.711 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.711 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.711 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:19.711 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.182544+0000", 2024-02-05T04:54:19.711 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9327999999999999e-05 2024-02-05T04:54:19.711 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.711 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.711 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:19.711 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.182572+0000", 2024-02-05T04:54:19.711 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8180000000000001e-05 2024-02-05T04:54:19.711 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.712 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.712 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:19.712 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.182643+0000", 2024-02-05T04:54:19.712 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1598000000000001e-05 2024-02-05T04:54:19.712 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.712 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.712 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:54:19.712 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.182744+0000", 2024-02-05T04:54:19.712 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000100881 2024-02-05T04:54:19.713 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.713 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.713 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:19.713 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.183600+0000", 2024-02-05T04:54:19.713 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00085556200000000003 2024-02-05T04:54:19.713 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.713 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.713 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:19.713 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.188714+0000", 2024-02-05T04:54:19.713 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.005113828 2024-02-05T04:54:19.713 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.713 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.713 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T04:54:19.713 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.191180+0000", 2024-02-05T04:54:19.713 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0024662130000000001 2024-02-05T04:54:19.714 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.714 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.714 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:19.714 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:35.191232+0000", 2024-02-05T04:54:19.714 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2496e-05 2024-02-05T04:54:19.714 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.714 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:19.714 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.714 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.714 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.714 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:525 3.ds0 3:be3a9027:::smithi19639362-44:head [delete] snapc f=[f,e,d,b,8,7] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e57)", 2024-02-05T04:54:19.714 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:48.958273+0000", 2024-02-05T04:54:19.714 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 30.724921685999998, 2024-02-05T04:54:19.714 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0081252289999999994, 2024-02-05T04:54:19.714 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:19.714 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:19.715 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:54:19.715 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:54:19.715 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:19.715 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 525 2024-02-05T04:54:19.715 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.715 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:19.715 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.715 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:19.715 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.958273+0000", 2024-02-05T04:54:19.715 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:19.715 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.715 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.715 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:19.715 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.958273+0000", 2024-02-05T04:54:19.715 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:19.715 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.715 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.716 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:19.716 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.958274+0000", 2024-02-05T04:54:19.716 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8870000000000001e-06 2024-02-05T04:54:19.716 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.716 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.716 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:19.716 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.958282+0000", 2024-02-05T04:54:19.716 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2599999999999999e-06 2024-02-05T04:54:19.716 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.716 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.717 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:19.717 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.958296+0000", 2024-02-05T04:54:19.717 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4157e-05 2024-02-05T04:54:19.717 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.717 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.717 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:19.717 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.958308+0000", 2024-02-05T04:54:19.717 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2075999999999999e-05 2024-02-05T04:54:19.717 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.717 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.717 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:19.718 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.958677+0000", 2024-02-05T04:54:19.718 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00036947599999999998 2024-02-05T04:54:19.718 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.718 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.718 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:54:19.718 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.958775+0000", 2024-02-05T04:54:19.718 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8039000000000001e-05 2024-02-05T04:54:19.718 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.718 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.718 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:19.718 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.959380+0000", 2024-02-05T04:54:19.718 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00060500599999999995 2024-02-05T04:54:19.718 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.718 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.718 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:19.719 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.966178+0000", 2024-02-05T04:54:19.719 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0067973230000000001 2024-02-05T04:54:19.719 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.719 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.719 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T04:54:19.719 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.966328+0000", 2024-02-05T04:54:19.719 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000149918 2024-02-05T04:54:19.719 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.719 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.719 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:19.719 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.966398+0000", 2024-02-05T04:54:19.719 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0086999999999995e-05 2024-02-05T04:54:19.719 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.719 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:19.720 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.720 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.720 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.720 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:526 3.ds0 3:b7ad4782:::smithi19639362-40:head [copy-from ver 29] snapc f=[f,e,d,b,8,7] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-02-05T04:54:19.720 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:48.966782+0000", 2024-02-05T04:54:19.720 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 30.716412206000001, 2024-02-05T04:54:19.720 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0165904020000001, 2024-02-05T04:54:19.720 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:19.720 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:19.720 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:54:19.720 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:54:19.720 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:19.720 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 526 2024-02-05T04:54:19.721 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.721 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:19.721 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.721 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:19.721 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.966782+0000", 2024-02-05T04:54:19.721 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:19.721 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.721 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.721 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:19.721 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.966782+0000", 2024-02-05T04:54:19.721 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:19.721 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.721 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.721 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:19.721 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.966784+0000", 2024-02-05T04:54:19.722 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5510000000000001e-06 2024-02-05T04:54:19.722 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.722 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.722 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:19.722 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.966790+0000", 2024-02-05T04:54:19.722 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7390000000000002e-06 2024-02-05T04:54:19.722 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.722 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.722 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:19.722 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.966805+0000", 2024-02-05T04:54:19.722 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4209999999999999e-05 2024-02-05T04:54:19.722 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.723 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.723 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:19.723 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.966813+0000", 2024-02-05T04:54:19.723 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0509999999999997e-06 2024-02-05T04:54:19.723 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.723 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.723 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:19.723 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.966917+0000", 2024-02-05T04:54:19.723 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010419799999999999 2024-02-05T04:54:19.723 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.723 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.723 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:54:19.723 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.967039+0000", 2024-02-05T04:54:19.723 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012211500000000001 2024-02-05T04:54:19.724 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.724 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.724 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:19.724 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.970041+0000", 2024-02-05T04:54:19.724 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0030016580000001 2024-02-05T04:54:19.724 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.724 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.724 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:19.724 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.978954+0000", 2024-02-05T04:54:19.724 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.008913397 2024-02-05T04:54:19.724 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.724 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.724 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T04:54:19.724 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.983312+0000", 2024-02-05T04:54:19.725 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0043576969999999998 2024-02-05T04:54:19.725 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.725 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.725 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:19.725 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.983372+0000", 2024-02-05T04:54:19.725 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0785999999999997e-05 2024-02-05T04:54:19.725 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.725 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:19.725 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.725 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.725 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.725 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:527 3.ds0 3:b7ad4782:::smithi19639362-40:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e57)", 2024-02-05T04:54:19.726 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:48.966876+0000", 2024-02-05T04:54:19.726 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 30.716318474000001, 2024-02-05T04:54:19.726 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.006645641, 2024-02-05T04:54:19.726 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:19.726 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-02-05T04:54:19.726 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:54:19.726 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:54:19.726 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:19.726 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 527 2024-02-05T04:54:19.726 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.726 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:19.726 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.726 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:19.726 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.966876+0000", 2024-02-05T04:54:19.726 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:19.727 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.727 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.727 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:19.727 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.966876+0000", 2024-02-05T04:54:19.727 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:19.727 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.727 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.727 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:19.727 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.966876+0000", 2024-02-05T04:54:19.727 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.68e-07 2024-02-05T04:54:19.728 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.728 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.728 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:19.728 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.966880+0000", 2024-02-05T04:54:19.728 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.1690000000000001e-06 2024-02-05T04:54:19.728 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.728 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.728 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:19.728 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.966889+0000", 2024-02-05T04:54:19.728 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.38e-06 2024-02-05T04:54:19.728 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.728 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.728 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:19.728 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.966896+0000", 2024-02-05T04:54:19.729 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2130000000000002e-06 2024-02-05T04:54:19.729 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.729 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.729 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:19.729 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.967519+0000", 2024-02-05T04:54:19.729 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00062232499999999996 2024-02-05T04:54:19.729 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.729 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.729 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for blocked object", 2024-02-05T04:54:19.729 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.967582+0000", 2024-02-05T04:54:19.729 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3142e-05 2024-02-05T04:54:19.729 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.729 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.729 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:19.729 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.973321+0000", 2024-02-05T04:54:19.729 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.005739114 2024-02-05T04:54:19.730 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.730 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.730 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:54:19.730 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.973429+0000", 2024-02-05T04:54:19.730 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000108207 2024-02-05T04:54:19.730 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.730 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.730 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:19.730 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.973521+0000", 2024-02-05T04:54:19.730 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2423000000000006e-05 2024-02-05T04:54:19.730 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.730 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:19.730 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.730 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.731 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.731 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.5s4 58/57 ECSubWrite(tid=58, reqid=osd.15.0:58, at_version=58'38, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:19.731 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:49.956419+0000", 2024-02-05T04:54:19.731 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 29.726775086, 2024-02-05T04:54:19.731 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0080420990000000005, 2024-02-05T04:54:19.731 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:19.731 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:19.731 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:19.731 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.731 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:19.731 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.956419+0000", 2024-02-05T04:54:19.731 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:19.731 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.731 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.731 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:19.732 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.956419+0000", 2024-02-05T04:54:19.732 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:19.732 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.732 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.732 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:19.732 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.956420+0000", 2024-02-05T04:54:19.732 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6499999999999998e-07 2024-02-05T04:54:19.732 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.732 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.732 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:19.732 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.956443+0000", 2024-02-05T04:54:19.732 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2725999999999999e-05 2024-02-05T04:54:19.732 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.732 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.732 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:19.733 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.956457+0000", 2024-02-05T04:54:19.733 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.451e-05 2024-02-05T04:54:19.733 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.733 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.733 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:19.733 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.956466+0000", 2024-02-05T04:54:19.733 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1279999999999997e-06 2024-02-05T04:54:19.733 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.733 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.733 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:19.734 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.957077+0000", 2024-02-05T04:54:19.734 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00061054900000000005 2024-02-05T04:54:19.734 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.734 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.734 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:19.734 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.957092+0000", 2024-02-05T04:54:19.734 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5242000000000001e-05 2024-02-05T04:54:19.734 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.734 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.734 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:19.734 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.964399+0000", 2024-02-05T04:54:19.734 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0073072420000000003 2024-02-05T04:54:19.734 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.734 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.734 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:19.734 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.964461+0000", 2024-02-05T04:54:19.735 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1937e-05 2024-02-05T04:54:19.735 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.735 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:19.735 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.735 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.735 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.735 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.bs5 58/57 ECSubWrite(tid=252, reqid=osd.13.0:252, at_version=58'54, trim_to=0'0, roll_forward_to=58'52))", 2024-02-05T04:54:19.735 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:49.968805+0000", 2024-02-05T04:54:19.735 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 29.714389203, 2024-02-05T04:54:19.735 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0074122190000000003, 2024-02-05T04:54:19.735 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:19.735 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:19.735 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:19.735 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.735 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:19.736 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.968805+0000", 2024-02-05T04:54:19.736 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:19.736 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.736 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.736 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:19.736 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.968805+0000", 2024-02-05T04:54:19.736 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:19.736 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.736 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.736 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:19.736 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.968806+0000", 2024-02-05T04:54:19.736 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0500000000000002e-07 2024-02-05T04:54:19.736 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.736 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.736 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:19.737 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.968831+0000", 2024-02-05T04:54:19.737 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5119000000000001e-05 2024-02-05T04:54:19.737 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.737 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.737 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:19.737 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.968843+0000", 2024-02-05T04:54:19.737 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2169e-05 2024-02-05T04:54:19.737 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.737 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.737 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:19.737 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.968852+0000", 2024-02-05T04:54:19.737 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.3729999999999999e-06 2024-02-05T04:54:19.737 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.737 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.738 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:19.738 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.968910+0000", 2024-02-05T04:54:19.738 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7883000000000001e-05 2024-02-05T04:54:19.738 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.738 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.738 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:19.738 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.968921+0000", 2024-02-05T04:54:19.738 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0473e-05 2024-02-05T04:54:19.738 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.738 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.739 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:19.739 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.976171+0000", 2024-02-05T04:54:19.739 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0072504680000000004 2024-02-05T04:54:19.739 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.739 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.739 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:19.739 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.976217+0000", 2024-02-05T04:54:19.739 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5929e-05 2024-02-05T04:54:19.739 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.739 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:19.739 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.739 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.739 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.739 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.5s4 58/57 ECSubWrite(tid=60, reqid=osd.15.0:60, at_version=58'40, trim_to=0'0, roll_forward_to=58'38))", 2024-02-05T04:54:19.739 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:49.969258+0000", 2024-02-05T04:54:19.740 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 29.713935822, 2024-02-05T04:54:19.740 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.009848295, 2024-02-05T04:54:19.740 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:19.740 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:19.740 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:19.740 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.740 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:19.740 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.969258+0000", 2024-02-05T04:54:19.740 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:19.740 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.740 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.740 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:19.740 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.969258+0000", 2024-02-05T04:54:19.740 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:19.740 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.741 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.741 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:19.741 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.969259+0000", 2024-02-05T04:54:19.741 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9899999999999999e-07 2024-02-05T04:54:19.741 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.741 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.741 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:19.741 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.969282+0000", 2024-02-05T04:54:19.741 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2297999999999999e-05 2024-02-05T04:54:19.741 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.741 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.741 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:19.741 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.969295+0000", 2024-02-05T04:54:19.741 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3447e-05 2024-02-05T04:54:19.741 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.742 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.742 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:19.742 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.969303+0000", 2024-02-05T04:54:19.742 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4090000000000003e-06 2024-02-05T04:54:19.742 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.742 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.742 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:19.742 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.969393+0000", 2024-02-05T04:54:19.742 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.9324000000000001e-05 2024-02-05T04:54:19.742 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.742 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.742 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:19.742 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.969403+0000", 2024-02-05T04:54:19.742 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0244000000000001e-05 2024-02-05T04:54:19.742 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.742 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.743 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:19.743 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.979067+0000", 2024-02-05T04:54:19.743 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0096643059999999992 2024-02-05T04:54:19.743 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.743 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.743 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:19.743 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:49.979107+0000", 2024-02-05T04:54:19.743 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9468000000000002e-05 2024-02-05T04:54:19.743 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.743 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:19.743 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.743 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.744 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.744 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.es2 69/57 ECSubWrite(tid=87, reqid=client.4634.0:625, at_version=69'62, trim_to=0'0, roll_forward_to=66'60))", 2024-02-05T04:54:19.744 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:54:00.976943+0000", 2024-02-05T04:54:19.744 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 18.706251651999999, 2024-02-05T04:54:19.744 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0081354099999999992, 2024-02-05T04:54:19.744 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:19.744 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:19.744 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:19.744 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.744 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:19.744 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:00.976943+0000", 2024-02-05T04:54:19.745 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:19.745 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.745 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.745 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:19.745 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:00.976943+0000", 2024-02-05T04:54:19.745 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:19.745 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.745 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.745 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:19.745 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:00.976943+0000", 2024-02-05T04:54:19.745 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3999999999999997e-07 2024-02-05T04:54:19.745 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.745 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.745 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:19.745 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:00.980005+0000", 2024-02-05T04:54:19.746 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0030625769999999999 2024-02-05T04:54:19.746 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.746 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.746 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:19.746 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:00.980016+0000", 2024-02-05T04:54:19.746 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0446e-05 2024-02-05T04:54:19.746 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.746 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.746 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:19.746 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:00.980021+0000", 2024-02-05T04:54:19.746 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4960000000000004e-06 2024-02-05T04:54:19.746 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.746 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.746 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:19.746 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:00.980140+0000", 2024-02-05T04:54:19.747 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000118532 2024-02-05T04:54:19.747 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.747 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.747 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:19.747 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:00.980149+0000", 2024-02-05T04:54:19.747 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.9020000000000005e-06 2024-02-05T04:54:19.747 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.747 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.747 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:19.747 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:00.985043+0000", 2024-02-05T04:54:19.747 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0048943470000000003 2024-02-05T04:54:19.747 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.747 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.747 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:19.747 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:00.985078+0000", 2024-02-05T04:54:19.748 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4770000000000001e-05 2024-02-05T04:54:19.748 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.748 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:19.748 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.748 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.748 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.748 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(osd.12.23:1 3.ds0 3:b7ad4782:::smithi19639362-40:head [assert-version v29,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-02-05T04:54:19.748 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:54:00.999327+0000", 2024-02-05T04:54:19.748 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 18.683867641999999, 2024-02-05T04:54:19.748 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0072280269999999997, 2024-02-05T04:54:19.748 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:19.749 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-02-05T04:54:19.749 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:19.749 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.749 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:19.749 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:00.999327+0000", 2024-02-05T04:54:19.749 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:19.749 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.749 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.749 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:19.749 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:00.999327+0000", 2024-02-05T04:54:19.749 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:19.750 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.750 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.750 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:19.750 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:00.999328+0000", 2024-02-05T04:54:19.750 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1289999999999999e-06 2024-02-05T04:54:19.750 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.750 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.750 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:19.750 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:00.999332+0000", 2024-02-05T04:54:19.750 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8070000000000001e-06 2024-02-05T04:54:19.750 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.750 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.750 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:19.750 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:00.999340+0000", 2024-02-05T04:54:19.751 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8489999999999995e-06 2024-02-05T04:54:19.751 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.751 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.751 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:19.751 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:00.999355+0000", 2024-02-05T04:54:19.751 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4647999999999999e-05 2024-02-05T04:54:19.751 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.751 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.751 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:19.751 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:00.999414+0000", 2024-02-05T04:54:19.751 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9438000000000002e-05 2024-02-05T04:54:19.751 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.752 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.752 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:54:19.752 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:00.999489+0000", 2024-02-05T04:54:19.752 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4589e-05 2024-02-05T04:54:19.752 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.752 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.752 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:19.752 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:01.006555+0000", 2024-02-05T04:54:19.752 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.007065567 2024-02-05T04:54:19.752 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.752 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:19.752 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.752 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.752 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.752 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.19s5 74/67 ECSubWrite(tid=8, reqid=osd.9.0:8, at_version=74'27, trim_to=0'0, roll_forward_to=72'25))", 2024-02-05T04:54:19.753 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:54:06.043813+0000", 2024-02-05T04:54:19.753 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 13.639381231, 2024-02-05T04:54:19.753 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0077211479999999997, 2024-02-05T04:54:19.753 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:19.753 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:19.753 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:19.753 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.753 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:19.753 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.043813+0000", 2024-02-05T04:54:19.753 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:19.753 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.754 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.754 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:19.754 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.043813+0000", 2024-02-05T04:54:19.754 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:19.754 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.754 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.754 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:19.754 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.043814+0000", 2024-02-05T04:54:19.755 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2299999999999999e-07 2024-02-05T04:54:19.755 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.755 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.755 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:19.755 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.043837+0000", 2024-02-05T04:54:19.755 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2957000000000001e-05 2024-02-05T04:54:19.755 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.755 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.755 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:19.755 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.043848+0000", 2024-02-05T04:54:19.755 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1413000000000001e-05 2024-02-05T04:54:19.755 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.755 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.755 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:19.755 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.043856+0000", 2024-02-05T04:54:19.756 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.089e-06 2024-02-05T04:54:19.756 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.756 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.756 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:19.756 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.043929+0000", 2024-02-05T04:54:19.756 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2453999999999994e-05 2024-02-05T04:54:19.756 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.756 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.756 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:19.756 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.043940+0000", 2024-02-05T04:54:19.756 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0872e-05 2024-02-05T04:54:19.756 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.756 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.756 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:19.757 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.051495+0000", 2024-02-05T04:54:19.757 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0075551910000000002 2024-02-05T04:54:19.757 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.757 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.757 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:19.757 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.051534+0000", 2024-02-05T04:54:19.757 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9249000000000003e-05 2024-02-05T04:54:19.757 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.757 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:19.757 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.757 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.757 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.757 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.19s5 74/67 ECSubWrite(tid=9, reqid=osd.9.0:9, at_version=74'29, trim_to=0'0, roll_forward_to=72'25))", 2024-02-05T04:54:19.757 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:54:06.045286+0000", 2024-02-05T04:54:19.758 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 13.637908501, 2024-02-05T04:54:19.758 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0075465910000000001, 2024-02-05T04:54:19.758 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:19.758 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:19.758 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:19.758 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.758 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:19.758 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.045286+0000", 2024-02-05T04:54:19.758 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:19.758 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.758 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.758 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:19.758 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.045286+0000", 2024-02-05T04:54:19.758 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:19.758 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.759 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.759 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:19.759 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.045286+0000", 2024-02-05T04:54:19.759 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5099999999999999e-07 2024-02-05T04:54:19.759 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.759 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.759 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:19.759 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.045309+0000", 2024-02-05T04:54:19.759 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2325999999999999e-05 2024-02-05T04:54:19.760 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.760 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.760 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:19.760 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.045319+0000", 2024-02-05T04:54:19.760 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0750999999999999e-05 2024-02-05T04:54:19.760 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.760 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.760 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:19.760 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.045328+0000", 2024-02-05T04:54:19.760 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.67e-06 2024-02-05T04:54:19.760 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.760 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.760 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:19.761 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.046742+0000", 2024-02-05T04:54:19.761 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001414219 2024-02-05T04:54:19.761 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.761 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.761 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:19.761 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.046752+0000", 2024-02-05T04:54:19.761 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8609999999999993e-06 2024-02-05T04:54:19.761 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.761 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.761 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:19.761 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.052781+0000", 2024-02-05T04:54:19.761 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0060288850000000003 2024-02-05T04:54:19.761 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.761 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.761 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:19.761 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.052832+0000", 2024-02-05T04:54:19.761 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1227999999999998e-05 2024-02-05T04:54:19.762 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.762 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:19.762 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.762 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.762 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.762 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.bs5 74/57 ECSubWrite(tid=339, reqid=osd.13.0:339, at_version=74'90, trim_to=0'0, roll_forward_to=72'86))", 2024-02-05T04:54:19.762 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:54:06.045835+0000", 2024-02-05T04:54:19.762 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 13.637359644, 2024-02-05T04:54:19.762 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0070078880000000003, 2024-02-05T04:54:19.762 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:19.762 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:19.762 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:19.762 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.762 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:19.762 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.045835+0000", 2024-02-05T04:54:19.763 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:19.763 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.763 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.763 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:19.763 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.045835+0000", 2024-02-05T04:54:19.763 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:19.763 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.763 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.763 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:19.763 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.045835+0000", 2024-02-05T04:54:19.763 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7000000000000004e-07 2024-02-05T04:54:19.763 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.763 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.764 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:19.764 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.045857+0000", 2024-02-05T04:54:19.764 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1846000000000001e-05 2024-02-05T04:54:19.764 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.764 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.764 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:19.765 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.045869+0000", 2024-02-05T04:54:19.765 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1610000000000001e-05 2024-02-05T04:54:19.765 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.765 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.765 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:19.765 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.045877+0000", 2024-02-05T04:54:19.765 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.2400000000000007e-06 2024-02-05T04:54:19.765 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.765 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.765 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:19.765 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.046861+0000", 2024-02-05T04:54:19.765 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00098359100000000002 2024-02-05T04:54:19.765 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.765 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.765 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:19.766 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.046870+0000", 2024-02-05T04:54:19.766 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.4080000000000004e-06 2024-02-05T04:54:19.766 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.766 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.766 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:19.766 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.052803+0000", 2024-02-05T04:54:19.766 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0059331760000000001 2024-02-05T04:54:19.766 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.766 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.766 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:19.766 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.052842+0000", 2024-02-05T04:54:19.766 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9347000000000001e-05 2024-02-05T04:54:19.766 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.766 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:19.766 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.767 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.767 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.767 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.3s1 84/62 ECSubWrite(tid=123, reqid=client.4634.0:847, at_version=84'61, trim_to=0'0, roll_forward_to=83'59))", 2024-02-05T04:54:19.767 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:54:16.122495+0000", 2024-02-05T04:54:19.767 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 3.5606996660000001, 2024-02-05T04:54:19.767 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0078373450000000008, 2024-02-05T04:54:19.767 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:19.767 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:19.767 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:19.767 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.767 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:19.767 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.122495+0000", 2024-02-05T04:54:19.767 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:19.767 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.768 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.768 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:19.768 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.122495+0000", 2024-02-05T04:54:19.768 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:19.768 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.768 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.768 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:19.768 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.122496+0000", 2024-02-05T04:54:19.768 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8599999999999996e-07 2024-02-05T04:54:19.768 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.768 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.768 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:19.768 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.124505+0000", 2024-02-05T04:54:19.768 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0020098389999999998 2024-02-05T04:54:19.768 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.768 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.769 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:19.769 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.124520+0000", 2024-02-05T04:54:19.769 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4497e-05 2024-02-05T04:54:19.769 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.769 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.769 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:19.769 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.124529+0000", 2024-02-05T04:54:19.769 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.7059999999999993e-06 2024-02-05T04:54:19.769 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.769 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.769 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:19.769 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.124587+0000", 2024-02-05T04:54:19.769 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8301999999999998e-05 2024-02-05T04:54:19.769 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.770 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.770 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:19.770 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.124594+0000", 2024-02-05T04:54:19.770 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8850000000000001e-06 2024-02-05T04:54:19.770 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.770 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.770 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:19.770 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.130249+0000", 2024-02-05T04:54:19.770 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0056555970000000001 2024-02-05T04:54:19.771 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.771 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.771 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:19.771 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.130332+0000", 2024-02-05T04:54:19.771 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.2533000000000004e-05 2024-02-05T04:54:19.771 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.771 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:19.771 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.771 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.771 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.771 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:870 3.ds0 3:be3a9027:::smithi19639362-44:head [rollback 12] snapc 1d=[1d,1c,19,18,17,16,12,8] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-02-05T04:54:19.771 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:54:16.142744+0000", 2024-02-05T04:54:19.771 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 3.5404500049999998, 2024-02-05T04:54:19.771 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.01001198, 2024-02-05T04:54:19.771 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:19.771 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:19.772 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:54:19.772 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:54:19.772 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:19.772 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 870 2024-02-05T04:54:19.772 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.772 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:19.772 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.772 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:19.772 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.142744+0000", 2024-02-05T04:54:19.772 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:19.772 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.772 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.772 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:19.772 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.142744+0000", 2024-02-05T04:54:19.773 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:19.773 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.773 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.773 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:19.773 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.142745+0000", 2024-02-05T04:54:19.773 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9299999999999998e-07 2024-02-05T04:54:19.773 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.773 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.773 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:19.773 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.142747+0000", 2024-02-05T04:54:19.773 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.616e-06 2024-02-05T04:54:19.773 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.773 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.773 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:19.773 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.142755+0000", 2024-02-05T04:54:19.774 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6899999999999992e-06 2024-02-05T04:54:19.774 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.774 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.774 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:19.774 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.142765+0000", 2024-02-05T04:54:19.774 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.533e-06 2024-02-05T04:54:19.774 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.774 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.774 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:19.774 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.145721+0000", 2024-02-05T04:54:19.774 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002956537 2024-02-05T04:54:19.774 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.774 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.774 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:54:19.775 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.145847+0000", 2024-02-05T04:54:19.775 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012576699999999999 2024-02-05T04:54:19.775 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.775 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.775 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:19.775 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.146602+0000", 2024-02-05T04:54:19.775 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00075510300000000002 2024-02-05T04:54:19.775 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.775 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.776 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:19.776 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.149201+0000", 2024-02-05T04:54:19.776 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0025993269999999998 2024-02-05T04:54:19.776 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.776 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.776 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T04:54:19.776 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.152481+0000", 2024-02-05T04:54:19.776 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0032792659999999999 2024-02-05T04:54:19.776 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.776 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.776 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:19.776 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.152756+0000", 2024-02-05T04:54:19.776 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00027554800000000003 2024-02-05T04:54:19.776 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.777 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:19.777 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.777 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.777 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.777 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:908 3.ds0 3:b4e8e82c:::smithi19639362-25:head [read 0~638976,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-02-05T04:54:19.777 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:54:16.192269+0000", 2024-02-05T04:54:19.777 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 3.490924868, 2024-02-05T04:54:19.777 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.007187873, 2024-02-05T04:54:19.777 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:19.777 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-02-05T04:54:19.777 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:54:19.777 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:54:19.777 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:19.777 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 908 2024-02-05T04:54:19.778 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.778 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:19.778 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.778 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:19.778 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.192269+0000", 2024-02-05T04:54:19.778 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:19.778 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.778 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.778 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:19.778 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.192269+0000", 2024-02-05T04:54:19.778 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:19.778 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.778 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.778 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:19.778 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.192270+0000", 2024-02-05T04:54:19.779 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.2099999999999996e-07 2024-02-05T04:54:19.779 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.779 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.779 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:19.779 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.192274+0000", 2024-02-05T04:54:19.779 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5810000000000001e-06 2024-02-05T04:54:19.779 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.779 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.779 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:19.779 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.192280+0000", 2024-02-05T04:54:19.779 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.2349999999999996e-06 2024-02-05T04:54:19.779 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.779 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.779 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:19.779 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.192285+0000", 2024-02-05T04:54:19.780 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9339999999999997e-06 2024-02-05T04:54:19.780 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.780 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.780 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:19.780 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.194212+0000", 2024-02-05T04:54:19.780 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0019268130000000001 2024-02-05T04:54:19.781 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.781 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.781 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:54:19.781 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.194323+0000", 2024-02-05T04:54:19.781 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011177 2024-02-05T04:54:19.781 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.781 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.781 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:19.781 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.199457+0000", 2024-02-05T04:54:19.781 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0051339189999999998 2024-02-05T04:54:19.781 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.782 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:19.782 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.782 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.782 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.782 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:909 3.ds0 3:b4e8e82c:::smithi19639362-25:head [read 0~638976,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-02-05T04:54:19.782 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:54:16.192291+0000", 2024-02-05T04:54:19.782 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 3.4909030360000002, 2024-02-05T04:54:19.782 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.010835067, 2024-02-05T04:54:19.782 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:19.782 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-02-05T04:54:19.782 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:54:19.782 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:54:19.782 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:19.782 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 909 2024-02-05T04:54:19.783 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.783 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:19.783 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.783 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:19.783 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.192291+0000", 2024-02-05T04:54:19.783 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:19.783 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.783 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.783 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:19.783 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.192291+0000", 2024-02-05T04:54:19.783 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:19.783 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.783 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.784 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:19.784 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.192292+0000", 2024-02-05T04:54:19.784 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.0800000000000001e-07 2024-02-05T04:54:19.784 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.784 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.784 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:19.784 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.192293+0000", 2024-02-05T04:54:19.784 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9589999999999998e-06 2024-02-05T04:54:19.784 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.784 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.784 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:19.784 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.192299+0000", 2024-02-05T04:54:19.784 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2889999999999997e-06 2024-02-05T04:54:19.784 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.784 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.785 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:19.785 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.192303+0000", 2024-02-05T04:54:19.785 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0620000000000002e-06 2024-02-05T04:54:19.785 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.785 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.785 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:19.785 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.199630+0000", 2024-02-05T04:54:19.785 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0073275709999999997 2024-02-05T04:54:19.785 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.785 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.785 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:54:19.785 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.199744+0000", 2024-02-05T04:54:19.785 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011384599999999999 2024-02-05T04:54:19.785 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:19.785 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:19.785 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:19.786 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.203126+0000", 2024-02-05T04:54:19.786 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0033820320000000001 2024-02-05T04:54:19.786 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.786 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:19.786 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.786 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:19.786 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:19.786 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T04:54:19.788 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-02-05T04:54:19.789 INFO:tasks.ceph.osd.12.smithi062.stderr:2024-02-05T04:54:19.787+0000 7f5e5847e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 34695) UID: 0 2024-02-05T04:54:19.889 INFO:tasks.ceph.osd.12.smithi062.stderr:2024-02-05T04:54:19.888+0000 7f5e5847e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 34695) UID: 0 2024-02-05T04:54:19.894 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T04:54:19.895 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-02-05T04:54:19.895 INFO:teuthology.orchestra.run.smithi196.stdout: "num_ops": 0 2024-02-05T04:54:19.895 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T04:54:19.903 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-02-05T04:54:19.989 INFO:tasks.ceph.osd.4.smithi062.stderr:2024-02-05T04:54:19.989+0000 7f708c308640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34689) UID: 0 2024-02-05T04:54:20.011 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T04:54:20.011 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-02-05T04:54:20.011 INFO:teuthology.orchestra.run.smithi196.stdout: "complaint_time": 30, 2024-02-05T04:54:20.011 INFO:teuthology.orchestra.run.smithi196.stdout: "num_blocked_ops": 0 2024-02-05T04:54:20.011 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T04:54:20.020 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-02-05T04:54:20.091 INFO:tasks.ceph.osd.14.smithi150.stderr:2024-02-05T04:54:20.089+0000 7f5f81083640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 34020) UID: 0 2024-02-05T04:54:20.092 INFO:tasks.rados.rados.0.smithi196.stdout:394: done (2 left) 2024-02-05T04:54:20.092 INFO:tasks.rados.rados.0.smithi196.stdout:395: setattr oid 20 current snap is 30 2024-02-05T04:54:20.093 INFO:tasks.rados.rados.0.smithi196.stdout:393: finishing rollback tid 0 to smithi19639362-14 2024-02-05T04:54:20.093 INFO:tasks.rados.rados.0.smithi196.stdout:393: finishing rollback tid 1 to smithi19639362-14 2024-02-05T04:54:20.093 INFO:tasks.rados.rados.0.smithi196.stdout:392: finishing write tid 1 to smithi19639362-48 2024-02-05T04:54:20.093 INFO:tasks.rados.rados.0.smithi196.stdout:392: finishing write tid 2 to smithi19639362-48 2024-02-05T04:54:20.093 INFO:tasks.rados.rados.0.smithi196.stdout:393: finishing rollback tid 2 to smithi19639362-14 2024-02-05T04:54:20.093 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 14 v 103 (ObjNum 59 snap 1 seq_num 59) dirty exists 2024-02-05T04:54:20.093 INFO:tasks.rados.rados.0.smithi196.stdout:392: finishing write tid 3 to smithi19639362-48 2024-02-05T04:54:20.093 INFO:tasks.rados.rados.0.smithi196.stdout:392: oid 48 updating version 0 to 89 2024-02-05T04:54:20.093 INFO:tasks.rados.rados.0.smithi196.stdout:392: oid 48 updating version 89 to 90 2024-02-05T04:54:20.093 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 48 v 90 (ObjNum 163 snap 30 seq_num 163) dirty exists 2024-02-05T04:54:20.093 INFO:tasks.rados.rados.0.smithi196.stdout:392: left oid 48 (ObjNum 163 snap 30 seq_num 163) 2024-02-05T04:54:20.097 INFO:tasks.rados.rados.0.smithi196.stdout:392: done (2 left) 2024-02-05T04:54:20.097 INFO:tasks.rados.rados.0.smithi196.stdout:393: done (1 left) 2024-02-05T04:54:20.097 INFO:tasks.rados.rados.0.smithi196.stdout:append oid 34 current snap is 30 2024-02-05T04:54:20.097 INFO:tasks.rados.rados.0.smithi196.stdout:396: seq_num 164 ranges {0=360448} 2024-02-05T04:54:20.100 INFO:tasks.rados.rados.0.smithi196.stdout:396: writing smithi19639362-34 from 0 to 360448 tid 1 2024-02-05T04:54:20.100 INFO:tasks.rados.rados.0.smithi196.stdout:397: rmattr oid 50 current snap is 30 2024-02-05T04:54:20.100 INFO:tasks.rados.rados.0.smithi196.stdout:397: done (2 left) 2024-02-05T04:54:20.100 INFO:tasks.rados.rados.0.smithi196.stdout:398: read oid 8 snap -1 2024-02-05T04:54:20.100 INFO:tasks.rados.rados.0.smithi196.stdout:398: expect (ObjNum 148 snap 28 seq_num 148) 2024-02-05T04:54:20.100 INFO:tasks.rados.rados.0.smithi196.stdout:399: read oid 43 snap 23 2024-02-05T04:54:20.101 INFO:tasks.rados.rados.0.smithi196.stdout:399: expect (ObjNum 122 snap 22 seq_num 122) 2024-02-05T04:54:20.101 INFO:tasks.rados.rados.0.smithi196.stdout: waiting on 4 2024-02-05T04:54:20.103 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-02-05T04:54:20.111 INFO:tasks.rados.rados.0.smithi196.stdout:396: finishing write tid 1 to smithi19639362-34 2024-02-05T04:54:20.111 INFO:tasks.rados.rados.0.smithi196.stdout:396: finishing write tid 2 to smithi19639362-34 2024-02-05T04:54:20.113 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 20 v 19 (ObjNum 112 snap 18 seq_num 112) dirty exists 2024-02-05T04:54:20.113 INFO:tasks.rados.rados.0.smithi196.stdout:395: done (3 left) 2024-02-05T04:54:20.113 INFO:tasks.rados.rados.0.smithi196.stdout: waiting on 3 2024-02-05T04:54:20.114 INFO:tasks.rados.rados.0.smithi196.stdout:396: finishing write tid 3 to smithi19639362-34 2024-02-05T04:54:20.114 INFO:tasks.rados.rados.0.smithi196.stdout:396: oid 34 updating version 0 to 71 2024-02-05T04:54:20.114 INFO:tasks.rados.rados.0.smithi196.stdout:396: oid 34 updating version 71 to 72 2024-02-05T04:54:20.114 INFO:tasks.rados.rados.0.smithi196.stdout:update_object_version oid 34 v 72 (ObjNum 164 snap 30 seq_num 164) dirty exists 2024-02-05T04:54:20.114 INFO:tasks.rados.rados.0.smithi196.stdout:396: left oid 34 (ObjNum 164 snap 30 seq_num 164) 2024-02-05T04:54:20.114 INFO:tasks.rados.rados.0.smithi196.stdout:396: done (2 left) 2024-02-05T04:54:20.115 INFO:tasks.rados.rados.0.smithi196.stdout: waiting on 2 2024-02-05T04:54:20.124 INFO:tasks.rados.rados.0.smithi196.stdout:399: expect (ObjNum 122 snap 22 seq_num 122) 2024-02-05T04:54:20.140 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T04:54:20.140 INFO:teuthology.orchestra.run.smithi196.stdout: "size": 20, 2024-02-05T04:54:20.141 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 600, 2024-02-05T04:54:20.141 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-02-05T04:54:20.141 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.141 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:293 3.fs0 3:f8d01528:::smithi19639362-17:head [setxattr _header (56) in=7b] snapc 3=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-02-05T04:54:20.141 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:30.385185+0000", 2024-02-05T04:54:20.141 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 49.753463428000003, 2024-02-05T04:54:20.141 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.010325663000000001, 2024-02-05T04:54:20.141 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:20.141 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:20.141 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:54:20.141 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:54:20.141 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:20.141 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 293 2024-02-05T04:54:20.141 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.142 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:20.142 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.142 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:20.142 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.385185+0000", 2024-02-05T04:54:20.142 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:20.142 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.142 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.142 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:20.142 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.385185+0000", 2024-02-05T04:54:20.142 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:20.142 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.142 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.142 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:20.142 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.385186+0000", 2024-02-05T04:54:20.142 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9400000000000005e-07 2024-02-05T04:54:20.143 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.143 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.143 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:20.143 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.385189+0000", 2024-02-05T04:54:20.143 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.6809999999999998e-06 2024-02-05T04:54:20.143 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.143 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.143 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:20.143 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.385195+0000", 2024-02-05T04:54:20.143 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0630000000000002e-06 2024-02-05T04:54:20.143 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.143 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.143 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:20.143 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.385200+0000", 2024-02-05T04:54:20.144 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0560000000000004e-06 2024-02-05T04:54:20.144 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.144 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.144 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:20.144 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.390293+0000", 2024-02-05T04:54:20.144 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0050930180000000004 2024-02-05T04:54:20.144 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.144 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.144 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:54:20.144 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.390337+0000", 2024-02-05T04:54:20.144 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3721999999999999e-05 2024-02-05T04:54:20.144 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.144 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.144 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:20.145 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.390566+0000", 2024-02-05T04:54:20.145 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00022925199999999999 2024-02-05T04:54:20.145 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.145 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.145 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:20.145 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.395220+0000", 2024-02-05T04:54:20.145 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0046545919999999999 2024-02-05T04:54:20.145 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.145 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.145 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T04:54:20.145 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.395254+0000", 2024-02-05T04:54:20.145 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3522000000000002e-05 2024-02-05T04:54:20.145 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.145 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.146 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:20.146 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.395511+0000", 2024-02-05T04:54:20.146 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00025716299999999998 2024-02-05T04:54:20.146 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.146 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:20.146 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.146 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.146 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.146 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:294 3.fs0 3:f8d01528:::smithi19639362-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-02-05T04:54:20.146 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:30.385209+0000", 2024-02-05T04:54:20.146 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 49.753439825999997, 2024-02-05T04:54:20.146 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.011163398999999999, 2024-02-05T04:54:20.146 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:20.146 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-02-05T04:54:20.147 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:54:20.147 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:54:20.147 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:20.147 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 294 2024-02-05T04:54:20.147 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.147 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:20.147 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.147 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:20.147 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.385209+0000", 2024-02-05T04:54:20.147 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:20.147 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.147 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.147 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:20.147 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.385209+0000", 2024-02-05T04:54:20.147 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:20.148 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.148 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.148 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:20.148 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.385209+0000", 2024-02-05T04:54:20.148 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.32e-07 2024-02-05T04:54:20.149 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.149 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.149 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:20.149 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.385212+0000", 2024-02-05T04:54:20.149 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1900000000000002e-06 2024-02-05T04:54:20.149 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.149 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.149 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:20.149 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.385218+0000", 2024-02-05T04:54:20.149 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4169999999999997e-06 2024-02-05T04:54:20.149 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.149 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.149 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:20.149 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.385229+0000", 2024-02-05T04:54:20.150 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0616e-05 2024-02-05T04:54:20.150 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.150 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.150 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:20.150 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.390742+0000", 2024-02-05T04:54:20.150 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0055133980000000001 2024-02-05T04:54:20.150 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.150 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.150 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for rw locks", 2024-02-05T04:54:20.150 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.390788+0000", 2024-02-05T04:54:20.150 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5627000000000002e-05 2024-02-05T04:54:20.150 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.150 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.150 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:20.151 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.395549+0000", 2024-02-05T04:54:20.151 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0047614520000000002 2024-02-05T04:54:20.151 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.151 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.151 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:54:20.151 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.395622+0000", 2024-02-05T04:54:20.151 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2916000000000005e-05 2024-02-05T04:54:20.151 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.151 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.151 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:20.151 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:30.396372+0000", 2024-02-05T04:54:20.151 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00075035100000000001 2024-02-05T04:54:20.151 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.151 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:20.151 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.151 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.152 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.152 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:531 3.cs0 3:30950ab7:::smithi19639362-4:head [append 0~606208] snapc f=[f,e,d,b,8,7] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-02-05T04:54:20.152 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:48.973323+0000", 2024-02-05T04:54:20.152 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 31.165325630000002, 2024-02-05T04:54:20.152 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.011870864, 2024-02-05T04:54:20.152 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:20.152 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:20.152 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:54:20.152 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:54:20.152 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:20.152 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 531 2024-02-05T04:54:20.152 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.152 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:20.152 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.153 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:20.153 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.973323+0000", 2024-02-05T04:54:20.153 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:20.153 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.153 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.153 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:20.153 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.973323+0000", 2024-02-05T04:54:20.153 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:20.153 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.153 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.153 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:20.153 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.973325+0000", 2024-02-05T04:54:20.153 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.092e-06 2024-02-05T04:54:20.153 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.153 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.153 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:20.154 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.973678+0000", 2024-02-05T04:54:20.154 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00035291599999999999 2024-02-05T04:54:20.154 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.154 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.154 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:20.154 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.973691+0000", 2024-02-05T04:54:20.154 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3217000000000001e-05 2024-02-05T04:54:20.154 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.154 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.154 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:20.154 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.973697+0000", 2024-02-05T04:54:20.154 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6729999999999997e-06 2024-02-05T04:54:20.154 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.154 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.154 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:20.155 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.973747+0000", 2024-02-05T04:54:20.155 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9583000000000003e-05 2024-02-05T04:54:20.155 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.155 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.155 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:54:20.155 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.973824+0000", 2024-02-05T04:54:20.155 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7082000000000005e-05 2024-02-05T04:54:20.155 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.155 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.155 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:20.155 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.974743+0000", 2024-02-05T04:54:20.155 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00091901600000000004 2024-02-05T04:54:20.155 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.155 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.155 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:20.156 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.976151+0000", 2024-02-05T04:54:20.156 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001408124 2024-02-05T04:54:20.156 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.156 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.156 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T04:54:20.156 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.985102+0000", 2024-02-05T04:54:20.156 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0089515270000000008 2024-02-05T04:54:20.156 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.156 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.156 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:20.156 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:48.985194+0000", 2024-02-05T04:54:20.156 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1633999999999997e-05 2024-02-05T04:54:20.156 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.156 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:20.157 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.157 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.157 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.157 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:708 3.fs0 3:f8474b4b:::smithi19639362-36:head [copy-from ver 12] snapc 18=[18,17,16,12,8,7] ondisk+write+known_if_redirected+supports_pool_eio e75)", 2024-02-05T04:54:20.157 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:54:07.056827+0000", 2024-02-05T04:54:20.157 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 13.081821505000001, 2024-02-05T04:54:20.157 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.027282862000000001, 2024-02-05T04:54:20.157 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:20.157 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:20.157 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:54:20.157 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:54:20.157 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:20.157 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 708 2024-02-05T04:54:20.158 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.158 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:20.158 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.158 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:20.158 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.056827+0000", 2024-02-05T04:54:20.158 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:20.158 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.158 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.158 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:20.158 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.056827+0000", 2024-02-05T04:54:20.158 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:20.158 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.158 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.158 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:20.158 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.056829+0000", 2024-02-05T04:54:20.159 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4810000000000001e-06 2024-02-05T04:54:20.159 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.159 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.159 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:20.159 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.056834+0000", 2024-02-05T04:54:20.159 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5969999999999999e-06 2024-02-05T04:54:20.159 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.159 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.159 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:20.159 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.056856+0000", 2024-02-05T04:54:20.159 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1713e-05 2024-02-05T04:54:20.159 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.159 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.160 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:20.160 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.056864+0000", 2024-02-05T04:54:20.160 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5170000000000002e-06 2024-02-05T04:54:20.160 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.160 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.160 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:20.160 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.056917+0000", 2024-02-05T04:54:20.160 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3074e-05 2024-02-05T04:54:20.160 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.160 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.160 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:54:20.160 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.057026+0000", 2024-02-05T04:54:20.160 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000108899 2024-02-05T04:54:20.161 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.161 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.161 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:20.161 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.069750+0000", 2024-02-05T04:54:20.161 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.012724044 2024-02-05T04:54:20.161 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.161 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.161 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:20.161 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.080050+0000", 2024-02-05T04:54:20.161 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.010299971999999999 2024-02-05T04:54:20.161 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.161 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.161 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T04:54:20.161 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.084024+0000", 2024-02-05T04:54:20.162 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0039740899999999996 2024-02-05T04:54:20.162 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.162 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.162 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:20.162 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.084110+0000", 2024-02-05T04:54:20.162 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.6475000000000001e-05 2024-02-05T04:54:20.162 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.162 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:20.162 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.162 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.162 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.162 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:709 3.fs0 3:f8474b4b:::smithi19639362-36:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e75)", 2024-02-05T04:54:20.162 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:54:07.056883+0000", 2024-02-05T04:54:20.162 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 13.081765711999999, 2024-02-05T04:54:20.162 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.016829403, 2024-02-05T04:54:20.163 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:20.163 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-02-05T04:54:20.163 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:54:20.163 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:54:20.163 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:20.163 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 709 2024-02-05T04:54:20.163 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.163 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:20.163 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.163 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:20.163 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.056883+0000", 2024-02-05T04:54:20.163 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:20.163 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.163 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.163 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:20.164 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.056883+0000", 2024-02-05T04:54:20.164 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:20.164 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.164 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.164 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:20.164 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.056884+0000", 2024-02-05T04:54:20.164 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5799999999999999e-07 2024-02-05T04:54:20.164 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.164 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.164 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:20.164 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.056888+0000", 2024-02-05T04:54:20.164 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4000000000000002e-06 2024-02-05T04:54:20.164 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.164 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.165 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:20.165 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.056897+0000", 2024-02-05T04:54:20.165 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8699999999999998e-06 2024-02-05T04:54:20.165 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.165 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.165 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:20.165 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.056919+0000", 2024-02-05T04:54:20.165 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2019e-05 2024-02-05T04:54:20.165 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.165 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.165 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:20.165 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.057198+0000", 2024-02-05T04:54:20.165 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00027921000000000001 2024-02-05T04:54:20.165 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.165 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.166 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for blocked object", 2024-02-05T04:54:20.166 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.057259+0000", 2024-02-05T04:54:20.166 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1014999999999998e-05 2024-02-05T04:54:20.166 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.166 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.166 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:20.166 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.073021+0000", 2024-02-05T04:54:20.166 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.015761773999999999 2024-02-05T04:54:20.166 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.166 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.166 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:54:20.166 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.073164+0000", 2024-02-05T04:54:20.166 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00014322 2024-02-05T04:54:20.166 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.167 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.167 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:20.167 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.073713+0000", 2024-02-05T04:54:20.167 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00054833699999999998 2024-02-05T04:54:20.167 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.167 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:20.167 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.167 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.167 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.167 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:722 3.fs0 3:ffdd5ea1:::smithi19639362-31:head [append 0~0] snapc 18=[18,17,16,12,8,7] ondisk+write+known_if_redirected+supports_pool_eio e75)", 2024-02-05T04:54:20.167 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:54:07.057219+0000", 2024-02-05T04:54:20.167 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 13.08142975, 2024-02-05T04:54:20.167 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.022296369, 2024-02-05T04:54:20.167 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:20.167 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:20.168 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:54:20.168 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:54:20.168 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:20.168 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 722 2024-02-05T04:54:20.168 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.168 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:20.168 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.168 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:20.168 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.057219+0000", 2024-02-05T04:54:20.168 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:20.168 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.168 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.168 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:20.168 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.057219+0000", 2024-02-05T04:54:20.169 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:20.169 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.169 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.169 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:20.169 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.057220+0000", 2024-02-05T04:54:20.169 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6499999999999998e-07 2024-02-05T04:54:20.169 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.169 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.169 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:20.169 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.057223+0000", 2024-02-05T04:54:20.169 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.1719999999999999e-06 2024-02-05T04:54:20.169 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.169 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.169 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:20.170 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.057236+0000", 2024-02-05T04:54:20.170 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3213999999999999e-05 2024-02-05T04:54:20.170 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.170 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.170 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:20.170 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.057242+0000", 2024-02-05T04:54:20.170 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1140000000000002e-06 2024-02-05T04:54:20.170 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.170 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.170 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:20.170 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.057287+0000", 2024-02-05T04:54:20.170 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4808999999999997e-05 2024-02-05T04:54:20.170 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.170 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.171 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:54:20.171 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.057367+0000", 2024-02-05T04:54:20.171 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9940999999999999e-05 2024-02-05T04:54:20.171 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.171 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.171 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:20.171 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.058369+0000", 2024-02-05T04:54:20.171 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00100219 2024-02-05T04:54:20.171 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.171 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.171 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:20.171 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.079367+0000", 2024-02-05T04:54:20.171 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.020998036000000001 2024-02-05T04:54:20.171 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.171 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.172 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T04:54:20.172 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.079434+0000", 2024-02-05T04:54:20.172 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6197000000000003e-05 2024-02-05T04:54:20.172 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.172 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.172 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:20.172 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.079515+0000", 2024-02-05T04:54:20.172 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.1930999999999999e-05 2024-02-05T04:54:20.172 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.172 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:20.172 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.172 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.172 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.172 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:723 3.fs0 3:ffdd5ea1:::smithi19639362-31:head [rollback 12] snapc 18=[18,17,16,12,8,7] ondisk+write+known_if_redirected+supports_pool_eio e75)", 2024-02-05T04:54:20.173 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:54:07.057255+0000", 2024-02-05T04:54:20.173 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 13.081393903, 2024-02-05T04:54:20.173 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.022439265, 2024-02-05T04:54:20.173 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:20.173 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:20.173 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:54:20.173 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:54:20.173 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:20.173 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 723 2024-02-05T04:54:20.173 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.173 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:20.173 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.173 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:20.173 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.057255+0000", 2024-02-05T04:54:20.173 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:20.174 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.174 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.174 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:20.174 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.057255+0000", 2024-02-05T04:54:20.174 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:20.174 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.174 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.174 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:20.174 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.057255+0000", 2024-02-05T04:54:20.174 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0900000000000002e-07 2024-02-05T04:54:20.174 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.174 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.174 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:20.174 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.057258+0000", 2024-02-05T04:54:20.174 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1940000000000001e-06 2024-02-05T04:54:20.175 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.175 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.175 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:20.175 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.057264+0000", 2024-02-05T04:54:20.175 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0390000000000001e-06 2024-02-05T04:54:20.175 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.175 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.175 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:20.175 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.057268+0000", 2024-02-05T04:54:20.175 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5900000000000001e-06 2024-02-05T04:54:20.175 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.175 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.175 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:20.175 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.058864+0000", 2024-02-05T04:54:20.175 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001596173 2024-02-05T04:54:20.176 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.176 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.176 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:54:20.176 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.058977+0000", 2024-02-05T04:54:20.176 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000112821 2024-02-05T04:54:20.176 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.176 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.176 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:20.176 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.059498+0000", 2024-02-05T04:54:20.176 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00052045900000000003 2024-02-05T04:54:20.176 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.176 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.176 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:20.176 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.079525+0000", 2024-02-05T04:54:20.177 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.020027663000000001 2024-02-05T04:54:20.177 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.177 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.177 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T04:54:20.177 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.079582+0000", 2024-02-05T04:54:20.177 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7216000000000002e-05 2024-02-05T04:54:20.177 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.177 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.177 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:20.177 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.079694+0000", 2024-02-05T04:54:20.177 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011170100000000001 2024-02-05T04:54:20.177 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.177 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:20.177 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.177 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.178 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.178 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:724 3.fs0 3:ffdd5ea1:::smithi19639362-31:head [append 0~0] snapc 18=[18,17,16,12,8,7] ondisk+write+known_if_redirected+supports_pool_eio e75)", 2024-02-05T04:54:20.178 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:54:07.057286+0000", 2024-02-05T04:54:20.178 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 13.081362742, 2024-02-05T04:54:20.178 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.022556502999999999, 2024-02-05T04:54:20.178 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:20.178 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:20.178 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:54:20.178 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:54:20.178 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:20.178 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 724 2024-02-05T04:54:20.178 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.178 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:20.178 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.178 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:20.179 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.057286+0000", 2024-02-05T04:54:20.179 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:20.179 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.179 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.179 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:20.179 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.057286+0000", 2024-02-05T04:54:20.179 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:20.179 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.179 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.179 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:20.179 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.057287+0000", 2024-02-05T04:54:20.179 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8599999999999998e-07 2024-02-05T04:54:20.179 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.179 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.179 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:20.180 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.057289+0000", 2024-02-05T04:54:20.180 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3429999999999998e-06 2024-02-05T04:54:20.180 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.180 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.180 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:20.180 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.057296+0000", 2024-02-05T04:54:20.180 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1280000000000002e-06 2024-02-05T04:54:20.180 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.180 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.180 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:20.180 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.057305+0000", 2024-02-05T04:54:20.180 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2450000000000004e-06 2024-02-05T04:54:20.180 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.181 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.181 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:20.181 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.062292+0000", 2024-02-05T04:54:20.181 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0049867089999999998 2024-02-05T04:54:20.181 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.181 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.181 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:54:20.181 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.062412+0000", 2024-02-05T04:54:20.181 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000119503 2024-02-05T04:54:20.181 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.181 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.181 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:20.181 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.062891+0000", 2024-02-05T04:54:20.181 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00047980599999999999 2024-02-05T04:54:20.182 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.182 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.182 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:20.182 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.079730+0000", 2024-02-05T04:54:20.182 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.016838750999999999 2024-02-05T04:54:20.182 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.182 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.182 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T04:54:20.182 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.079783+0000", 2024-02-05T04:54:20.182 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2917999999999997e-05 2024-02-05T04:54:20.182 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.182 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.182 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:20.182 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.079843+0000", 2024-02-05T04:54:20.182 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9614000000000002e-05 2024-02-05T04:54:20.183 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.183 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:20.183 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.183 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.183 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.183 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.10s6 75/67 ECSubWrite(tid=98, reqid=client.4634.0:704, at_version=75'10, trim_to=0'0, roll_forward_to=75'7))", 2024-02-05T04:54:20.183 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:54:07.058966+0000", 2024-02-05T04:54:20.183 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 13.079682478000001, 2024-02-05T04:54:20.183 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.017166873999999999, 2024-02-05T04:54:20.183 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:20.183 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:20.183 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:20.183 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.183 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:20.183 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.058966+0000", 2024-02-05T04:54:20.184 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:20.184 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.184 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.184 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:20.184 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.058966+0000", 2024-02-05T04:54:20.184 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:20.184 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.184 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.184 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:20.184 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.058967+0000", 2024-02-05T04:54:20.184 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9299999999999998e-07 2024-02-05T04:54:20.184 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.184 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.184 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:20.185 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.059046+0000", 2024-02-05T04:54:20.185 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8916000000000001e-05 2024-02-05T04:54:20.185 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.185 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.185 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:20.185 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.059056+0000", 2024-02-05T04:54:20.185 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8670000000000006e-06 2024-02-05T04:54:20.185 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.185 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.185 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:20.185 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.059063+0000", 2024-02-05T04:54:20.185 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4000000000000003e-06 2024-02-05T04:54:20.185 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.185 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.186 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:20.186 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.059210+0000", 2024-02-05T04:54:20.186 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00014668600000000001 2024-02-05T04:54:20.186 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.186 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.186 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:20.186 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.059219+0000", 2024-02-05T04:54:20.186 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.0250000000000008e-06 2024-02-05T04:54:20.186 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.186 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.186 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:20.186 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.076081+0000", 2024-02-05T04:54:20.186 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.016862426999999999 2024-02-05T04:54:20.186 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.187 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.187 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:20.187 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.076133+0000", 2024-02-05T04:54:20.187 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1959999999999997e-05 2024-02-05T04:54:20.187 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.187 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:20.187 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.187 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.187 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.187 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.10s6 75/67 ECSubWrite(tid=99, reqid=client.4634.0:705, at_version=75'11, trim_to=0'0, roll_forward_to=75'7))", 2024-02-05T04:54:20.187 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:54:07.059476+0000", 2024-02-05T04:54:20.187 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 13.079173308, 2024-02-05T04:54:20.187 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.016689868, 2024-02-05T04:54:20.187 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:20.187 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:20.188 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:20.188 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.188 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:20.188 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.059476+0000", 2024-02-05T04:54:20.188 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:20.188 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.188 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.188 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:20.188 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.059476+0000", 2024-02-05T04:54:20.188 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:20.188 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.188 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.188 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:20.189 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.059476+0000", 2024-02-05T04:54:20.189 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9200000000000001e-07 2024-02-05T04:54:20.189 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.189 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.189 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:20.189 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.059490+0000", 2024-02-05T04:54:20.189 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3597e-05 2024-02-05T04:54:20.189 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.189 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.189 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:20.189 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.059497+0000", 2024-02-05T04:54:20.189 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.807e-06 2024-02-05T04:54:20.189 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.190 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.190 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:20.190 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.059504+0000", 2024-02-05T04:54:20.190 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0140000000000001e-06 2024-02-05T04:54:20.190 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.190 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.190 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:20.190 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.062215+0000", 2024-02-05T04:54:20.190 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002710161 2024-02-05T04:54:20.190 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.190 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.190 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:20.190 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.062226+0000", 2024-02-05T04:54:20.190 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1058e-05 2024-02-05T04:54:20.190 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.191 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.191 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:20.191 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.076138+0000", 2024-02-05T04:54:20.191 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.013912165000000001 2024-02-05T04:54:20.191 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.191 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.191 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:20.191 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.076165+0000", 2024-02-05T04:54:20.191 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.7574000000000001e-05 2024-02-05T04:54:20.191 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.191 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:20.191 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.191 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.191 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.192 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.17s6 75/67 ECSubWrite(tid=13, reqid=client.4634.0:713, at_version=75'39, trim_to=0'0, roll_forward_to=74'38))", 2024-02-05T04:54:20.192 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:54:07.059712+0000", 2024-02-05T04:54:20.192 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 13.078936651999999, 2024-02-05T04:54:20.192 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.020014256000000001, 2024-02-05T04:54:20.192 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:20.192 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:20.192 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:20.192 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.192 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:20.192 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.059712+0000", 2024-02-05T04:54:20.192 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:20.192 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.192 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.192 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:20.193 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.059712+0000", 2024-02-05T04:54:20.193 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:20.193 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.193 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.193 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:20.193 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.059713+0000", 2024-02-05T04:54:20.193 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8800000000000003e-07 2024-02-05T04:54:20.193 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.193 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.193 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:20.193 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.059732+0000", 2024-02-05T04:54:20.193 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9202e-05 2024-02-05T04:54:20.193 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.193 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.193 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:20.194 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.059741+0000", 2024-02-05T04:54:20.194 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6059999999999992e-06 2024-02-05T04:54:20.194 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.194 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.194 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:20.194 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.059749+0000", 2024-02-05T04:54:20.194 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1810000000000004e-06 2024-02-05T04:54:20.194 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.194 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.194 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:20.194 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.062093+0000", 2024-02-05T04:54:20.194 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0023447329999999999 2024-02-05T04:54:20.194 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.194 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.194 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:20.195 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.062103+0000", 2024-02-05T04:54:20.195 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.4849999999999998e-06 2024-02-05T04:54:20.195 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.195 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.195 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:20.195 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.079696+0000", 2024-02-05T04:54:20.195 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.017593320999999999 2024-02-05T04:54:20.195 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.195 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.195 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:20.195 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.079726+0000", 2024-02-05T04:54:20.195 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.0239999999999998e-05 2024-02-05T04:54:20.195 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.195 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:20.196 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.196 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.196 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.196 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.1s1 75/62 ECSubWrite(tid=362, reqid=client.4634.0:707, at_version=75'82, trim_to=0'0, roll_forward_to=74'80))", 2024-02-05T04:54:20.196 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:54:07.059721+0000", 2024-02-05T04:54:20.196 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 13.078928084999999, 2024-02-05T04:54:20.196 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.016900801, 2024-02-05T04:54:20.196 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:20.196 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:20.196 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:20.196 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.196 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:20.196 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.059721+0000", 2024-02-05T04:54:20.196 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:20.197 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.197 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.197 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:20.197 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.059721+0000", 2024-02-05T04:54:20.197 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:20.197 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.197 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.197 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:20.197 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.059721+0000", 2024-02-05T04:54:20.197 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1e-07 2024-02-05T04:54:20.197 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.197 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.197 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:20.197 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.059767+0000", 2024-02-05T04:54:20.198 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5561e-05 2024-02-05T04:54:20.198 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.198 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.198 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:20.198 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.059774+0000", 2024-02-05T04:54:20.198 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6599999999999995e-06 2024-02-05T04:54:20.198 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.198 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.198 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:20.198 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.059780+0000", 2024-02-05T04:54:20.198 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.015e-06 2024-02-05T04:54:20.198 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.198 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.198 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:20.198 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.059836+0000", 2024-02-05T04:54:20.199 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5784e-05 2024-02-05T04:54:20.199 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.199 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.199 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:20.199 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.059844+0000", 2024-02-05T04:54:20.199 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4770000000000003e-06 2024-02-05T04:54:20.199 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.199 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.199 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:20.199 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.076593+0000", 2024-02-05T04:54:20.199 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.016748906000000001 2024-02-05T04:54:20.199 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.199 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.200 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:20.200 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.076622+0000", 2024-02-05T04:54:20.200 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8187999999999998e-05 2024-02-05T04:54:20.200 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.200 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:20.200 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.200 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.200 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.200 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.17s6 75/67 ECSubWrite(tid=14, reqid=client.4634.0:714, at_version=75'40, trim_to=0'0, roll_forward_to=74'38))", 2024-02-05T04:54:20.200 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:54:07.061386+0000", 2024-02-05T04:54:20.200 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 13.077262863, 2024-02-05T04:54:20.200 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.018481184000000001, 2024-02-05T04:54:20.200 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:20.201 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:20.201 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:20.201 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.201 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:20.201 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.061386+0000", 2024-02-05T04:54:20.201 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:20.201 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.201 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.201 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:20.201 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.061386+0000", 2024-02-05T04:54:20.201 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:20.201 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.201 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.201 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:20.201 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.061387+0000", 2024-02-05T04:54:20.202 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9599999999999999e-07 2024-02-05T04:54:20.202 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.202 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.202 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:20.202 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.061404+0000", 2024-02-05T04:54:20.202 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6974e-05 2024-02-05T04:54:20.202 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.202 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.202 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:20.202 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.061419+0000", 2024-02-05T04:54:20.202 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5145e-05 2024-02-05T04:54:20.202 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.202 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.202 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:20.203 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.061426+0000", 2024-02-05T04:54:20.203 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6639999999999998e-06 2024-02-05T04:54:20.203 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.203 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.203 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:20.203 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.063191+0000", 2024-02-05T04:54:20.203 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001764484 2024-02-05T04:54:20.203 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.203 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.203 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:20.203 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.063200+0000", 2024-02-05T04:54:20.203 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5119999999999997e-06 2024-02-05T04:54:20.203 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.203 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.204 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:20.204 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.079845+0000", 2024-02-05T04:54:20.204 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.016644097 2024-02-05T04:54:20.204 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.204 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.204 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:20.204 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.079867+0000", 2024-02-05T04:54:20.204 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2612e-05 2024-02-05T04:54:20.204 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.204 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:20.204 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.204 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.204 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.205 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.17s6 75/67 ECSubWrite(tid=15, reqid=client.4634.0:715, at_version=75'41, trim_to=0'0, roll_forward_to=74'38))", 2024-02-05T04:54:20.205 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:54:07.063374+0000", 2024-02-05T04:54:20.205 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 13.075275099000001, 2024-02-05T04:54:20.205 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.016512723, 2024-02-05T04:54:20.205 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:20.205 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:20.205 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:20.205 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.205 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:20.205 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.063374+0000", 2024-02-05T04:54:20.205 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:20.205 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.205 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.205 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:20.206 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.063374+0000", 2024-02-05T04:54:20.206 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:20.206 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.206 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.206 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:20.206 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.063374+0000", 2024-02-05T04:54:20.206 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4799999999999998e-07 2024-02-05T04:54:20.206 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.206 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.206 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:20.206 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.063403+0000", 2024-02-05T04:54:20.206 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8705999999999999e-05 2024-02-05T04:54:20.206 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.206 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.207 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:20.207 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.063417+0000", 2024-02-05T04:54:20.207 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.383e-05 2024-02-05T04:54:20.207 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.207 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.207 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:20.207 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.063426+0000", 2024-02-05T04:54:20.207 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.5839999999999995e-06 2024-02-05T04:54:20.207 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.207 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.207 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:20.207 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.063598+0000", 2024-02-05T04:54:20.207 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00017286100000000001 2024-02-05T04:54:20.207 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.207 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.208 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:20.208 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.063606+0000", 2024-02-05T04:54:20.208 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7000000000000008e-06 2024-02-05T04:54:20.208 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.208 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.208 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:20.208 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.079870+0000", 2024-02-05T04:54:20.208 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.016264015999999999 2024-02-05T04:54:20.208 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.208 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.208 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:20.208 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.079886+0000", 2024-02-05T04:54:20.208 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6378000000000001e-05 2024-02-05T04:54:20.208 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.208 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:20.209 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.209 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.209 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.209 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.17s6 75/67 ECSubWrite(tid=16, reqid=osd.0.0:16, at_version=75'43, trim_to=0'0, roll_forward_to=74'38))", 2024-02-05T04:54:20.209 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:54:07.063441+0000", 2024-02-05T04:54:20.209 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 13.07520819, 2024-02-05T04:54:20.209 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.016472174999999999, 2024-02-05T04:54:20.209 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:20.209 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:20.209 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:20.209 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.209 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:20.209 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.063441+0000", 2024-02-05T04:54:20.209 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:20.210 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.210 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.210 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:20.210 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.063441+0000", 2024-02-05T04:54:20.210 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:20.210 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.210 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.210 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:20.210 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.063441+0000", 2024-02-05T04:54:20.210 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.67e-07 2024-02-05T04:54:20.210 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.210 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.210 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:20.210 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.063463+0000", 2024-02-05T04:54:20.211 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1599000000000001e-05 2024-02-05T04:54:20.211 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.211 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.211 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:20.211 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.063469+0000", 2024-02-05T04:54:20.211 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8820000000000003e-06 2024-02-05T04:54:20.211 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.211 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.211 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:20.211 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.063475+0000", 2024-02-05T04:54:20.211 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8359999999999997e-06 2024-02-05T04:54:20.211 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.211 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.211 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:20.211 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.063770+0000", 2024-02-05T04:54:20.212 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00029445600000000001 2024-02-05T04:54:20.212 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.212 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.212 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:20.212 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.063779+0000", 2024-02-05T04:54:20.212 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2369999999999998e-06 2024-02-05T04:54:20.212 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.212 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.212 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:20.212 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.079890+0000", 2024-02-05T04:54:20.212 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.016111223000000001 2024-02-05T04:54:20.212 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.212 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.212 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:20.213 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.079913+0000", 2024-02-05T04:54:20.213 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2674999999999999e-05 2024-02-05T04:54:20.213 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.213 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:20.213 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.213 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.213 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.213 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:731 3.fs0 3:f8d01528:::smithi19639362-17:head [append 0~589824] snapc 18=[18,17,16,12,8,7] ondisk+write+known_if_redirected+supports_pool_eio e75)", 2024-02-05T04:54:20.213 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:54:07.070323+0000", 2024-02-05T04:54:20.213 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 13.068326071, 2024-02-05T04:54:20.213 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.014336896, 2024-02-05T04:54:20.213 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:20.213 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:20.213 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:54:20.213 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:54:20.214 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:20.214 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 731 2024-02-05T04:54:20.214 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.214 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:20.214 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.214 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:20.214 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.070323+0000", 2024-02-05T04:54:20.214 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:20.214 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.214 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.214 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:20.214 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.070323+0000", 2024-02-05T04:54:20.214 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:20.214 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.214 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.215 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:20.215 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.070324+0000", 2024-02-05T04:54:20.215 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6419999999999999e-06 2024-02-05T04:54:20.215 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.215 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.215 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:20.215 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.070703+0000", 2024-02-05T04:54:20.215 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00037887499999999998 2024-02-05T04:54:20.215 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.215 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.215 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:20.215 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.070723+0000", 2024-02-05T04:54:20.215 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9354000000000001e-05 2024-02-05T04:54:20.215 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.215 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.215 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:20.216 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.070751+0000", 2024-02-05T04:54:20.216 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8792000000000001e-05 2024-02-05T04:54:20.216 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.216 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.216 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:20.216 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.074099+0000", 2024-02-05T04:54:20.216 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0033476270000000002 2024-02-05T04:54:20.216 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.216 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.216 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:54:20.216 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.074218+0000", 2024-02-05T04:54:20.216 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011851200000000001 2024-02-05T04:54:20.216 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.216 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.217 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:20.217 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.075492+0000", 2024-02-05T04:54:20.217 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001274327 2024-02-05T04:54:20.217 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.217 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.217 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:20.217 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.083419+0000", 2024-02-05T04:54:20.217 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0079274770000000005 2024-02-05T04:54:20.217 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.217 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.217 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T04:54:20.217 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.084591+0000", 2024-02-05T04:54:20.217 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001171144 2024-02-05T04:54:20.217 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.217 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.218 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:20.218 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.084660+0000", 2024-02-05T04:54:20.218 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9146000000000005e-05 2024-02-05T04:54:20.218 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.218 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:20.218 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.218 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.218 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.218 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:732 3.fs0 3:f8d01528:::smithi19639362-17:head [append 0~753664] snapc 18=[18,17,16,12,8,7] ondisk+write+known_if_redirected+supports_pool_eio e75)", 2024-02-05T04:54:20.218 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:54:07.075216+0000", 2024-02-05T04:54:20.218 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 13.063432383, 2024-02-05T04:54:20.218 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.012269877, 2024-02-05T04:54:20.218 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:20.218 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:20.218 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:54:20.219 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:54:20.219 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:20.219 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 732 2024-02-05T04:54:20.219 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.219 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:20.219 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.219 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:20.219 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.075216+0000", 2024-02-05T04:54:20.219 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:20.219 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.219 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.219 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:20.219 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.075216+0000", 2024-02-05T04:54:20.219 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:20.219 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.220 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.220 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:20.220 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.075219+0000", 2024-02-05T04:54:20.220 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3089999999999998e-06 2024-02-05T04:54:20.220 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.220 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.220 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:20.220 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.075622+0000", 2024-02-05T04:54:20.220 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00040325300000000001 2024-02-05T04:54:20.220 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.220 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.220 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:20.220 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.075636+0000", 2024-02-05T04:54:20.220 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3967e-05 2024-02-05T04:54:20.220 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.221 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.221 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:20.221 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.075645+0000", 2024-02-05T04:54:20.221 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.5299999999999996e-06 2024-02-05T04:54:20.221 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.221 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.221 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:20.221 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.077207+0000", 2024-02-05T04:54:20.221 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00156277 2024-02-05T04:54:20.221 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.221 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.221 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:54:20.221 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.077312+0000", 2024-02-05T04:54:20.221 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000104658 2024-02-05T04:54:20.221 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.222 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.222 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:20.222 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.078604+0000", 2024-02-05T04:54:20.222 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0012916939999999999 2024-02-05T04:54:20.222 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.222 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.222 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:20.222 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.083421+0000", 2024-02-05T04:54:20.222 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0048170519999999996 2024-02-05T04:54:20.222 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.222 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.222 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T04:54:20.222 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.087435+0000", 2024-02-05T04:54:20.222 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0040147799999999999 2024-02-05T04:54:20.222 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.222 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.223 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:20.223 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.087486+0000", 2024-02-05T04:54:20.223 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0864e-05 2024-02-05T04:54:20.223 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.223 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:20.223 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.223 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.223 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.223 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:733 3.fs0 3:f8d01528:::smithi19639362-17:head [append 0~475136] snapc 18=[18,17,16,12,8,7] ondisk+write+known_if_redirected+supports_pool_eio e75)", 2024-02-05T04:54:20.223 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:54:07.078157+0000", 2024-02-05T04:54:20.223 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 13.060492182999999, 2024-02-05T04:54:20.223 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.011443841999999999, 2024-02-05T04:54:20.223 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:20.223 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:20.224 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:54:20.224 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:54:20.224 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:20.224 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 733 2024-02-05T04:54:20.224 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.224 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:20.224 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.224 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:20.224 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.078157+0000", 2024-02-05T04:54:20.224 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:20.224 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.224 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.224 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:20.224 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.078157+0000", 2024-02-05T04:54:20.225 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:20.225 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.225 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.225 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:20.225 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.078159+0000", 2024-02-05T04:54:20.225 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9309999999999998e-06 2024-02-05T04:54:20.225 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.225 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.225 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:20.225 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.078533+0000", 2024-02-05T04:54:20.225 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000374476 2024-02-05T04:54:20.225 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.225 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.225 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:20.225 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.078547+0000", 2024-02-05T04:54:20.226 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3531e-05 2024-02-05T04:54:20.226 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.226 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.226 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:20.226 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.078556+0000", 2024-02-05T04:54:20.226 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1989999999999995e-06 2024-02-05T04:54:20.226 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.226 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.226 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:20.226 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.080076+0000", 2024-02-05T04:54:20.226 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0015201489999999999 2024-02-05T04:54:20.226 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.226 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.226 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:54:20.226 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.080187+0000", 2024-02-05T04:54:20.227 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011131799999999999 2024-02-05T04:54:20.227 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.227 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.227 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:20.227 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.081013+0000", 2024-02-05T04:54:20.227 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00082612600000000005 2024-02-05T04:54:20.227 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.227 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.227 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:20.227 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.085831+0000", 2024-02-05T04:54:20.227 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0048172730000000004 2024-02-05T04:54:20.227 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.227 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.227 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T04:54:20.227 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.089535+0000", 2024-02-05T04:54:20.228 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0037040269999999999 2024-02-05T04:54:20.228 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.228 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.228 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:20.228 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.089600+0000", 2024-02-05T04:54:20.228 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5811999999999999e-05 2024-02-05T04:54:20.228 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.228 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:20.228 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.228 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.228 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.228 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:736 3.fs0 3:f8d01528:::smithi19639362-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e75)", 2024-02-05T04:54:20.228 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:54:07.080280+0000", 2024-02-05T04:54:20.228 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 13.058368824, 2024-02-05T04:54:20.228 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.011099471, 2024-02-05T04:54:20.229 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:20.229 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-02-05T04:54:20.229 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:54:20.229 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:54:20.229 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:20.229 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 736 2024-02-05T04:54:20.229 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.229 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:20.229 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.229 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:20.229 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.080280+0000", 2024-02-05T04:54:20.229 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:20.229 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.229 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.229 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:20.230 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.080280+0000", 2024-02-05T04:54:20.230 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:20.230 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.230 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.230 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:20.230 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.080281+0000", 2024-02-05T04:54:20.230 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9999999999999998e-07 2024-02-05T04:54:20.230 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.230 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.230 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:20.230 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.080285+0000", 2024-02-05T04:54:20.230 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0659999999999997e-06 2024-02-05T04:54:20.230 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.230 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.230 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:20.230 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.080292+0000", 2024-02-05T04:54:20.231 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5340000000000002e-06 2024-02-05T04:54:20.231 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.231 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.231 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:20.231 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.080298+0000", 2024-02-05T04:54:20.231 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6119999999999998e-06 2024-02-05T04:54:20.231 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.231 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.231 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:20.231 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.085738+0000", 2024-02-05T04:54:20.231 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0054405139999999996 2024-02-05T04:54:20.231 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.231 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.231 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for rw locks", 2024-02-05T04:54:20.231 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.085823+0000", 2024-02-05T04:54:20.232 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4480000000000004e-05 2024-02-05T04:54:20.232 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.232 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.232 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:20.232 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.090407+0000", 2024-02-05T04:54:20.232 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0045846000000000003 2024-02-05T04:54:20.232 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.232 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.232 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:54:20.232 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.090505+0000", 2024-02-05T04:54:20.232 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.7862000000000006e-05 2024-02-05T04:54:20.232 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.232 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.232 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:20.232 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.091379+0000", 2024-02-05T04:54:20.233 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00087430299999999999 2024-02-05T04:54:20.233 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.233 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:20.233 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.233 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.233 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.233 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.19s2 84/67 ECSubWrite(tid=20, reqid=client.4634.0:852, at_version=84'41, trim_to=0'0, roll_forward_to=81'39))", 2024-02-05T04:54:20.233 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:54:16.114588+0000", 2024-02-05T04:54:20.233 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 4.0240609620000001, 2024-02-05T04:54:20.233 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.015076984, 2024-02-05T04:54:20.233 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:20.233 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:20.233 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:20.233 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.233 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:20.234 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.114588+0000", 2024-02-05T04:54:20.234 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:20.234 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.234 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.234 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:20.234 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.114588+0000", 2024-02-05T04:54:20.234 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:20.234 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.234 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.234 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:20.234 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.114589+0000", 2024-02-05T04:54:20.234 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5900000000000005e-07 2024-02-05T04:54:20.234 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.234 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.234 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:20.235 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.115555+0000", 2024-02-05T04:54:20.235 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00096632200000000004 2024-02-05T04:54:20.235 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.235 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.235 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:20.235 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.115573+0000", 2024-02-05T04:54:20.235 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7728e-05 2024-02-05T04:54:20.235 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.235 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.235 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:20.235 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.115584+0000", 2024-02-05T04:54:20.235 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.08e-05 2024-02-05T04:54:20.235 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.235 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.235 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:20.236 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.115667+0000", 2024-02-05T04:54:20.236 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3061999999999999e-05 2024-02-05T04:54:20.236 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.236 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.236 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:20.236 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.115678+0000", 2024-02-05T04:54:20.236 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1633e-05 2024-02-05T04:54:20.236 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.236 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.236 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:20.236 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.129621+0000", 2024-02-05T04:54:20.236 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.013942718 2024-02-05T04:54:20.236 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.236 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.236 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:20.237 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.129665+0000", 2024-02-05T04:54:20.237 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3761999999999999e-05 2024-02-05T04:54:20.237 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.237 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:20.237 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.237 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.237 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:20.237 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T04:54:20.239 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-02-05T04:54:20.239 INFO:tasks.rados.rados.0.smithi196.stdout:398: expect (ObjNum 148 snap 28 seq_num 148) 2024-02-05T04:54:20.239 INFO:tasks.rados.rados.0.smithi196.stdout:398: done (1 left) 2024-02-05T04:54:20.239 INFO:tasks.rados.rados.0.smithi196.stdout:399: done (0 left) 2024-02-05T04:54:20.240 INFO:tasks.ceph.osd.14.smithi150.stderr:2024-02-05T04:54:20.237+0000 7f5f81083640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 34020) UID: 0 2024-02-05T04:54:20.240 INFO:tasks.rados.rados.0.smithi196.stderr:0 errors. 2024-02-05T04:54:20.240 INFO:tasks.rados.rados.0.smithi196.stderr: 2024-02-05T04:54:20.244 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-05T04:54:20.259 INFO:teuthology.orchestra.run.smithi091.stdout:get_pg_num_history pg_num_history pg_num_history(e80 pg_nums {1={23=1},2={26=8},3={31=16,62=26}} deleted_pools ) 2024-02-05T04:54:20.259 INFO:teuthology.orchestra.run.smithi091.stdout:Importing pgid 3.7s0 2024-02-05T04:54:20.259 INFO:teuthology.orchestra.run.smithi091.stdout:Write 0#3:e54c8e71:::smithi19639362-27:c# 2024-02-05T04:54:20.260 INFO:teuthology.orchestra.run.smithi091.stdout:Write 0#3:e54c8e71:::smithi19639362-27:1b# 2024-02-05T04:54:20.262 INFO:teuthology.orchestra.run.smithi091.stdout:Write 0#3:e54c8e71:::smithi19639362-27:head# 2024-02-05T04:54:20.262 INFO:teuthology.orchestra.run.smithi091.stdout:snapset 1b=[]:{c=[8,7],1b=[1b,19,18,17,16,12]} 2024-02-05T04:54:20.264 INFO:teuthology.orchestra.run.smithi091.stdout:write_pg epoch 79 info 3.7s0( v 79'38 (0'0,79'38] local-lis/les=62/63 n=3 ec=31/31 lis/c=62/62 les/c/f=63/63/0 sis=62) 2024-02-05T04:54:20.264 INFO:teuthology.orchestra.run.smithi091.stderr:*** Caught signal (Segmentation fault) ** 2024-02-05T04:54:20.265 INFO:teuthology.orchestra.run.smithi091.stderr: in thread 7f5e4ce18400 thread_name:ceph-objectstor 2024-02-05T04:54:20.265 INFO:teuthology.orchestra.run.smithi091.stderr: ceph version 19.0.0-658-gd4ed20cf (d4ed20cfb238ef2c47d64ca4533eca00b8968b98) squid (dev) 2024-02-05T04:54:20.265 INFO:teuthology.orchestra.run.smithi091.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f5e4cc54db0] 2024-02-05T04:54:20.265 INFO:teuthology.orchestra.run.smithi091.stderr: 2: (PGLog::_write_log_and_missing(ceph::common::CephContext*, ceph::os::Transaction&, std::map, std::allocator >, ceph::buffer::v15_2_0::list, std::less, std::allocator > >, std::allocator, std::allocator > const, ceph::buffer::v15_2_0::list> > >*, pg_log_t&, coll_t const&, ghobject_t const&, eversion_t, eversion_t, eversion_t, std::set, std::allocator >&&, std::set, std::allocator >, std::less, std::allocator > >, std::allocator, std::allocator > > >&&, pg_missing_set const&, bool, bool, bool, eversion_t, eversion_t, eversion_t, bool*, std::set, std::allocator >, std::less, std::allocator > >, std::allocator, std::allocator > > >*, DoutPrefixProvider const*)+0x8d5) [0x565316495b75] 2024-02-05T04:54:20.265 INFO:teuthology.orchestra.run.smithi091.stderr: 3: (PGLog::write_log_and_missing(ceph::common::CephContext*, ceph::os::Transaction&, std::map, std::allocator >, ceph::buffer::v15_2_0::list, std::less, std::allocator > >, std::allocator, std::allocator > const, ceph::buffer::v15_2_0::list> > >*, pg_log_t&, coll_t const&, ghobject_t const&, pg_missing_set const&, bool, bool*, DoutPrefixProvider const*)+0x178) [0x565316497e68] 2024-02-05T04:54:20.266 INFO:teuthology.orchestra.run.smithi091.stderr: 4: (write_pg(ceph::os::Transaction&, unsigned int, pg_info_t&, pg_log_t&, PastIntervals&, std::map, std::allocator > >&, pg_missing_set&)+0x4f9) [0x565316339b19] 2024-02-05T04:54:20.266 INFO:teuthology.orchestra.run.smithi091.stderr: 5: (ObjectStoreTool::do_import(ObjectStore*, OSDSuperblock&, bool, std::__cxx11::basic_string, std::allocator >)+0x1e2a) [0x565316350e2a] 2024-02-05T04:54:20.266 INFO:teuthology.orchestra.run.smithi091.stderr: 6: main() 2024-02-05T04:54:20.266 INFO:teuthology.orchestra.run.smithi091.stderr: 7: /lib64/libc.so.6(+0x3feb0) [0x7f5e4cc3feb0] 2024-02-05T04:54:20.266 INFO:teuthology.orchestra.run.smithi091.stderr: 8: __libc_start_main() 2024-02-05T04:54:20.266 INFO:teuthology.orchestra.run.smithi091.stderr: 9: _start() 2024-02-05T04:54:20.266 INFO:teuthology.orchestra.run.smithi091.stderr:2024-02-05T04:54:20.264+0000 7f5e4ce18400 -1 *** Caught signal (Segmentation fault) ** 2024-02-05T04:54:20.266 INFO:teuthology.orchestra.run.smithi091.stderr: in thread 7f5e4ce18400 thread_name:ceph-objectstor 2024-02-05T04:54:20.266 INFO:teuthology.orchestra.run.smithi091.stderr: 2024-02-05T04:54:20.266 INFO:teuthology.orchestra.run.smithi091.stderr: ceph version 19.0.0-658-gd4ed20cf (d4ed20cfb238ef2c47d64ca4533eca00b8968b98) squid (dev) 2024-02-05T04:54:20.266 INFO:teuthology.orchestra.run.smithi091.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f5e4cc54db0] 2024-02-05T04:54:20.266 INFO:teuthology.orchestra.run.smithi091.stderr: 2: (PGLog::_write_log_and_missing(ceph::common::CephContext*, ceph::os::Transaction&, std::map, std::allocator >, ceph::buffer::v15_2_0::list, std::less, std::allocator > >, std::allocator, std::allocator > const, ceph::buffer::v15_2_0::list> > >*, pg_log_t&, coll_t const&, ghobject_t const&, eversion_t, eversion_t, eversion_t, std::set, std::allocator >&&, std::set, std::allocator >, std::less, std::allocator > >, std::allocator, std::allocator > > >&&, pg_missing_set const&, bool, bool, bool, eversion_t, eversion_t, eversion_t, bool*, std::set, std::allocator >, std::less, std::allocator > >, std::allocator, std::allocator > > >*, DoutPrefixProvider const*)+0x8d5) [0x565316495b75] 2024-02-05T04:54:20.266 INFO:teuthology.orchestra.run.smithi091.stderr: 3: (PGLog::write_log_and_missing(ceph::common::CephContext*, ceph::os::Transaction&, std::map, std::allocator >, ceph::buffer::v15_2_0::list, std::less, std::allocator > >, std::allocator, std::allocator > const, ceph::buffer::v15_2_0::list> > >*, pg_log_t&, coll_t const&, ghobject_t const&, pg_missing_set const&, bool, bool*, DoutPrefixProvider const*)+0x178) [0x565316497e68] 2024-02-05T04:54:20.267 INFO:teuthology.orchestra.run.smithi091.stderr: 4: (write_pg(ceph::os::Transaction&, unsigned int, pg_info_t&, pg_log_t&, PastIntervals&, std::map, std::allocator > >&, pg_missing_set&)+0x4f9) [0x565316339b19] 2024-02-05T04:54:20.267 INFO:teuthology.orchestra.run.smithi091.stderr: 5: (ObjectStoreTool::do_import(ObjectStore*, OSDSuperblock&, bool, std::__cxx11::basic_string, std::allocator >)+0x1e2a) [0x565316350e2a] 2024-02-05T04:54:20.267 INFO:teuthology.orchestra.run.smithi091.stderr: 6: main() 2024-02-05T04:54:20.267 INFO:teuthology.orchestra.run.smithi091.stderr: 7: /lib64/libc.so.6(+0x3feb0) [0x7f5e4cc3feb0] 2024-02-05T04:54:20.267 INFO:teuthology.orchestra.run.smithi091.stderr: 8: __libc_start_main() 2024-02-05T04:54:20.267 INFO:teuthology.orchestra.run.smithi091.stderr: 9: _start() 2024-02-05T04:54:20.267 INFO:teuthology.orchestra.run.smithi091.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-02-05T04:54:20.267 INFO:teuthology.orchestra.run.smithi091.stderr: 2024-02-05T04:54:20.267 INFO:teuthology.orchestra.run.smithi091.stderr: 0> 2024-02-05T04:54:20.264+0000 7f5e4ce18400 -1 *** Caught signal (Segmentation fault) ** 2024-02-05T04:54:20.268 INFO:teuthology.orchestra.run.smithi091.stderr: in thread 7f5e4ce18400 thread_name:ceph-objectstor 2024-02-05T04:54:20.268 INFO:teuthology.orchestra.run.smithi091.stderr: 2024-02-05T04:54:20.268 INFO:teuthology.orchestra.run.smithi091.stderr: ceph version 19.0.0-658-gd4ed20cf (d4ed20cfb238ef2c47d64ca4533eca00b8968b98) squid (dev) 2024-02-05T04:54:20.268 INFO:teuthology.orchestra.run.smithi091.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f5e4cc54db0] 2024-02-05T04:54:20.268 INFO:teuthology.orchestra.run.smithi091.stderr: 2: (PGLog::_write_log_and_missing(ceph::common::CephContext*, ceph::os::Transaction&, std::map, std::allocator >, ceph::buffer::v15_2_0::list, std::less, std::allocator > >, std::allocator, std::allocator > const, ceph::buffer::v15_2_0::list> > >*, pg_log_t&, coll_t const&, ghobject_t const&, eversion_t, eversion_t, eversion_t, std::set, std::allocator >&&, std::set, std::allocator >, std::less, std::allocator > >, std::allocator, std::allocator > > >&&, pg_missing_set const&, bool, bool, bool, eversion_t, eversion_t, eversion_t, bool*, std::set, std::allocator >, std::less, std::allocator > >, std::allocator, std::allocator > > >*, DoutPrefixProvider const*)+0x8d5) [0x565316495b75] 2024-02-05T04:54:20.268 INFO:teuthology.orchestra.run.smithi091.stderr: 3: (PGLog::write_log_and_missing(ceph::common::CephContext*, ceph::os::Transaction&, std::map, std::allocator >, ceph::buffer::v15_2_0::list, std::less, std::allocator > >, std::allocator, std::allocator > const, ceph::buffer::v15_2_0::list> > >*, pg_log_t&, coll_t const&, ghobject_t const&, pg_missing_set const&, bool, bool*, DoutPrefixProvider const*)+0x178) [0x565316497e68] 2024-02-05T04:54:20.268 INFO:teuthology.orchestra.run.smithi091.stderr: 4: (write_pg(ceph::os::Transaction&, unsigned int, pg_info_t&, pg_log_t&, PastIntervals&, std::map, std::allocator > >&, pg_missing_set&)+0x4f9) [0x565316339b19] 2024-02-05T04:54:20.268 INFO:teuthology.orchestra.run.smithi091.stderr: 5: (ObjectStoreTool::do_import(ObjectStore*, OSDSuperblock&, bool, std::__cxx11::basic_string, std::allocator >)+0x1e2a) [0x565316350e2a] 2024-02-05T04:54:20.268 INFO:teuthology.orchestra.run.smithi091.stderr: 6: main() 2024-02-05T04:54:20.268 INFO:teuthology.orchestra.run.smithi091.stderr: 7: /lib64/libc.so.6(+0x3feb0) [0x7f5e4cc3feb0] 2024-02-05T04:54:20.268 INFO:teuthology.orchestra.run.smithi091.stderr: 8: __libc_start_main() 2024-02-05T04:54:20.268 INFO:teuthology.orchestra.run.smithi091.stderr: 9: _start() 2024-02-05T04:54:20.268 INFO:teuthology.orchestra.run.smithi091.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-02-05T04:54:20.269 INFO:teuthology.orchestra.run.smithi091.stderr: 2024-02-05T04:54:20.269 INFO:teuthology.orchestra.run.smithi091.stderr: 0> 2024-02-05T04:54:20.264+0000 7f5e4ce18400 -1 *** Caught signal (Segmentation fault) ** 2024-02-05T04:54:20.269 INFO:teuthology.orchestra.run.smithi091.stderr: in thread 7f5e4ce18400 thread_name:ceph-objectstor 2024-02-05T04:54:20.269 INFO:teuthology.orchestra.run.smithi091.stderr: 2024-02-05T04:54:20.269 INFO:teuthology.orchestra.run.smithi091.stderr: ceph version 19.0.0-658-gd4ed20cf (d4ed20cfb238ef2c47d64ca4533eca00b8968b98) squid (dev) 2024-02-05T04:54:20.269 INFO:teuthology.orchestra.run.smithi091.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f5e4cc54db0] 2024-02-05T04:54:20.269 INFO:teuthology.orchestra.run.smithi091.stderr: 2: (PGLog::_write_log_and_missing(ceph::common::CephContext*, ceph::os::Transaction&, std::map, std::allocator >, ceph::buffer::v15_2_0::list, std::less, std::allocator > >, std::allocator, std::allocator > const, ceph::buffer::v15_2_0::list> > >*, pg_log_t&, coll_t const&, ghobject_t const&, eversion_t, eversion_t, eversion_t, std::set, std::allocator >&&, std::set, std::allocator >, std::less, std::allocator > >, std::allocator, std::allocator > > >&&, pg_missing_set const&, bool, bool, bool, eversion_t, eversion_t, eversion_t, bool*, std::set, std::allocator >, std::less, std::allocator > >, std::allocator, std::allocator > > >*, DoutPrefixProvider const*)+0x8d5) [0x565316495b75] 2024-02-05T04:54:20.269 INFO:teuthology.orchestra.run.smithi091.stderr: 3: (PGLog::write_log_and_missing(ceph::common::CephContext*, ceph::os::Transaction&, std::map, std::allocator >, ceph::buffer::v15_2_0::list, std::less, std::allocator > >, std::allocator, std::allocator > const, ceph::buffer::v15_2_0::list> > >*, pg_log_t&, coll_t const&, ghobject_t const&, pg_missing_set const&, bool, bool*, DoutPrefixProvider const*)+0x178) [0x565316497e68] 2024-02-05T04:54:20.269 INFO:teuthology.orchestra.run.smithi091.stderr: 4: (write_pg(ceph::os::Transaction&, unsigned int, pg_info_t&, pg_log_t&, PastIntervals&, std::map, std::allocator > >&, pg_missing_set&)+0x4f9) [0x565316339b19] 2024-02-05T04:54:20.269 INFO:teuthology.orchestra.run.smithi091.stderr: 5: (ObjectStoreTool::do_import(ObjectStore*, OSDSuperblock&, bool, std::__cxx11::basic_string, std::allocator >)+0x1e2a) [0x565316350e2a] 2024-02-05T04:54:20.269 INFO:teuthology.orchestra.run.smithi091.stderr: 6: main() 2024-02-05T04:54:20.269 INFO:teuthology.orchestra.run.smithi091.stderr: 7: /lib64/libc.so.6(+0x3feb0) [0x7f5e4cc3feb0] 2024-02-05T04:54:20.269 INFO:teuthology.orchestra.run.smithi091.stderr: 8: __libc_start_main() 2024-02-05T04:54:20.269 INFO:teuthology.orchestra.run.smithi091.stderr: 9: _start() 2024-02-05T04:54:20.269 INFO:teuthology.orchestra.run.smithi091.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-02-05T04:54:20.270 INFO:teuthology.orchestra.run.smithi091.stderr: 2024-02-05T04:54:20.339 INFO:tasks.ceph.osd.10.smithi150.stderr:2024-02-05T04:54:20.339+0000 7f3075a4d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 34018) UID: 0 2024-02-05T04:54:20.345 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T04:54:20.345 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-02-05T04:54:20.345 INFO:teuthology.orchestra.run.smithi196.stdout: "num_ops": 0 2024-02-05T04:54:20.345 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T04:54:20.354 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-02-05T04:54:20.400 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-05T04:54:20.401 INFO:tasks.thrashosds.thrasher:Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/ceph_manager.py", line 190, in wrapper return func(self) File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/ceph_manager.py", line 1433, in _do_thrash self.choose_action()() File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/ceph_manager.py", line 1168, in test_bluestore_reshard self.kill_osd(mark_down=True, mark_out=True) File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/ceph_manager.py", line 531, in kill_osd raise Exception("ceph-objectstore-tool: " Exception: ceph-objectstore-tool: import failure with status -1 2024-02-05T04:54:20.402 ERROR:tasks.thrashosds.thrasher:exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/ceph_manager.py", line 1301, in do_thrash self._do_thrash() File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/ceph_manager.py", line 190, in wrapper return func(self) File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/ceph_manager.py", line 1433, in _do_thrash self.choose_action()() File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/ceph_manager.py", line 1168, in test_bluestore_reshard self.kill_osd(mark_down=True, mark_out=True) File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/ceph_manager.py", line 531, in kill_osd raise Exception("ceph-objectstore-tool: " Exception: ceph-objectstore-tool: import failure with status -1 2024-02-05T04:54:20.440 INFO:tasks.ceph.osd.10.smithi150.stderr:2024-02-05T04:54:20.440+0000 7f3075a4d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 34018) UID: 0 2024-02-05T04:54:20.459 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T04:54:20.459 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-02-05T04:54:20.460 INFO:teuthology.orchestra.run.smithi196.stdout: "complaint_time": 30, 2024-02-05T04:54:20.460 INFO:teuthology.orchestra.run.smithi196.stdout: "num_blocked_ops": 0 2024-02-05T04:54:20.460 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T04:54:20.468 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-02-05T04:54:20.541 INFO:tasks.ceph.osd.4.smithi062.stderr:2024-02-05T04:54:20.540+0000 7f708c308640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34689) UID: 0 2024-02-05T04:54:20.543 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:54:20.543 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":88,"fsid":"88c11a2d-382e-4f21-b8af-45ab9c504b93","created":"2024-02-05T04:52:55.182961+0000","modified":"2024-02-05T04:54:20.081849+0000","last_up_change":"2024-02-05T04:54:13.063833+0000","last_in_change":"2024-02-05T04:54:14.065808+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-05T04:53:07.549923+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-02-05T04:53:10.126961+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2024-02-05T04:53:22.088665+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":3,"size":7,"min_size":5,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":26,"pg_placement_num":26,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":26,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"62","auid":0,"snap_mode":"selfmanaged","snap_seq":31,"snap_epoch":88,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"shecprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":16384,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"cdb503c5-a32c-4a9a-b1d9-194ab5ee48aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":1189094240},{"type":"v1","addr":"172.21.15.62:6816","nonce":1189094240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":1189094240},{"type":"v1","addr":"172.21.15.62:6819","nonce":1189094240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":1189094240},{"type":"v1","addr":"172.21.15.62:6823","nonce":1189094240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":1189094240},{"type":"v1","addr":"172.21.15.62:6821","nonce":1189094240}]},"public_addr":"172.21.15.62:6816/1189094240","cluster_addr":"172.21.15.62:6819/1189094240","heartbeat_back_addr":"172.21.15.62:6823/1189094240","heartbeat_front_addr":"172.21.15.62:6821/1189094240","state":["exists","up"]},{"osd":1,"uuid":"d7a3d394-74b2-4cb1-a754-4fe77e605ea9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6808","nonce":2878185190},{"type":"v1","addr":"172.21.15.91:6811","nonce":2878185190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6814","nonce":2878185190},{"type":"v1","addr":"172.21.15.91:6818","nonce":2878185190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6828","nonce":2878185190},{"type":"v1","addr":"172.21.15.91:6830","nonce":2878185190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6821","nonce":2878185190},{"type":"v1","addr":"172.21.15.91:6824","nonce":2878185190}]},"public_addr":"172.21.15.91:6811/2878185190","cluster_addr":"172.21.15.91:6818/2878185190","heartbeat_back_addr":"172.21.15.91:6830/2878185190","heartbeat_front_addr":"172.21.15.91:6824/2878185190","state":["exists","up"]},{"osd":2,"uuid":"d6e99efe-475f-41c1-88a3-87ce6b26ce75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":40232889},{"type":"v1","addr":"172.21.15.150:6825","nonce":40232889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":40232889},{"type":"v1","addr":"172.21.15.150:6827","nonce":40232889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":40232889},{"type":"v1","addr":"172.21.15.150:6831","nonce":40232889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":40232889},{"type":"v1","addr":"172.21.15.150:6829","nonce":40232889}]},"public_addr":"172.21.15.150:6825/40232889","cluster_addr":"172.21.15.150:6827/40232889","heartbeat_back_addr":"172.21.15.150:6831/40232889","heartbeat_front_addr":"172.21.15.150:6829/40232889","state":["exists","up"]},{"osd":3,"uuid":"3c964090-5e07-40bd-ac7c-b66e29eb25b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":3834523862},{"type":"v1","addr":"172.21.15.196:6803","nonce":3834523862}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":3834523862},{"type":"v1","addr":"172.21.15.196:6805","nonce":3834523862}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":3834523862},{"type":"v1","addr":"172.21.15.196:6809","nonce":3834523862}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":3834523862},{"type":"v1","addr":"172.21.15.196:6807","nonce":3834523862}]},"public_addr":"172.21.15.196:6803/3834523862","cluster_addr":"172.21.15.196:6805/3834523862","heartbeat_back_addr":"172.21.15.196:6809/3834523862","heartbeat_front_addr":"172.21.15.196:6807/3834523862","state":["exists","up"]},{"osd":4,"uuid":"22bd25f9-3d93-4bcb-bcfc-71feae1097f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":2244900897},{"type":"v1","addr":"172.21.15.62:6801","nonce":2244900897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":2244900897},{"type":"v1","addr":"172.21.15.62:6803","nonce":2244900897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":2244900897},{"type":"v1","addr":"172.21.15.62:6807","nonce":2244900897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":2244900897},{"type":"v1","addr":"172.21.15.62:6805","nonce":2244900897}]},"public_addr":"172.21.15.62:6801/2244900897","cluster_addr":"172.21.15.62:6803/2244900897","heartbeat_back_addr":"172.21.15.62:6807/2244900897","heartbeat_front_addr":"172.21.15.62:6805/2244900897","state":["exists","up"]},{"osd":5,"uuid":"c6ae2bd4-4a4f-4f23-8991-743ffa42ed3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6800","nonce":473596866},{"type":"v1","addr":"172.21.15.91:6801","nonce":473596866}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6802","nonce":473596866},{"type":"v1","addr":"172.21.15.91:6803","nonce":473596866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6806","nonce":473596866},{"type":"v1","addr":"172.21.15.91:6807","nonce":473596866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6804","nonce":473596866},{"type":"v1","addr":"172.21.15.91:6805","nonce":473596866}]},"public_addr":"172.21.15.91:6801/473596866","cluster_addr":"172.21.15.91:6803/473596866","heartbeat_back_addr":"172.21.15.91:6807/473596866","heartbeat_front_addr":"172.21.15.91:6805/473596866","state":["exists","up"]},{"osd":6,"uuid":"49da8b05-ffa3-4172-983e-f081d3eb9ae6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":2199648645},{"type":"v1","addr":"172.21.15.150:6801","nonce":2199648645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":2199648645},{"type":"v1","addr":"172.21.15.150:6803","nonce":2199648645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":2199648645},{"type":"v1","addr":"172.21.15.150:6807","nonce":2199648645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":2199648645},{"type":"v1","addr":"172.21.15.150:6805","nonce":2199648645}]},"public_addr":"172.21.15.150:6801/2199648645","cluster_addr":"172.21.15.150:6803/2199648645","heartbeat_back_addr":"172.21.15.150:6807/2199648645","heartbeat_front_addr":"172.21.15.150:6805/2199648645","state":["exists","up"]},{"osd":7,"uuid":"4ece26e1-2c63-4223-baaf-882cd8261ade","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":254726044},{"type":"v1","addr":"172.21.15.196:6811","nonce":254726044}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":254726044},{"type":"v1","addr":"172.21.15.196:6813","nonce":254726044}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":254726044},{"type":"v1","addr":"172.21.15.196:6820","nonce":254726044}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":254726044},{"type":"v1","addr":"172.21.15.196:6816","nonce":254726044}]},"public_addr":"172.21.15.196:6811/254726044","cluster_addr":"172.21.15.196:6813/254726044","heartbeat_back_addr":"172.21.15.196:6820/254726044","heartbeat_front_addr":"172.21.15.196:6816/254726044","state":["exists","up"]},{"osd":8,"uuid":"9ac666e9-291e-4696-a6bb-e10c9deef3f5","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":3461147157},{"type":"v1","addr":"172.21.15.62:6809","nonce":3461147157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":3461147157},{"type":"v1","addr":"172.21.15.62:6811","nonce":3461147157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":3461147157},{"type":"v1","addr":"172.21.15.62:6817","nonce":3461147157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":3461147157},{"type":"v1","addr":"172.21.15.62:6813","nonce":3461147157}]},"public_addr":"172.21.15.62:6809/3461147157","cluster_addr":"172.21.15.62:6811/3461147157","heartbeat_back_addr":"172.21.15.62:6817/3461147157","heartbeat_front_addr":"172.21.15.62:6813/3461147157","state":["exists","up"]},{"osd":9,"uuid":"7fe04cb8-ac67-46ac-b2de-1f8f60a50373","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6810","nonce":1859221562},{"type":"v1","addr":"172.21.15.91:6813","nonce":1859221562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6816","nonce":1859221562},{"type":"v1","addr":"172.21.15.91:6820","nonce":1859221562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6829","nonce":1859221562},{"type":"v1","addr":"172.21.15.91:6831","nonce":1859221562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6823","nonce":1859221562},{"type":"v1","addr":"172.21.15.91:6826","nonce":1859221562}]},"public_addr":"172.21.15.91:6813/1859221562","cluster_addr":"172.21.15.91:6820/1859221562","heartbeat_back_addr":"172.21.15.91:6831/1859221562","heartbeat_front_addr":"172.21.15.91:6826/1859221562","state":["exists","up"]},{"osd":10,"uuid":"a6b0ac25-da07-4fee-b646-a4530ce5b343","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2448697550},{"type":"v1","addr":"172.21.15.150:6809","nonce":2448697550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2448697550},{"type":"v1","addr":"172.21.15.150:6811","nonce":2448697550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":2448697550},{"type":"v1","addr":"172.21.15.150:6819","nonce":2448697550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6813","nonce":2448697550},{"type":"v1","addr":"172.21.15.150:6815","nonce":2448697550}]},"public_addr":"172.21.15.150:6809/2448697550","cluster_addr":"172.21.15.150:6811/2448697550","heartbeat_back_addr":"172.21.15.150:6819/2448697550","heartbeat_front_addr":"172.21.15.150:6815/2448697550","state":["exists","up"]},{"osd":11,"uuid":"31b22258-77da-495c-b518-9ce0297b9165","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6815","nonce":1170435852},{"type":"v1","addr":"172.21.15.196:6817","nonce":1170435852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6819","nonce":1170435852},{"type":"v1","addr":"172.21.15.196:6821","nonce":1170435852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":1170435852},{"type":"v1","addr":"172.21.15.196:6825","nonce":1170435852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":1170435852},{"type":"v1","addr":"172.21.15.196:6823","nonce":1170435852}]},"public_addr":"172.21.15.196:6817/1170435852","cluster_addr":"172.21.15.196:6821/1170435852","heartbeat_back_addr":"172.21.15.196:6825/1170435852","heartbeat_front_addr":"172.21.15.196:6823/1170435852","state":["exists","up"]},{"osd":12,"uuid":"322917eb-26ba-4439-8300-4d723bada623","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":1173398345},{"type":"v1","addr":"172.21.15.62:6825","nonce":1173398345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":1173398345},{"type":"v1","addr":"172.21.15.62:6827","nonce":1173398345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":1173398345},{"type":"v1","addr":"172.21.15.62:6831","nonce":1173398345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":1173398345},{"type":"v1","addr":"172.21.15.62:6829","nonce":1173398345}]},"public_addr":"172.21.15.62:6825/1173398345","cluster_addr":"172.21.15.62:6827/1173398345","heartbeat_back_addr":"172.21.15.62:6831/1173398345","heartbeat_front_addr":"172.21.15.62:6829/1173398345","state":["exists","up"]},{"osd":13,"uuid":"9c708044-4726-42b6-9c13-03963f956d2d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":62,"down_at":81,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6809","nonce":3656255795},{"type":"v1","addr":"172.21.15.91:6812","nonce":3656255795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6815","nonce":3656255795},{"type":"v1","addr":"172.21.15.91:6817","nonce":3656255795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6825","nonce":3656255795},{"type":"v1","addr":"172.21.15.91:6827","nonce":3656255795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6819","nonce":3656255795},{"type":"v1","addr":"172.21.15.91:6822","nonce":3656255795}]},"public_addr":"172.21.15.91:6812/3656255795","cluster_addr":"172.21.15.91:6817/3656255795","heartbeat_back_addr":"172.21.15.91:6827/3656255795","heartbeat_front_addr":"172.21.15.91:6822/3656255795","state":["exists"]},{"osd":14,"uuid":"0208144a-db9a-404c-b31e-13bc2e276b09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2102158830},{"type":"v1","addr":"172.21.15.150:6814","nonce":2102158830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":2102158830},{"type":"v1","addr":"172.21.15.150:6818","nonce":2102158830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":2102158830},{"type":"v1","addr":"172.21.15.150:6824","nonce":2102158830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":2102158830},{"type":"v1","addr":"172.21.15.150:6821","nonce":2102158830}]},"public_addr":"172.21.15.150:6814/2102158830","cluster_addr":"172.21.15.150:6818/2102158830","heartbeat_back_addr":"172.21.15.150:6824/2102158830","heartbeat_front_addr":"172.21.15.150:6821/2102158830","state":["exists","up"]},{"osd":15,"uuid":"0591e7e1-dc07-4d80-b574-9b8348635764","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":3187639683},{"type":"v1","addr":"172.21.15.196:6827","nonce":3187639683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":3187639683},{"type":"v1","addr":"172.21.15.196:6829","nonce":3187639683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":3187639683},{"type":"v1","addr":"172.21.15.196:6833","nonce":3187639683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":3187639683},{"type":"v1","addr":"172.21.15.196:6831","nonce":3187639683}]},"public_addr":"172.21.15.196:6827/3187639683","cluster_addr":"172.21.15.196:6829/3187639683","heartbeat_back_addr":"172.21.15.196:6833/3187639683","heartbeat_front_addr":"172.21.15.196:6831/3187639683","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T04:53:04.797667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T04:54:16.107345+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T04:53:04.897353+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T04:53:04.736168+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T04:53:04.778014+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T04:53:04.777687+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T04:54:11.123847+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T04:54:16.474065+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":65536,"last_purged_snaps_scrub":"2024-02-05T04:53:04.851604+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T04:53:04.907238+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T04:54:12.049048+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T04:54:13.398408+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T04:54:05.894943+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":65536,"last_purged_snaps_scrub":"2024-02-05T04:53:04.815624+0000","dead_epoch":81},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T04:54:06.295323+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T04:54:14.351177+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":7,"length":1},{"begin":22,"length":1},{"begin":26,"length":2},{"begin":29,"length":1}]}],"new_removed_snaps":[{"pool":3,"snaps":[{"begin":29,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-05T04:54:20.554 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-05T04:54:20.576 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T04:54:20.577 INFO:teuthology.orchestra.run.smithi196.stdout: "size": 20, 2024-02-05T04:54:20.577 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 600, 2024-02-05T04:54:20.577 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-02-05T04:54:20.577 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.577 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.0s2 63/62 ECSubWrite(tid=52, reqid=client.4634.0:576, at_version=63'6, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:20.577 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:55.000860+0000", 2024-02-05T04:54:20.577 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 25.574052893000001, 2024-02-05T04:54:20.577 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.010870894000000001, 2024-02-05T04:54:20.577 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:20.577 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:20.577 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:20.577 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.577 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:20.577 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.000860+0000", 2024-02-05T04:54:20.578 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:20.578 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.578 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.578 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:20.578 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.000860+0000", 2024-02-05T04:54:20.578 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:20.578 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.578 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.578 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:20.578 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.000861+0000", 2024-02-05T04:54:20.578 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.61e-07 2024-02-05T04:54:20.578 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.578 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.578 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:20.578 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.001743+0000", 2024-02-05T04:54:20.579 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00088195999999999999 2024-02-05T04:54:20.579 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.579 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.579 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:20.579 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.001753+0000", 2024-02-05T04:54:20.579 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0370000000000001e-05 2024-02-05T04:54:20.579 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.579 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.579 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:20.579 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.001762+0000", 2024-02-05T04:54:20.579 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.6649999999999992e-06 2024-02-05T04:54:20.579 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.579 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.579 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:20.580 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.002420+0000", 2024-02-05T04:54:20.580 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00065761699999999999 2024-02-05T04:54:20.580 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.580 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.580 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:20.580 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.002430+0000", 2024-02-05T04:54:20.580 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0366e-05 2024-02-05T04:54:20.580 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.580 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.580 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:20.580 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.011697+0000", 2024-02-05T04:54:20.580 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0092668270000000001 2024-02-05T04:54:20.580 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.580 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.580 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:20.581 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.011731+0000", 2024-02-05T04:54:20.581 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4428000000000003e-05 2024-02-05T04:54:20.581 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.581 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:20.581 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.581 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.581 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.581 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.0s2 63/62 ECSubWrite(tid=53, reqid=client.4634.0:577, at_version=63'7, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:20.581 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:55.003409+0000", 2024-02-05T04:54:20.581 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 25.57150481, 2024-02-05T04:54:20.581 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.011422261, 2024-02-05T04:54:20.581 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:20.581 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:20.581 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:20.581 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.581 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:20.582 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.003409+0000", 2024-02-05T04:54:20.582 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:20.582 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.582 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.582 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:20.582 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.003409+0000", 2024-02-05T04:54:20.582 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:20.582 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.582 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.582 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:20.582 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.003409+0000", 2024-02-05T04:54:20.582 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5899999999999995e-07 2024-02-05T04:54:20.582 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.582 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.582 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:20.582 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.003761+0000", 2024-02-05T04:54:20.583 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00035215600000000001 2024-02-05T04:54:20.583 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.583 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.583 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:20.583 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.003774+0000", 2024-02-05T04:54:20.583 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.289e-05 2024-02-05T04:54:20.583 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.583 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.583 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:20.583 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.003785+0000", 2024-02-05T04:54:20.583 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0634e-05 2024-02-05T04:54:20.583 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.583 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.583 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:20.583 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.005279+0000", 2024-02-05T04:54:20.584 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001494287 2024-02-05T04:54:20.584 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.584 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.584 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:20.584 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.005289+0000", 2024-02-05T04:54:20.584 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.7839999999999998e-06 2024-02-05T04:54:20.584 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.584 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.584 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:20.584 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.014797+0000", 2024-02-05T04:54:20.584 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0095077860000000007 2024-02-05T04:54:20.584 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.584 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.584 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:20.584 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.014831+0000", 2024-02-05T04:54:20.585 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3964999999999998e-05 2024-02-05T04:54:20.585 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.585 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:20.585 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.585 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.585 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.585 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.0s2 63/62 ECSubWrite(tid=54, reqid=client.4634.0:578, at_version=63'8, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:20.585 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:55.006124+0000", 2024-02-05T04:54:20.585 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 25.568789250999998, 2024-02-05T04:54:20.585 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0087667140000000001, 2024-02-05T04:54:20.585 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:20.585 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:20.585 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:20.585 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.585 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:20.585 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.006124+0000", 2024-02-05T04:54:20.586 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:20.586 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.586 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.586 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:20.586 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.006124+0000", 2024-02-05T04:54:20.586 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:20.586 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.586 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.586 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:20.586 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.006125+0000", 2024-02-05T04:54:20.586 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8100000000000002e-07 2024-02-05T04:54:20.586 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.586 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.586 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:20.586 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.006211+0000", 2024-02-05T04:54:20.587 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.6119000000000007e-05 2024-02-05T04:54:20.587 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.587 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.587 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:20.587 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.006226+0000", 2024-02-05T04:54:20.587 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4535999999999999e-05 2024-02-05T04:54:20.587 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.587 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.587 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:20.587 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.006235+0000", 2024-02-05T04:54:20.587 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6080000000000002e-06 2024-02-05T04:54:20.587 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.587 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.587 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:20.587 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.006281+0000", 2024-02-05T04:54:20.587 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5664000000000002e-05 2024-02-05T04:54:20.588 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.588 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.588 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:20.588 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.006287+0000", 2024-02-05T04:54:20.588 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9850000000000003e-06 2024-02-05T04:54:20.588 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.588 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.588 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:20.588 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.014862+0000", 2024-02-05T04:54:20.588 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0085751730000000002 2024-02-05T04:54:20.588 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.588 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.588 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:20.588 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.014891+0000", 2024-02-05T04:54:20.588 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8847999999999999e-05 2024-02-05T04:54:20.589 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.589 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:20.589 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.589 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.589 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.589 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.1s4 63/62 ECSubWrite(tid=265, reqid=client.4634.0:594, at_version=63'53, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:20.589 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:53:55.008496+0000", 2024-02-05T04:54:20.589 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 25.566417494, 2024-02-05T04:54:20.589 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0086120399999999996, 2024-02-05T04:54:20.589 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:20.589 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:20.589 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:20.589 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.589 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:20.589 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.008496+0000", 2024-02-05T04:54:20.590 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:20.590 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.590 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.590 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:20.590 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.008496+0000", 2024-02-05T04:54:20.590 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:20.590 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.590 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.590 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:20.590 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.008497+0000", 2024-02-05T04:54:20.590 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.0599999999999999e-07 2024-02-05T04:54:20.590 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.590 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.590 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:20.590 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.008863+0000", 2024-02-05T04:54:20.590 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000366468 2024-02-05T04:54:20.591 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.591 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.591 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:20.591 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.008875+0000", 2024-02-05T04:54:20.591 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1980999999999999e-05 2024-02-05T04:54:20.591 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.591 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.591 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:20.591 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.008884+0000", 2024-02-05T04:54:20.591 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.9299999999999992e-06 2024-02-05T04:54:20.591 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.591 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.591 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:20.591 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.008965+0000", 2024-02-05T04:54:20.591 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.1299999999999997e-05 2024-02-05T04:54:20.592 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.592 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.592 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:20.592 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.008977+0000", 2024-02-05T04:54:20.592 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1874e-05 2024-02-05T04:54:20.592 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.592 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.592 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:20.592 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.017067+0000", 2024-02-05T04:54:20.592 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0080894299999999999 2024-02-05T04:54:20.592 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.592 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.592 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:20.592 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:53:55.017108+0000", 2024-02-05T04:54:20.592 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1151e-05 2024-02-05T04:54:20.592 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.593 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:20.593 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.593 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.593 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.593 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.6s3 74/62 ECSubWrite(tid=337, reqid=osd.13.0:337, at_version=74'38, trim_to=0'0, roll_forward_to=72'36))", 2024-02-05T04:54:20.593 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:54:06.043960+0000", 2024-02-05T04:54:20.593 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 14.530953310999999, 2024-02-05T04:54:20.593 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0097474689999999999, 2024-02-05T04:54:20.593 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:20.593 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:20.593 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:20.593 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.593 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:20.593 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.043960+0000", 2024-02-05T04:54:20.593 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:20.594 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.594 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.594 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:20.594 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.043960+0000", 2024-02-05T04:54:20.594 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:20.594 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.594 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.594 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:20.594 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.043960+0000", 2024-02-05T04:54:20.594 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6300000000000001e-07 2024-02-05T04:54:20.594 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.594 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.594 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:20.594 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.043986+0000", 2024-02-05T04:54:20.594 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.6101999999999998e-05 2024-02-05T04:54:20.594 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.595 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.595 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:20.595 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.044000+0000", 2024-02-05T04:54:20.595 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3277e-05 2024-02-05T04:54:20.595 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.595 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.595 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:20.595 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.044012+0000", 2024-02-05T04:54:20.595 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1878999999999999e-05 2024-02-05T04:54:20.595 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.595 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.595 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:20.595 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.044097+0000", 2024-02-05T04:54:20.595 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.5630999999999994e-05 2024-02-05T04:54:20.595 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.595 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.596 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:20.596 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.044108+0000", 2024-02-05T04:54:20.596 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.117e-05 2024-02-05T04:54:20.596 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.596 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.596 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:20.596 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.053658+0000", 2024-02-05T04:54:20.596 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0095500180000000004 2024-02-05T04:54:20.596 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.596 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.596 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:20.596 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.053708+0000", 2024-02-05T04:54:20.596 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9029000000000001e-05 2024-02-05T04:54:20.596 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.596 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:20.597 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.597 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.597 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.597 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.13s2 74/67 ECSubWrite(tid=153, reqid=osd.11.0:153, at_version=74'48, trim_to=0'0, roll_forward_to=72'46))", 2024-02-05T04:54:20.597 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:54:06.050145+0000", 2024-02-05T04:54:20.597 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 14.524768728, 2024-02-05T04:54:20.597 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0097172219999999993, 2024-02-05T04:54:20.597 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:20.597 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:20.597 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:20.597 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.597 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:20.597 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.050145+0000", 2024-02-05T04:54:20.597 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:20.597 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.597 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.597 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:20.598 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.050145+0000", 2024-02-05T04:54:20.598 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:20.598 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.598 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.598 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:20.598 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.050145+0000", 2024-02-05T04:54:20.598 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8900000000000001e-07 2024-02-05T04:54:20.598 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.598 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.598 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:20.598 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.050169+0000", 2024-02-05T04:54:20.598 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.385e-05 2024-02-05T04:54:20.598 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.598 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.598 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:20.599 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.050178+0000", 2024-02-05T04:54:20.599 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.3079999999999997e-06 2024-02-05T04:54:20.599 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.599 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.599 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:20.599 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.050186+0000", 2024-02-05T04:54:20.599 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.5120000000000008e-06 2024-02-05T04:54:20.599 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.599 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.599 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:20.599 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.050307+0000", 2024-02-05T04:54:20.599 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000120029 2024-02-05T04:54:20.599 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.599 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.599 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:20.599 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.050317+0000", 2024-02-05T04:54:20.599 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0475999999999999e-05 2024-02-05T04:54:20.600 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.600 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.600 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:20.600 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.059813+0000", 2024-02-05T04:54:20.600 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0094956680000000005 2024-02-05T04:54:20.600 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.600 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.600 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:20.600 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.059862+0000", 2024-02-05T04:54:20.600 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9190000000000002e-05 2024-02-05T04:54:20.600 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.600 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:20.600 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.600 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.601 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.601 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.2s4 74/62 ECSubWrite(tid=90, reqid=osd.12.0:90, at_version=74'71, trim_to=0'0, roll_forward_to=72'67))", 2024-02-05T04:54:20.601 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:54:06.051389+0000", 2024-02-05T04:54:20.601 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 14.523524535, 2024-02-05T04:54:20.601 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0085637179999999997, 2024-02-05T04:54:20.601 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:20.601 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:20.601 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:20.601 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.601 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:20.601 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.051389+0000", 2024-02-05T04:54:20.601 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:20.601 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.601 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.601 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:20.601 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.051389+0000", 2024-02-05T04:54:20.602 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:20.602 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.602 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.602 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:20.602 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.051389+0000", 2024-02-05T04:54:20.602 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5700000000000002e-07 2024-02-05T04:54:20.602 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.602 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.602 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:20.602 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.051411+0000", 2024-02-05T04:54:20.602 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1872e-05 2024-02-05T04:54:20.602 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.602 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.602 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:20.602 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.051421+0000", 2024-02-05T04:54:20.603 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8209999999999997e-06 2024-02-05T04:54:20.603 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.603 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.603 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:20.603 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.051429+0000", 2024-02-05T04:54:20.603 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.43e-06 2024-02-05T04:54:20.603 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.603 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.603 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:20.603 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.054003+0000", 2024-02-05T04:54:20.603 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0025740350000000001 2024-02-05T04:54:20.603 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.603 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.603 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:20.603 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.054013+0000", 2024-02-05T04:54:20.604 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0158e-05 2024-02-05T04:54:20.604 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.604 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.604 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:20.604 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.059928+0000", 2024-02-05T04:54:20.604 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0059154639999999996 2024-02-05T04:54:20.604 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.605 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.605 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:20.605 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:06.059953+0000", 2024-02-05T04:54:20.605 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.4281e-05 2024-02-05T04:54:20.605 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.606 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:20.606 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.606 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.606 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.606 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.2s4 75/62 ECSubWrite(tid=96, reqid=osd.12.0:93, at_version=0'0, trim_to=0'0, roll_forward_to=75'75))", 2024-02-05T04:54:20.606 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:54:07.053489+0000", 2024-02-05T04:54:20.606 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 13.521424403999999, 2024-02-05T04:54:20.606 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.011682962, 2024-02-05T04:54:20.606 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:20.606 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:20.606 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:20.606 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.606 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:20.606 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.053489+0000", 2024-02-05T04:54:20.606 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:20.606 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.607 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.607 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:20.607 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.053489+0000", 2024-02-05T04:54:20.607 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:20.607 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.607 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.607 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:20.607 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.053490+0000", 2024-02-05T04:54:20.607 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.1200000000000002e-07 2024-02-05T04:54:20.607 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.607 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.607 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:20.607 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.053499+0000", 2024-02-05T04:54:20.607 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5599999999999999e-06 2024-02-05T04:54:20.607 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.608 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.608 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:20.608 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.053514+0000", 2024-02-05T04:54:20.608 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4630000000000001e-05 2024-02-05T04:54:20.608 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.608 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.608 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:20.608 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.053524+0000", 2024-02-05T04:54:20.608 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.7510000000000003e-06 2024-02-05T04:54:20.608 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.608 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.608 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:20.608 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.053580+0000", 2024-02-05T04:54:20.608 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6665e-05 2024-02-05T04:54:20.608 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.609 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.609 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:20.609 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.053590+0000", 2024-02-05T04:54:20.609 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5300000000000002e-06 2024-02-05T04:54:20.609 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.609 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.609 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:20.609 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.065131+0000", 2024-02-05T04:54:20.609 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.011540912 2024-02-05T04:54:20.609 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.609 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.609 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:20.609 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.065172+0000", 2024-02-05T04:54:20.609 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1102000000000001e-05 2024-02-05T04:54:20.609 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.609 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:20.610 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.610 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.610 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.610 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.3s3 75/62 ECSubWrite(tid=108, reqid=osd.4.0:108, at_version=75'49, trim_to=0'0, roll_forward_to=74'47))", 2024-02-05T04:54:20.610 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:54:07.054226+0000", 2024-02-05T04:54:20.610 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 13.520687307999999, 2024-02-05T04:54:20.610 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.011135473999999999, 2024-02-05T04:54:20.610 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:20.610 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:20.610 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:20.610 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.610 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:20.610 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.054226+0000", 2024-02-05T04:54:20.610 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:20.610 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.610 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.611 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:20.611 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.054226+0000", 2024-02-05T04:54:20.611 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:20.611 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.611 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.611 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:20.611 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.054226+0000", 2024-02-05T04:54:20.611 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.7399999999999999e-07 2024-02-05T04:54:20.611 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.611 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.611 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:20.611 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.054244+0000", 2024-02-05T04:54:20.611 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8150000000000001e-05 2024-02-05T04:54:20.611 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.611 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.612 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:20.612 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.054254+0000", 2024-02-05T04:54:20.612 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8460000000000003e-06 2024-02-05T04:54:20.612 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.612 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.612 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:20.612 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.054281+0000", 2024-02-05T04:54:20.612 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.6942000000000001e-05 2024-02-05T04:54:20.612 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.612 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.612 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:20.612 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.054341+0000", 2024-02-05T04:54:20.612 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9988999999999998e-05 2024-02-05T04:54:20.612 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.612 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.612 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:20.613 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.054351+0000", 2024-02-05T04:54:20.613 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0017000000000001e-05 2024-02-05T04:54:20.613 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.613 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.613 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:20.613 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.065327+0000", 2024-02-05T04:54:20.613 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.010975509 2024-02-05T04:54:20.613 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.613 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.613 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:20.613 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.065362+0000", 2024-02-05T04:54:20.613 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4746999999999998e-05 2024-02-05T04:54:20.613 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.613 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:20.613 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.613 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.614 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.614 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.16s1 75/67 ECSubWrite(tid=165, reqid=osd.11.0:164, at_version=0'0, trim_to=0'0, roll_forward_to=75'36))", 2024-02-05T04:54:20.614 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:54:07.054631+0000", 2024-02-05T04:54:20.614 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 13.520282268000001, 2024-02-05T04:54:20.614 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.011922159999999999, 2024-02-05T04:54:20.614 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:20.614 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:20.614 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:20.614 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.614 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:20.614 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.054631+0000", 2024-02-05T04:54:20.614 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:20.614 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.614 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.614 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:20.614 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.054631+0000", 2024-02-05T04:54:20.615 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:20.615 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.615 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.615 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:20.615 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.054632+0000", 2024-02-05T04:54:20.615 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7199999999999999e-07 2024-02-05T04:54:20.615 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.615 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.615 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:20.615 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.054642+0000", 2024-02-05T04:54:20.615 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0569e-05 2024-02-05T04:54:20.615 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.615 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.615 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:20.615 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.054654+0000", 2024-02-05T04:54:20.615 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2113000000000001e-05 2024-02-05T04:54:20.616 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.616 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.616 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:20.616 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.054663+0000", 2024-02-05T04:54:20.616 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.1850000000000004e-06 2024-02-05T04:54:20.616 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.616 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.616 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:20.616 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.054735+0000", 2024-02-05T04:54:20.616 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2243000000000006e-05 2024-02-05T04:54:20.616 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.616 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.616 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:20.616 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.054744+0000", 2024-02-05T04:54:20.617 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.3859999999999996e-06 2024-02-05T04:54:20.617 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.617 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.617 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:20.617 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.066512+0000", 2024-02-05T04:54:20.617 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.011768120999999999 2024-02-05T04:54:20.617 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.617 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.617 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:20.617 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.066553+0000", 2024-02-05T04:54:20.617 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0970999999999998e-05 2024-02-05T04:54:20.617 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.617 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:20.617 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.617 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.617 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.618 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.6s3 75/62 ECSubWrite(tid=360, reqid=osd.13.0:360, at_version=75'40, trim_to=0'0, roll_forward_to=74'38))", 2024-02-05T04:54:20.618 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:54:07.057866+0000", 2024-02-05T04:54:20.618 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 13.517047484000001, 2024-02-05T04:54:20.618 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.008722268, 2024-02-05T04:54:20.618 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:20.618 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:20.618 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:20.618 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.618 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:20.618 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.057866+0000", 2024-02-05T04:54:20.618 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:20.618 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.618 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.618 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:20.618 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.057866+0000", 2024-02-05T04:54:20.618 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:20.619 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.619 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.619 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:20.619 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.057867+0000", 2024-02-05T04:54:20.619 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2699999999999999e-07 2024-02-05T04:54:20.619 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.619 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.619 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:20.619 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.057894+0000", 2024-02-05T04:54:20.619 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.7670000000000001e-05 2024-02-05T04:54:20.619 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.619 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.619 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:20.619 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.057907+0000", 2024-02-05T04:54:20.619 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2325e-05 2024-02-05T04:54:20.620 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.620 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.620 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:20.620 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.057912+0000", 2024-02-05T04:54:20.620 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.728e-06 2024-02-05T04:54:20.620 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.620 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.620 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:20.620 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.058074+0000", 2024-02-05T04:54:20.620 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000161638 2024-02-05T04:54:20.620 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.620 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.620 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:20.620 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.058087+0000", 2024-02-05T04:54:20.620 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3417e-05 2024-02-05T04:54:20.621 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.621 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.621 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:20.621 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.066560+0000", 2024-02-05T04:54:20.621 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0084721989999999997 2024-02-05T04:54:20.621 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.621 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.621 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:20.621 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.066588+0000", 2024-02-05T04:54:20.621 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8564e-05 2024-02-05T04:54:20.621 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.621 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:20.621 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.621 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.621 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.621 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.1s4 75/62 ECSubWrite(tid=362, reqid=client.4634.0:707, at_version=75'82, trim_to=0'0, roll_forward_to=74'80))", 2024-02-05T04:54:20.622 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:54:07.059868+0000", 2024-02-05T04:54:20.622 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 13.515044957000001, 2024-02-05T04:54:20.622 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.012263005, 2024-02-05T04:54:20.622 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:20.622 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:20.622 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:20.622 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.622 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:20.622 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.059868+0000", 2024-02-05T04:54:20.622 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:20.622 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.622 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.622 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:20.622 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.059868+0000", 2024-02-05T04:54:20.622 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:20.623 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.623 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.623 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:20.623 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.059869+0000", 2024-02-05T04:54:20.623 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7000000000000004e-07 2024-02-05T04:54:20.623 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.623 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.623 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:20.623 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.060075+0000", 2024-02-05T04:54:20.623 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000205649 2024-02-05T04:54:20.623 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.623 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.623 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:20.623 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.060086+0000", 2024-02-05T04:54:20.623 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1691e-05 2024-02-05T04:54:20.623 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.624 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.624 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:20.624 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.060094+0000", 2024-02-05T04:54:20.624 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9529999999999999e-06 2024-02-05T04:54:20.624 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.624 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.624 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:20.624 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.060148+0000", 2024-02-05T04:54:20.624 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3646000000000001e-05 2024-02-05T04:54:20.624 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.624 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.624 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:20.624 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.060157+0000", 2024-02-05T04:54:20.624 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.7369999999999995e-06 2024-02-05T04:54:20.625 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.625 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.625 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:20.625 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.072088+0000", 2024-02-05T04:54:20.625 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.011930926 2024-02-05T04:54:20.625 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.625 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.625 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:20.625 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.072131+0000", 2024-02-05T04:54:20.625 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3633000000000001e-05 2024-02-05T04:54:20.625 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.625 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:20.625 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.625 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.625 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.625 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.9s4 75/62 ECSubWrite(tid=58, reqid=client.4634.0:710, at_version=75'23, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:20.626 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:54:07.059893+0000", 2024-02-05T04:54:20.626 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 13.515020408, 2024-02-05T04:54:20.626 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.012271172, 2024-02-05T04:54:20.626 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:20.626 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:20.626 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:20.626 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.626 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:20.626 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.059893+0000", 2024-02-05T04:54:20.626 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:20.626 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.626 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.626 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:20.626 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.059893+0000", 2024-02-05T04:54:20.626 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:20.627 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.627 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.627 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:20.627 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.059893+0000", 2024-02-05T04:54:20.627 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5499999999999999e-07 2024-02-05T04:54:20.627 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.627 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.627 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:20.627 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.059915+0000", 2024-02-05T04:54:20.627 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2042999999999999e-05 2024-02-05T04:54:20.627 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.627 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.627 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:20.627 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.059925+0000", 2024-02-05T04:54:20.627 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6609999999999995e-06 2024-02-05T04:54:20.627 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.628 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.628 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:20.628 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.059940+0000", 2024-02-05T04:54:20.628 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.518e-05 2024-02-05T04:54:20.628 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.628 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.628 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:20.628 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.060008+0000", 2024-02-05T04:54:20.628 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7354000000000006e-05 2024-02-05T04:54:20.628 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.628 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.629 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:20.629 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.060017+0000", 2024-02-05T04:54:20.629 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2099999999999999e-06 2024-02-05T04:54:20.629 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.629 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.629 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:20.629 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.072137+0000", 2024-02-05T04:54:20.629 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.012120277 2024-02-05T04:54:20.629 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.629 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.629 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:20.629 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.072164+0000", 2024-02-05T04:54:20.629 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.7092000000000001e-05 2024-02-05T04:54:20.629 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.630 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:20.630 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.630 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.630 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.630 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.9s4 75/62 ECSubWrite(tid=59, reqid=client.4634.0:711, at_version=75'24, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:20.630 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:54:07.060128+0000", 2024-02-05T04:54:20.630 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 13.514785374000001, 2024-02-05T04:54:20.630 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.014880060000000001, 2024-02-05T04:54:20.630 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:20.630 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:20.630 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:20.630 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.630 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:20.630 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.060128+0000", 2024-02-05T04:54:20.630 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:20.631 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.631 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.631 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:20.631 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.060128+0000", 2024-02-05T04:54:20.631 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:20.631 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.631 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.631 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:20.631 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.060128+0000", 2024-02-05T04:54:20.631 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1200000000000003e-07 2024-02-05T04:54:20.631 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.631 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.631 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:20.631 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.060144+0000", 2024-02-05T04:54:20.631 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5316999999999999e-05 2024-02-05T04:54:20.631 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.632 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.632 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:20.632 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.060152+0000", 2024-02-05T04:54:20.632 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3820000000000004e-06 2024-02-05T04:54:20.632 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.632 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.632 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:20.632 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.060159+0000", 2024-02-05T04:54:20.632 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4579999999999998e-06 2024-02-05T04:54:20.632 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.632 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.632 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:20.632 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.066361+0000", 2024-02-05T04:54:20.632 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0062027050000000002 2024-02-05T04:54:20.632 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.633 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.633 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:20.633 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.066371+0000", 2024-02-05T04:54:20.633 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.3880000000000006e-06 2024-02-05T04:54:20.633 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.633 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.633 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:20.633 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.074954+0000", 2024-02-05T04:54:20.633 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0085832100000000008 2024-02-05T04:54:20.633 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.633 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.633 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:20.633 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.075008+0000", 2024-02-05T04:54:20.633 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4088000000000001e-05 2024-02-05T04:54:20.633 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.633 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:20.634 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.634 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.634 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.634 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.9s4 75/62 ECSubWrite(tid=60, reqid=client.4634.0:712, at_version=75'25, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:20.634 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:54:07.061297+0000", 2024-02-05T04:54:20.634 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 13.513616043000001, 2024-02-05T04:54:20.634 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.013754094999999999, 2024-02-05T04:54:20.634 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:20.634 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:20.634 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:20.634 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.634 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:20.634 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.061297+0000", 2024-02-05T04:54:20.634 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:20.634 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.635 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.635 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:20.635 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.061297+0000", 2024-02-05T04:54:20.635 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:20.635 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.635 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.635 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:20.635 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.061298+0000", 2024-02-05T04:54:20.635 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3900000000000004e-07 2024-02-05T04:54:20.635 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.635 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.635 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:20.635 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.061324+0000", 2024-02-05T04:54:20.635 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.6313999999999999e-05 2024-02-05T04:54:20.635 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.635 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.636 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:20.636 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.061340+0000", 2024-02-05T04:54:20.636 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5554999999999999e-05 2024-02-05T04:54:20.636 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.636 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.636 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:20.636 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.061348+0000", 2024-02-05T04:54:20.636 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9500000000000001e-06 2024-02-05T04:54:20.636 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.636 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.636 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:20.636 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.066920+0000", 2024-02-05T04:54:20.636 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0055718920000000002 2024-02-05T04:54:20.636 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.636 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.637 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:20.637 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.066928+0000", 2024-02-05T04:54:20.637 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.1969999999999996e-06 2024-02-05T04:54:20.637 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.637 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.637 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:20.637 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.075014+0000", 2024-02-05T04:54:20.637 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0080865669999999994 2024-02-05T04:54:20.637 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.637 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.637 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:20.637 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.075051+0000", 2024-02-05T04:54:20.637 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6980999999999997e-05 2024-02-05T04:54:20.637 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.637 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:20.638 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.638 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.638 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.638 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.3s3 75/62 ECSubWrite(tid=109, reqid=osd.4.0:108, at_version=0'0, trim_to=0'0, roll_forward_to=75'49))", 2024-02-05T04:54:20.638 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:54:07.066584+0000", 2024-02-05T04:54:20.638 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 13.508329767999999, 2024-02-05T04:54:20.638 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0085627979999999999, 2024-02-05T04:54:20.638 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:20.638 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:20.638 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:20.638 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.638 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:20.638 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.066584+0000", 2024-02-05T04:54:20.638 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:20.638 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.638 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.639 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:20.639 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.066584+0000", 2024-02-05T04:54:20.639 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:20.639 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.639 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.639 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:20.639 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.066584+0000", 2024-02-05T04:54:20.639 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.6600000000000005e-07 2024-02-05T04:54:20.639 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.639 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.639 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:20.639 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.066594+0000", 2024-02-05T04:54:20.639 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1840000000000005e-06 2024-02-05T04:54:20.639 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.639 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.639 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:20.640 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.066605+0000", 2024-02-05T04:54:20.640 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1442e-05 2024-02-05T04:54:20.640 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.640 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.640 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:20.640 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.066613+0000", 2024-02-05T04:54:20.640 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0760000000000003e-06 2024-02-05T04:54:20.640 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.640 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.640 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:20.640 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.066682+0000", 2024-02-05T04:54:20.640 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8948999999999993e-05 2024-02-05T04:54:20.640 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.640 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.640 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:20.640 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.066692+0000", 2024-02-05T04:54:20.641 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.4399999999999994e-06 2024-02-05T04:54:20.641 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.641 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.641 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:20.641 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.075108+0000", 2024-02-05T04:54:20.641 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0084167519999999996 2024-02-05T04:54:20.641 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.641 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.641 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:20.641 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:07.075146+0000", 2024-02-05T04:54:20.641 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8089000000000001e-05 2024-02-05T04:54:20.641 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.641 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:20.641 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.641 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.642 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.642 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDECSubOpWrite(3.3s3 84/62 ECSubWrite(tid=123, reqid=client.4634.0:847, at_version=84'61, trim_to=0'0, roll_forward_to=83'59))", 2024-02-05T04:54:20.642 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:54:16.116906+0000", 2024-02-05T04:54:20.642 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 4.4580070899999997, 2024-02-05T04:54:20.642 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0097546690000000005, 2024-02-05T04:54:20.642 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:20.642 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T04:54:20.642 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:20.642 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.642 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:20.642 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.116906+0000", 2024-02-05T04:54:20.642 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:20.642 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.642 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.642 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:20.643 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.116906+0000", 2024-02-05T04:54:20.643 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:20.643 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.643 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.643 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:20.643 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.116907+0000", 2024-02-05T04:54:20.643 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0900000000000002e-07 2024-02-05T04:54:20.643 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.643 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.643 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:20.643 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.121227+0000", 2024-02-05T04:54:20.643 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0043199379999999997 2024-02-05T04:54:20.643 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.643 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.643 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:20.643 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.121245+0000", 2024-02-05T04:54:20.644 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8357999999999999e-05 2024-02-05T04:54:20.644 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.644 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.644 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:20.644 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.121255+0000", 2024-02-05T04:54:20.644 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0018e-05 2024-02-05T04:54:20.644 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.644 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.644 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:20.644 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.121326+0000", 2024-02-05T04:54:20.644 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0791999999999996e-05 2024-02-05T04:54:20.644 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.644 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.644 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:20.644 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.121340+0000", 2024-02-05T04:54:20.645 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.438e-05 2024-02-05T04:54:20.645 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.645 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.645 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:20.645 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.126615+0000", 2024-02-05T04:54:20.645 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0052748480000000004 2024-02-05T04:54:20.645 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.645 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.645 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:20.645 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.126661+0000", 2024-02-05T04:54:20.645 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5926e-05 2024-02-05T04:54:20.645 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.645 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:20.645 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.645 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.645 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.646 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:875 3.5s0 3:a78fa099:::smithi19639362-42:head [stat,delete] snapc 1d=[1d,1c,19,18,17,16,12,8] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-02-05T04:54:20.646 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:54:16.144888+0000", 2024-02-05T04:54:20.646 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 4.4300253380000001, 2024-02-05T04:54:20.646 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0099152530000000006, 2024-02-05T04:54:20.646 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:20.646 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:20.646 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:54:20.646 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:54:20.646 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:20.646 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 875 2024-02-05T04:54:20.646 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.646 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:20.646 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.646 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:20.646 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.144888+0000", 2024-02-05T04:54:20.647 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:20.647 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.647 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.647 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:20.647 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.144888+0000", 2024-02-05T04:54:20.647 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:20.647 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.647 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.647 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:20.647 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.144890+0000", 2024-02-05T04:54:20.647 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8759999999999999e-06 2024-02-05T04:54:20.647 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.647 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.647 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:20.647 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.144898+0000", 2024-02-05T04:54:20.647 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.5799999999999992e-06 2024-02-05T04:54:20.648 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.648 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.648 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:20.648 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.144920+0000", 2024-02-05T04:54:20.648 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1702e-05 2024-02-05T04:54:20.648 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.648 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.648 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:20.648 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.144930+0000", 2024-02-05T04:54:20.648 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6900000000000004e-06 2024-02-05T04:54:20.648 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.648 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.648 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:20.648 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.145012+0000", 2024-02-05T04:54:20.648 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.1730000000000005e-05 2024-02-05T04:54:20.649 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.649 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.649 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:54:20.649 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.145303+0000", 2024-02-05T04:54:20.649 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00029113000000000001 2024-02-05T04:54:20.649 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.649 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.649 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_started", 2024-02-05T04:54:20.649 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.146365+0000", 2024-02-05T04:54:20.649 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0010627270000000001 2024-02-05T04:54:20.649 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.649 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.649 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_committed", 2024-02-05T04:54:20.649 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.149000+0000", 2024-02-05T04:54:20.649 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0026341699999999999 2024-02-05T04:54:20.649 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.650 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.650 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T04:54:20.650 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.154571+0000", 2024-02-05T04:54:20.650 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0055711320000000003 2024-02-05T04:54:20.650 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.650 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.650 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:20.650 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.154803+0000", 2024-02-05T04:54:20.650 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00023251599999999999 2024-02-05T04:54:20.650 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.650 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:20.650 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.650 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.650 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.650 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:900 3.5s0 3:a5929521:::smithi19639362-37:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e84)", 2024-02-05T04:54:20.650 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:54:16.191800+0000", 2024-02-05T04:54:20.651 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 4.3831137130000002, 2024-02-05T04:54:20.651 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.011587867, 2024-02-05T04:54:20.651 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:20.651 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-02-05T04:54:20.651 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:54:20.651 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:54:20.651 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:20.651 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 900 2024-02-05T04:54:20.651 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.651 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:20.651 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.651 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:20.651 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.191800+0000", 2024-02-05T04:54:20.651 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:20.651 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.651 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.652 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:20.652 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.191800+0000", 2024-02-05T04:54:20.652 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:20.652 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.652 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.652 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:20.652 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.191800+0000", 2024-02-05T04:54:20.652 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9299999999999998e-07 2024-02-05T04:54:20.652 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.652 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.652 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:20.652 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.191803+0000", 2024-02-05T04:54:20.652 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3870000000000002e-06 2024-02-05T04:54:20.652 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.652 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.652 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:20.653 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.191819+0000", 2024-02-05T04:54:20.653 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6385999999999999e-05 2024-02-05T04:54:20.653 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.653 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.653 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:20.653 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.191827+0000", 2024-02-05T04:54:20.653 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8509999999999999e-06 2024-02-05T04:54:20.653 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.653 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.653 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:20.653 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.194663+0000", 2024-02-05T04:54:20.653 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002836734 2024-02-05T04:54:20.653 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.653 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.653 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for rw locks", 2024-02-05T04:54:20.654 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.194862+0000", 2024-02-05T04:54:20.654 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000198693 2024-02-05T04:54:20.654 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.654 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.654 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:20.654 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.199697+0000", 2024-02-05T04:54:20.654 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0048349500000000002 2024-02-05T04:54:20.654 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.654 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.654 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:54:20.654 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.199862+0000", 2024-02-05T04:54:20.654 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00016524899999999999 2024-02-05T04:54:20.654 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.654 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.654 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:20.654 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.203387+0000", 2024-02-05T04:54:20.655 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0035251240000000001 2024-02-05T04:54:20.655 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.655 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:20.655 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.655 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.655 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.655 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.4634.0:904 3.5s0 3:a6614921:::smithi19639362-38:head [rollback 1c] snapc 1d=[1d,1c,19,18,17,16,12,8] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-02-05T04:54:20.655 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T04:54:16.191846+0000", 2024-02-05T04:54:20.655 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 4.383067456, 2024-02-05T04:54:20.655 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.010190513, 2024-02-05T04:54:20.655 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T04:54:20.655 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-02-05T04:54:20.655 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T04:54:20.655 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.4634", 2024-02-05T04:54:20.655 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:20.656 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 904 2024-02-05T04:54:20.656 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.656 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T04:54:20.656 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.656 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T04:54:20.656 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.191846+0000", 2024-02-05T04:54:20.656 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:20.656 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.656 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.656 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T04:54:20.656 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.191846+0000", 2024-02-05T04:54:20.656 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T04:54:20.656 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.656 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.656 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T04:54:20.656 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.191846+0000", 2024-02-05T04:54:20.657 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6199999999999998e-07 2024-02-05T04:54:20.657 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.657 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.657 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T04:54:20.657 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.191850+0000", 2024-02-05T04:54:20.657 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9129999999999996e-06 2024-02-05T04:54:20.657 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.657 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.657 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T04:54:20.657 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.191858+0000", 2024-02-05T04:54:20.657 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9449999999999993e-06 2024-02-05T04:54:20.657 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.657 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.657 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T04:54:20.657 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.191864+0000", 2024-02-05T04:54:20.657 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5840000000000002e-06 2024-02-05T04:54:20.658 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.658 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.658 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T04:54:20.658 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.195033+0000", 2024-02-05T04:54:20.658 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0031686119999999999 2024-02-05T04:54:20.658 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.658 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.658 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T04:54:20.658 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.195210+0000", 2024-02-05T04:54:20.658 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000177925 2024-02-05T04:54:20.658 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T04:54:20.658 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T04:54:20.658 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T04:54:20.658 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T04:54:16.202036+0000", 2024-02-05T04:54:20.658 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0068259719999999996 2024-02-05T04:54:20.658 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.659 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:20.659 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.659 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T04:54:20.659 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T04:54:20.659 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T04:54:20.661 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-02-05T04:54:20.661 INFO:tasks.ceph.osd.1.smithi091.stderr:2024-02-05T04:54:20.659+0000 7ff9dc283640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34275) UID: 0 2024-02-05T04:54:20.761 INFO:tasks.ceph.osd.14.smithi150.stderr:2024-02-05T04:54:20.761+0000 7f5f81083640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 34020) UID: 0 2024-02-05T04:54:20.769 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-02-05T04:54:20.769 INFO:teuthology.orchestra.run.smithi062.stdout: "ops": [], 2024-02-05T04:54:20.770 INFO:teuthology.orchestra.run.smithi062.stdout: "num_ops": 0 2024-02-05T04:54:20.770 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-02-05T04:54:20.779 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-02-05T04:54:20.862 INFO:tasks.ceph.osd.4.smithi062.stderr:2024-02-05T04:54:20.862+0000 7f708c308640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34689) UID: 0 2024-02-05T04:54:20.886 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-02-05T04:54:20.886 INFO:teuthology.orchestra.run.smithi062.stdout: "ops": [], 2024-02-05T04:54:20.886 INFO:teuthology.orchestra.run.smithi062.stdout: "complaint_time": 30, 2024-02-05T04:54:20.886 INFO:teuthology.orchestra.run.smithi062.stdout: "num_blocked_ops": 0 2024-02-05T04:54:20.886 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-02-05T04:54:20.895 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-02-05T04:54:20.958 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:54:20.959 INFO:teuthology.orchestra.run.smithi062.stderr:dumped all 2024-02-05T04:54:20.963 INFO:tasks.ceph.osd.4.smithi062.stderr:2024-02-05T04:54:20.963+0000 7f708c308640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34689) UID: 0 2024-02-05T04:54:20.970 INFO:teuthology.orchestra.run.smithi062.stdout:{"pg_ready":true,"pg_map":{"version":112,"stamp":"2024-02-05T04:54:20.087238+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":135692832,"num_objects":131,"num_object_clones":82,"num_object_copies":907,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":131,"num_whiteouts":11,"num_read":369,"num_read_kb":105886,"num_write":543,"num_write_kb":89252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":75,"num_bytes_recovered":95506976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1602,"ondisk_log_size":1602,"up":200,"acting":200,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":196,"num_osds":14,"num_per_pool_osds":14,"num_per_pool_omap_osds":14,"kb":1321205760,"kb_used":696812,"kb_used_data":276856,"kb_used_omap":21,"kb_used_meta":419690,"kb_avail":1320508948,"statfs":{"total":1352914698240,"available":1352201162752,"internally_reserved":0,"allocated":283500544,"data_stored":394403641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22330,"internal_metadata":429762758},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":8093696,"num_objects":13,"num_object_clones":14,"num_object_copies":91,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":3,"num_read":20,"num_read_kb":-13482,"num_write":39,"num_write_kb":10832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":54,"num_bytes_recovered":80084992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.011837"},"pg_stats":[{"pgid":"3.19","version":"84'48","reported_seq":160,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-05T04:54:17.084041+0000","last_change":"2024-02-05T04:54:15.077024+0000","last_active":"2024-02-05T04:54:17.084041+0000","last_peered":"2024-02-05T04:54:17.084041+0000","last_clean":"2024-02-05T04:54:17.084041+0000","last_became_active":"2024-02-05T04:53:59.958048+0000","last_became_peered":"2024-02-05T04:53:59.958048+0000","last_unstale":"2024-02-05T04:54:17.084041+0000","last_undegraded":"2024-02-05T04:54:17.084041+0000","last_fullsized":"2024-02-05T04:54:17.084041+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":5,"last_scrub":"57'21","last_scrub_stamp":"2024-02-05T04:54:03.585473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:54:03.585473+0000","objects_scrubbed":3,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:55:15.304273+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0013419339999999999,"stat_sum":{"num_bytes":3637248,"num_objects":7,"num_object_clones":5,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":8,"num_write_kb":1696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,11,12,14,7,6],"acting":[9,1,11,12,14,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"7","length":"1"},{"start":"b","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"15","length":"1"},{"start":"1a","length":"2"}]},{"pgid":"3.18","version":"45'23","reported_seq":129,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-05T04:54:18.097431+0000","last_change":"2024-02-05T04:54:18.097431+0000","last_active":"2024-02-05T04:54:18.097431+0000","last_peered":"2024-02-05T04:54:18.097431+0000","last_clean":"2024-02-05T04:54:18.097431+0000","last_became_active":"2024-02-05T04:54:18.094178+0000","last_became_peered":"2024-02-05T04:54:18.094178+0000","last_unstale":"2024-02-05T04:54:18.097431+0000","last_undegraded":"2024-02-05T04:54:18.097431+0000","last_fullsized":"2024-02-05T04:54:18.097431+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":5,"last_scrub":"45'23","last_scrub_stamp":"2024-02-05T04:54:03.694275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:54:03.694275+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:55:28.519797+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.002250532,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,6,1,9,15,7,5],"acting":[12,6,1,9,15,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"7","length":"1"},{"start":"b","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"15","length":"1"},{"start":"1a","length":"2"}]},{"pgid":"3.17","version":"83'51","reported_seq":160,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-05T04:54:18.087699+0000","last_change":"2024-02-05T04:54:15.092952+0000","last_active":"2024-02-05T04:54:18.087699+0000","last_peered":"2024-02-05T04:54:18.087699+0000","last_clean":"2024-02-05T04:54:18.087699+0000","last_became_active":"2024-02-05T04:53:59.955376+0000","last_became_peered":"2024-02-05T04:53:59.955376+0000","last_unstale":"2024-02-05T04:54:18.087699+0000","last_undegraded":"2024-02-05T04:54:18.087699+0000","last_fullsized":"2024-02-05T04:54:18.087699+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":5,"last_scrub":"58'30","last_scrub_stamp":"2024-02-05T04:54:03.827890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:54:03.827890+0000","objects_scrubbed":2,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:55:30.326917+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.014965716,"stat_sum":{"num_bytes":2195456,"num_objects":4,"num_object_clones":3,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":3746,"num_write":9,"num_write_kb":2048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,10,4,7,14,11],"acting":[0,3,10,4,7,14,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"7","length":"1"},{"start":"b","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"15","length":"1"},{"start":"1a","length":"2"}]},{"pgid":"3.16","version":"81'38","reported_seq":150,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-05T04:54:18.086225+0000","last_change":"2024-02-05T04:54:15.084287+0000","last_active":"2024-02-05T04:54:18.086225+0000","last_peered":"2024-02-05T04:54:18.086225+0000","last_clean":"2024-02-05T04:54:18.086225+0000","last_became_active":"2024-02-05T04:53:59.956531+0000","last_became_peered":"2024-02-05T04:53:59.956531+0000","last_unstale":"2024-02-05T04:54:18.086225+0000","last_undegraded":"2024-02-05T04:54:18.086225+0000","last_fullsized":"2024-02-05T04:54:18.086225+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":5,"last_scrub":"71'30","last_scrub_stamp":"2024-02-05T04:54:03.634933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:54:03.634933+0000","objects_scrubbed":2,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:55:04.814842+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0064460059999999998,"stat_sum":{"num_bytes":622592,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,15,9,5,10,12,1],"acting":[11,15,9,5,10,12,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"7","length":"1"},{"start":"b","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"15","length":"1"},{"start":"1a","length":"2"}]},{"pgid":"3.15","version":"58'40","reported_seq":112,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-05T04:54:18.085776+0000","last_change":"2024-02-05T04:54:15.077348+0000","last_active":"2024-02-05T04:54:18.085776+0000","last_peered":"2024-02-05T04:54:18.085776+0000","last_clean":"2024-02-05T04:54:18.085776+0000","last_became_active":"2024-02-05T04:53:59.955382+0000","last_became_peered":"2024-02-05T04:53:59.955382+0000","last_unstale":"2024-02-05T04:54:18.085776+0000","last_undegraded":"2024-02-05T04:54:18.085776+0000","last_fullsized":"2024-02-05T04:54:18.085776+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":5,"last_scrub":"58'40","last_scrub_stamp":"2024-02-05T04:54:04.666054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:54:04.666054+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:55:19.618239+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00073255200000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,6,3,4,14,2,15],"acting":[12,6,3,4,14,2,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"7","length":"1"},{"start":"b","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"15","length":"1"},{"start":"1a","length":"2"}]},{"pgid":"3.14","version":"84'89","reported_seq":176,"reported_epoch":84,"state":"active+undersized+remapped","last_fresh":"2024-02-05T04:54:16.270890+0000","last_change":"2024-02-05T04:54:16.271393+0000","last_active":"2024-02-05T04:54:16.270890+0000","last_peered":"2024-02-05T04:54:16.270890+0000","last_clean":"2024-02-05T04:54:12.871747+0000","last_became_active":"2024-02-05T04:54:16.086692+0000","last_became_peered":"2024-02-05T04:54:16.086692+0000","last_unstale":"2024-02-05T04:54:16.270890+0000","last_undegraded":"2024-02-05T04:54:16.077361+0000","last_fullsized":"2024-02-05T04:54:16.076604+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":5,"last_scrub":"71'79","last_scrub_stamp":"2024-02-05T04:54:03.639971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:54:03.639971+0000","objects_scrubbed":5,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00061923700000000002,"stat_sum":{"num_bytes":7454720,"num_objects":7,"num_object_clones":5,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":2,"num_read":2,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":12533760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,14,15,0,12,9,7],"acting":[3,14,15,0,12,9,2147483647],"avail_no_missing":["3(0)","0(3)","7(6)","9(5)","12(4)","14(1)","15(2)"],"object_location_counts":[{"shards":"0(3),3(0),9(5),12(4),14(1),15(2)","objects":7}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"b","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"15","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":65,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-05T04:54:16.079945+0000","last_change":"2024-02-05T04:53:12.781269+0000","last_active":"2024-02-05T04:54:16.079945+0000","last_peered":"2024-02-05T04:54:16.079945+0000","last_clean":"2024-02-05T04:54:16.079945+0000","last_became_active":"2024-02-05T04:53:12.780970+0000","last_became_peered":"2024-02-05T04:53:12.780970+0000","last_unstale":"2024-02-05T04:54:16.079945+0000","last_undegraded":"2024-02-05T04:54:16.079945+0000","last_fullsized":"2024-02-05T04:54:16.079945+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_clean_scrub_stamp":"2024-02-05T04:53:11.017155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:36.119048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,0],"acting":[15,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.6","version":"75'44","reported_seq":272,"reported_epoch":81,"state":"peering","last_fresh":"2024-02-05T04:54:13.071116+0000","last_change":"2024-02-05T04:54:13.071116+0000","last_active":"2024-02-05T04:54:13.071116+0000","last_peered":"2024-02-05T04:54:07.089035+0000","last_clean":"2024-02-05T04:54:07.089035+0000","last_became_active":"2024-02-05T04:53:54.992358+0000","last_became_peered":"2024-02-05T04:53:54.992358+0000","last_unstale":"2024-02-05T04:54:13.071116+0000","last_undegraded":"2024-02-05T04:54:13.071116+0000","last_fullsized":"2024-02-05T04:54:13.071116+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'26","last_scrub_stamp":"2024-02-05T04:53:55.597534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:55.597534+0000","objects_scrubbed":1,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.017878090999999999,"stat_sum":{"num_bytes":4571136,"num_objects":3,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,12,4,15,1,7,10],"acting":[2147483647,12,4,15,1,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"b","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"15","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":67,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-05T04:54:16.078330+0000","last_change":"2024-02-05T04:53:12.037950+0000","last_active":"2024-02-05T04:54:16.078330+0000","last_peered":"2024-02-05T04:54:16.078330+0000","last_clean":"2024-02-05T04:54:16.078330+0000","last_became_active":"2024-02-05T04:53:12.037750+0000","last_became_peered":"2024-02-05T04:53:12.037750+0000","last_unstale":"2024-02-05T04:54:16.078330+0000","last_undegraded":"2024-02-05T04:54:16.078330+0000","last_fullsized":"2024-02-05T04:54:16.078330+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_clean_scrub_stamp":"2024-02-05T04:53:11.017155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:26.129079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"79'38","reported_seq":244,"reported_epoch":85,"state":"peering","last_fresh":"2024-02-05T04:54:11.148154+0000","last_change":"2024-02-05T04:54:17.837401+0000","last_active":"2024-02-05T04:54:17.837401+0000","last_peered":"2024-02-05T04:54:11.148154+0000","last_clean":"2024-02-05T04:54:11.148154+0000","last_became_active":"2024-02-05T04:53:54.997596+0000","last_became_peered":"2024-02-05T04:53:54.997596+0000","last_unstale":"2024-02-05T04:54:11.148154+0000","last_undegraded":"2024-02-05T04:54:11.148154+0000","last_fullsized":"2024-02-05T04:54:11.148154+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'30","last_scrub_stamp":"2024-02-05T04:53:56.628250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:56.628250+0000","objects_scrubbed":2,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001778817,"stat_sum":{"num_bytes":5996544,"num_objects":3,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,6,12,1,4,5,11],"acting":[14,6,12,1,4,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"b","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"15","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":66,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-05T04:54:15.077855+0000","last_change":"2024-02-05T04:53:12.753233+0000","last_active":"2024-02-05T04:54:15.077855+0000","last_peered":"2024-02-05T04:54:15.077855+0000","last_clean":"2024-02-05T04:54:15.077855+0000","last_became_active":"2024-02-05T04:53:12.752915+0000","last_became_peered":"2024-02-05T04:53:12.752915+0000","last_unstale":"2024-02-05T04:54:15.077855+0000","last_undegraded":"2024-02-05T04:54:15.077855+0000","last_fullsized":"2024-02-05T04:54:15.077855+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_clean_scrub_stamp":"2024-02-05T04:53:11.017155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:18.391150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"84'86","reported_seq":383,"reported_epoch":84,"state":"peering","last_fresh":"2024-02-05T04:54:16.284837+0000","last_change":"2024-02-05T04:54:17.080114+0000","last_active":"2024-02-05T04:54:17.080114+0000","last_peered":"2024-02-05T04:54:16.284837+0000","last_clean":"2024-02-05T04:54:12.097439+0000","last_became_active":"2024-02-05T04:54:16.084127+0000","last_became_peered":"2024-02-05T04:54:16.084127+0000","last_unstale":"2024-02-05T04:54:16.284837+0000","last_undegraded":"2024-02-05T04:54:16.284837+0000","last_fullsized":"2024-02-05T04:54:16.075904+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'73","last_scrub_stamp":"2024-02-05T04:53:55.059635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:55.059635+0000","objects_scrubbed":6,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.0021948129999999999,"stat_sum":{"num_bytes":9502720,"num_objects":7,"num_object_clones":4,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":5,"num_read_kb":6625,"num_write":6,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":16777216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5,12,9,4,0],"acting":[1,2,5,12,9,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,2,4,5,9,12],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"b","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"15","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":75,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-05T04:54:16.076137+0000","last_change":"2024-02-05T04:54:15.974830+0000","last_active":"2024-02-05T04:54:16.076137+0000","last_peered":"2024-02-05T04:54:16.076137+0000","last_clean":"2024-02-05T04:54:16.076137+0000","last_became_active":"2024-02-05T04:53:12.032664+0000","last_became_peered":"2024-02-05T04:53:12.032664+0000","last_unstale":"2024-02-05T04:54:16.076137+0000","last_undegraded":"2024-02-05T04:54:16.076137+0000","last_fullsized":"2024-02-05T04:54:16.076137+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:54:15.974746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_clean_scrub_stamp":"2024-02-05T04:54:15.974746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:55:21.829978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"84'71","reported_seq":346,"reported_epoch":84,"state":"peering","last_fresh":"2024-02-05T04:54:16.321687+0000","last_change":"2024-02-05T04:54:17.083437+0000","last_active":"2024-02-05T04:54:17.083437+0000","last_peered":"2024-02-05T04:54:16.321687+0000","last_clean":"2024-02-05T04:54:12.079328+0000","last_became_active":"2024-02-05T04:54:16.088858+0000","last_became_peered":"2024-02-05T04:54:16.088858+0000","last_unstale":"2024-02-05T04:54:16.321687+0000","last_undegraded":"2024-02-05T04:54:16.321687+0000","last_fullsized":"2024-02-05T04:54:16.076454+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'42","last_scrub_stamp":"2024-02-05T04:53:55.638186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:55.638186+0000","objects_scrubbed":6,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00035909700000000001,"stat_sum":{"num_bytes":7831552,"num_objects":8,"num_object_clones":5,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":2,"num_read":7,"num_read_kb":15363,"num_write":8,"num_write_kb":2016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":12812288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,10,5,3,7,1,6],"acting":[15,10,5,3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,3,5,6,7,10],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"b","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"15","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"3.3","version":"84'61","reported_seq":280,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-05T04:54:16.119643+0000","last_change":"2024-02-05T04:54:15.088141+0000","last_active":"2024-02-05T04:54:16.119643+0000","last_peered":"2024-02-05T04:54:16.119643+0000","last_clean":"2024-02-05T04:54:16.119643+0000","last_became_active":"2024-02-05T04:53:54.990470+0000","last_became_peered":"2024-02-05T04:53:54.990470+0000","last_unstale":"2024-02-05T04:54:16.119643+0000","last_undegraded":"2024-02-05T04:54:16.119643+0000","last_fullsized":"2024-02-05T04:54:16.119643+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'37","last_scrub_stamp":"2024-02-05T04:53:56.868534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:56.868534+0000","objects_scrubbed":1,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:55:19.789513+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0065110979999999999,"stat_sum":{"num_bytes":8945664,"num_objects":5,"num_object_clones":3,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":12,"num_write_kb":6800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1,15,2,5,3],"acting":[4,7,1,15,2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"7","length":"1"},{"start":"b","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"15","length":"1"},{"start":"1a","length":"2"}]},{"pgid":"2.2","version":"0'0","reported_seq":65,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-05T04:54:15.078609+0000","last_change":"2024-02-05T04:53:12.035901+0000","last_active":"2024-02-05T04:54:15.078609+0000","last_peered":"2024-02-05T04:54:15.078609+0000","last_clean":"2024-02-05T04:54:15.078609+0000","last_became_active":"2024-02-05T04:53:12.035739+0000","last_became_peered":"2024-02-05T04:53:12.035739+0000","last_unstale":"2024-02-05T04:54:15.078609+0000","last_undegraded":"2024-02-05T04:54:15.078609+0000","last_fullsized":"2024-02-05T04:54:15.078609+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_clean_scrub_stamp":"2024-02-05T04:53:11.017155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9],"acting":[3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"84'13","reported_seq":257,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-05T04:54:17.080984+0000","last_change":"2024-02-05T04:54:15.082333+0000","last_active":"2024-02-05T04:54:17.080984+0000","last_peered":"2024-02-05T04:54:17.080984+0000","last_clean":"2024-02-05T04:54:17.080984+0000","last_became_active":"2024-02-05T04:53:54.996542+0000","last_became_peered":"2024-02-05T04:53:54.996542+0000","last_unstale":"2024-02-05T04:54:17.080984+0000","last_undegraded":"2024-02-05T04:54:17.080984+0000","last_fullsized":"2024-02-05T04:54:17.080984+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-05T04:53:57.002316+0000","last_deep_scrub":"63'9","last_deep_scrub_stamp":"2024-02-05T04:53:57.002316+0000","last_clean_scrub_stamp":"2024-02-05T04:53:57.002316+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:59.825993+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0.001929863,"stat_sum":{"num_bytes":1884160,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,2,15,0,12,6,9],"acting":[14,2,15,0,12,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"7","length":"1"},{"start":"b","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"15","length":"1"},{"start":"1a","length":"2"}]},{"pgid":"2.1","version":"0'0","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-05T04:54:15.081462+0000","last_change":"2024-02-05T04:53:46.969248+0000","last_active":"2024-02-05T04:54:15.081462+0000","last_peered":"2024-02-05T04:54:15.081462+0000","last_clean":"2024-02-05T04:54:15.081462+0000","last_became_active":"2024-02-05T04:53:46.969047+0000","last_became_peered":"2024-02-05T04:53:46.969047+0000","last_unstale":"2024-02-05T04:54:15.081462+0000","last_undegraded":"2024-02-05T04:54:15.081462+0000","last_fullsized":"2024-02-05T04:54:15.081462+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_clean_scrub_stamp":"2024-02-05T04:53:11.017155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1","version":"84'89","reported_seq":358,"reported_epoch":85,"state":"creating+peering","last_fresh":"2024-02-05T04:54:16.170957+0000","last_change":"2024-02-05T04:54:17.843335+0000","last_active":"2024-02-05T04:54:17.843335+0000","last_peered":"2024-02-05T04:54:16.170957+0000","last_clean":"2024-02-05T04:54:12.083488+0000","last_became_active":"2024-02-05T04:54:16.089615+0000","last_became_peered":"2024-02-05T04:54:16.089615+0000","last_unstale":"2024-02-05T04:54:16.170957+0000","last_undegraded":"2024-02-05T04:54:16.078256+0000","last_fullsized":"2024-02-05T04:54:16.077132+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'72","last_scrub_stamp":"2024-02-05T04:54:03.657329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:54:03.657329+0000","objects_scrubbed":9,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00094656099999999995,"stat_sum":{"num_bytes":6668288,"num_objects":11,"num_object_clones":8,"num_object_copies":77,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19,"num_write_kb":80,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,4,10,15,2,14],"acting":[1,11,4,10,15,2,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"b","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"15","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":65,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-05T04:54:15.078086+0000","last_change":"2024-02-05T04:53:12.040500+0000","last_active":"2024-02-05T04:54:15.078086+0000","last_peered":"2024-02-05T04:54:15.078086+0000","last_clean":"2024-02-05T04:54:15.078086+0000","last_became_active":"2024-02-05T04:53:12.040297+0000","last_became_peered":"2024-02-05T04:53:12.040297+0000","last_unstale":"2024-02-05T04:54:15.078086+0000","last_undegraded":"2024-02-05T04:54:15.078086+0000","last_fullsized":"2024-02-05T04:54:15.078086+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_clean_scrub_stamp":"2024-02-05T04:53:11.017155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:20.053075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"84'86","reported_seq":348,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-05T04:54:18.085590+0000","last_change":"2024-02-05T04:54:15.079713+0000","last_active":"2024-02-05T04:54:18.085590+0000","last_peered":"2024-02-05T04:54:18.085590+0000","last_clean":"2024-02-05T04:54:18.085590+0000","last_became_active":"2024-02-05T04:53:54.991703+0000","last_became_peered":"2024-02-05T04:53:54.991703+0000","last_unstale":"2024-02-05T04:54:18.085590+0000","last_undegraded":"2024-02-05T04:54:18.085590+0000","last_fullsized":"2024-02-05T04:54:18.085590+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'53","last_scrub_stamp":"2024-02-05T04:53:55.645439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:55.645439+0000","objects_scrubbed":2,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:55:15.538730+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0015878330000000001,"stat_sum":{"num_bytes":11272192,"num_objects":6,"num_object_clones":4,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":8,"num_read_kb":3747,"num_write":13,"num_write_kb":8192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,2,3,6,15,10,11],"acting":[12,2,3,6,15,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"7","length":"1"},{"start":"b","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"15","length":"1"},{"start":"1a","length":"2"}]},{"pgid":"2.3","version":"0'0","reported_seq":65,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-05T04:54:15.081653+0000","last_change":"2024-02-05T04:53:12.028466+0000","last_active":"2024-02-05T04:54:15.081653+0000","last_peered":"2024-02-05T04:54:15.081653+0000","last_clean":"2024-02-05T04:54:15.081653+0000","last_became_active":"2024-02-05T04:53:12.028138+0000","last_became_peered":"2024-02-05T04:53:12.028138+0000","last_unstale":"2024-02-05T04:54:15.081653+0000","last_undegraded":"2024-02-05T04:54:15.081653+0000","last_fullsized":"2024-02-05T04:54:15.081653+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:11.017155+0000","last_clean_scrub_stamp":"2024-02-05T04:53:11.017155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:38.427375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,15],"acting":[4,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"26'157","reported_seq":259,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-05T04:54:17.078171+0000","last_change":"2024-02-05T04:53:49.948646+0000","last_active":"2024-02-05T04:54:17.078171+0000","last_peered":"2024-02-05T04:54:17.078171+0000","last_clean":"2024-02-05T04:54:17.078171+0000","last_became_active":"2024-02-05T04:53:49.948231+0000","last_became_peered":"2024-02-05T04:53:49.948231+0000","last_unstale":"2024-02-05T04:54:17.078171+0000","last_undegraded":"2024-02-05T04:54:17.078171+0000","last_fullsized":"2024-02-05T04:54:17.078171+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:07.900741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:07.900741+0000","last_clean_scrub_stamp":"2024-02-05T04:53:07.900741+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:29.642907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"81'39","reported_seq":263,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-05T04:54:16.157873+0000","last_change":"2024-02-05T04:54:15.082501+0000","last_active":"2024-02-05T04:54:16.157873+0000","last_peered":"2024-02-05T04:54:16.157873+0000","last_clean":"2024-02-05T04:54:16.157873+0000","last_became_active":"2024-02-05T04:53:54.985712+0000","last_became_peered":"2024-02-05T04:53:54.985712+0000","last_unstale":"2024-02-05T04:54:16.157873+0000","last_undegraded":"2024-02-05T04:54:16.157873+0000","last_fullsized":"2024-02-05T04:54:16.157873+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'27","last_scrub_stamp":"2024-02-05T04:53:55.900246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:55.900246+0000","objects_scrubbed":4,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:55:18.716643+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0019324889999999999,"stat_sum":{"num_bytes":1359872,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":241,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,0,14,15,6],"acting":[4,1,2,0,14,15,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"7","length":"1"},{"start":"b","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"15","length":"1"},{"start":"1a","length":"2"}]},{"pgid":"3.9","version":"84'27","reported_seq":265,"reported_epoch":84,"state":"peering","last_fresh":"2024-02-05T04:54:16.179492+0000","last_change":"2024-02-05T04:54:17.082648+0000","last_active":"2024-02-05T04:54:17.082648+0000","last_peered":"2024-02-05T04:54:16.179492+0000","last_clean":"2024-02-05T04:54:12.083688+0000","last_became_active":"2024-02-05T04:54:16.089165+0000","last_became_peered":"2024-02-05T04:54:16.089165+0000","last_unstale":"2024-02-05T04:54:16.179492+0000","last_undegraded":"2024-02-05T04:54:16.179492+0000","last_fullsized":"2024-02-05T04:54:16.078935+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'21","last_scrub_stamp":"2024-02-05T04:53:55.970267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:55.970267+0000","objects_scrubbed":1,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00078582000000000001,"stat_sum":{"num_bytes":983040,"num_objects":3,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":3,"num_write_kb":448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1507328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0,4,3,15,9,10],"acting":[14,0,4,3,15,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,3,4,9,10,15],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"b","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"15","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"3.a","version":"0'0","reported_seq":207,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-05T04:54:15.082751+0000","last_change":"2024-02-05T04:54:15.082751+0000","last_active":"2024-02-05T04:54:15.082751+0000","last_peered":"2024-02-05T04:54:15.082751+0000","last_clean":"2024-02-05T04:54:15.082751+0000","last_became_active":"2024-02-05T04:53:46.968151+0000","last_became_peered":"2024-02-05T04:53:46.968151+0000","last_unstale":"2024-02-05T04:54:15.082751+0000","last_undegraded":"2024-02-05T04:54:15.082751+0000","last_fullsized":"2024-02-05T04:54:15.082751+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:23.066897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:39.169161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.004329734,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,10,9,14,6,11],"acting":[3,7,10,9,14,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"7","length":"1"},{"start":"b","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"15","length":"1"},{"start":"1a","length":"2"}]},{"pgid":"3.b","version":"75'105","reported_seq":344,"reported_epoch":75,"state":"stale+active+clean","last_fresh":"2024-02-05T04:54:07.071586+0000","last_change":"2024-02-05T04:54:07.072264+0000","last_active":"2024-02-05T04:54:07.071586+0000","last_peered":"2024-02-05T04:54:07.071586+0000","last_clean":"2024-02-05T04:54:07.071586+0000","last_became_active":"2024-02-05T04:53:49.953645+0000","last_became_peered":"2024-02-05T04:53:49.953645+0000","last_unstale":"2024-02-05T04:54:13.069385+0000","last_undegraded":"2024-02-05T04:54:07.071586+0000","last_fullsized":"2024-02-05T04:54:07.071586+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:23.066897+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:44.803323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.017960402,"stat_sum":{"num_bytes":12271616,"num_objects":11,"num_object_clones":6,"num_object_copies":77,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":58,"num_read_kb":18358,"num_write":46,"num_write_kb":14368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":8683520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,1,10,0,6,7,12],"acting":[13,1,10,0,6,7,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"9","length":"9"},{"start":"13","length":"3"}]},{"pgid":"3.c","version":"86'37","reported_seq":270,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-05T04:54:18.106728+0000","last_change":"2024-02-05T04:54:18.107517+0000","last_active":"2024-02-05T04:54:18.106728+0000","last_peered":"2024-02-05T04:54:18.106728+0000","last_clean":"2024-02-05T04:54:18.106728+0000","last_became_active":"2024-02-05T04:54:18.094384+0000","last_became_peered":"2024-02-05T04:54:18.094384+0000","last_unstale":"2024-02-05T04:54:18.106728+0000","last_undegraded":"2024-02-05T04:54:18.106728+0000","last_fullsized":"2024-02-05T04:54:18.106728+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:23.066897+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:48.520656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011297293,"stat_sum":{"num_bytes":5406720,"num_objects":3,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":10,"num_read_kb":197,"num_write":14,"num_write_kb":5280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":7716864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5,10,6,1,9],"acting":[11,2,5,10,6,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"7","length":"1"},{"start":"b","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"15","length":"1"},{"start":"1a","length":"2"}]},{"pgid":"3.d","version":"86'68","reported_seq":329,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-05T04:54:18.132509+0000","last_change":"2024-02-05T04:54:18.132946+0000","last_active":"2024-02-05T04:54:18.132509+0000","last_peered":"2024-02-05T04:54:18.132509+0000","last_clean":"2024-02-05T04:54:18.132509+0000","last_became_active":"2024-02-05T04:54:18.094570+0000","last_became_peered":"2024-02-05T04:54:18.094570+0000","last_unstale":"2024-02-05T04:54:18.132509+0000","last_undegraded":"2024-02-05T04:54:18.132509+0000","last_fullsized":"2024-02-05T04:54:18.132509+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:23.066897+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:48.569612+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.036415363999999999,"stat_sum":{"num_bytes":9781248,"num_objects":7,"num_object_clones":4,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":2,"num_read":37,"num_read_kb":28141,"num_write":46,"num_write_kb":8928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":10616832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,14,10,9,6,5],"acting":[7,3,14,10,9,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"7","length":"1"},{"start":"b","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"15","length":"1"},{"start":"1a","length":"2"}]},{"pgid":"3.e","version":"86'87","reported_seq":371,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-05T04:54:18.123521+0000","last_change":"2024-02-05T04:54:18.124583+0000","last_active":"2024-02-05T04:54:18.123521+0000","last_peered":"2024-02-05T04:54:18.123521+0000","last_clean":"2024-02-05T04:54:18.123521+0000","last_became_active":"2024-02-05T04:54:18.093557+0000","last_became_peered":"2024-02-05T04:54:18.093557+0000","last_unstale":"2024-02-05T04:54:18.123521+0000","last_undegraded":"2024-02-05T04:54:18.123521+0000","last_fullsized":"2024-02-05T04:54:18.123521+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:23.066897+0000","objects_scrubbed":0,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:40.237897+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.029511677,"stat_sum":{"num_bytes":11517952,"num_objects":8,"num_object_clones":6,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":19,"num_read_kb":8156,"num_write":76,"num_write_kb":11008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":24330240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7,11,14,6,5],"acting":[2,4,7,11,14,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"7","length":"1"},{"start":"b","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"15","length":"1"},{"start":"1a","length":"2"}]},{"pgid":"3.f","version":"84'120","reported_seq":342,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-05T04:54:18.087997+0000","last_change":"2024-02-05T04:54:15.093674+0000","last_active":"2024-02-05T04:54:18.087997+0000","last_peered":"2024-02-05T04:54:18.087997+0000","last_clean":"2024-02-05T04:54:18.087997+0000","last_became_active":"2024-02-05T04:53:24.723119+0000","last_became_peered":"2024-02-05T04:53:24.723119+0000","last_unstale":"2024-02-05T04:54:18.087997+0000","last_undegraded":"2024-02-05T04:54:18.087997+0000","last_fullsized":"2024-02-05T04:54:18.087997+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:53:23.066897+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:54:34.255445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014056618,"stat_sum":{"num_bytes":17121280,"num_objects":19,"num_object_clones":12,"num_object_copies":133,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":3,"num_read":59,"num_read_kb":21030,"num_write":56,"num_write_kb":17216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,14,4,0,10,1],"acting":[11,2,14,4,0,10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"7","length":"1"},{"start":"b","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"15","length":"1"},{"start":"1a","length":"2"}]},{"pgid":"3.10","version":"83'17","reported_seq":114,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-05T04:54:18.086488+0000","last_change":"2024-02-05T04:54:15.084913+0000","last_active":"2024-02-05T04:54:18.086488+0000","last_peered":"2024-02-05T04:54:18.086488+0000","last_clean":"2024-02-05T04:54:18.086488+0000","last_became_active":"2024-02-05T04:53:59.961184+0000","last_became_peered":"2024-02-05T04:53:59.961184+0000","last_unstale":"2024-02-05T04:54:18.086488+0000","last_undegraded":"2024-02-05T04:54:18.086488+0000","last_fullsized":"2024-02-05T04:54:18.086488+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":5,"last_scrub":"34'5","last_scrub_stamp":"2024-02-05T04:54:05.647545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:54:05.647545+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:55:13.787915+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0080963400000000005,"stat_sum":{"num_bytes":2244608,"num_objects":3,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":3,"num_read_kb":1,"num_write":6,"num_write_kb":2080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,6,7,1,5,9,11],"acting":[12,6,7,1,5,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"7","length":"1"},{"start":"b","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"15","length":"1"},{"start":"1a","length":"2"}]},{"pgid":"3.11","version":"61'51","reported_seq":112,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-05T04:54:18.088674+0000","last_change":"2024-02-05T04:54:15.086013+0000","last_active":"2024-02-05T04:54:18.088674+0000","last_peered":"2024-02-05T04:54:18.088674+0000","last_clean":"2024-02-05T04:54:18.088674+0000","last_became_active":"2024-02-05T04:53:59.956213+0000","last_became_peered":"2024-02-05T04:53:59.956213+0000","last_unstale":"2024-02-05T04:54:18.088674+0000","last_undegraded":"2024-02-05T04:54:18.088674+0000","last_fullsized":"2024-02-05T04:54:18.088674+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":5,"last_scrub":"61'51","last_scrub_stamp":"2024-02-05T04:54:03.753770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:54:03.753770+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:55:05.531171+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0011613909999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,15,0,5,4,12],"acting":[10,11,15,0,5,4,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"7","length":"1"},{"start":"b","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"15","length":"1"},{"start":"1a","length":"2"}]},{"pgid":"3.12","version":"84'65","reported_seq":174,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-05T04:54:18.085453+0000","last_change":"2024-02-05T04:54:15.080530+0000","last_active":"2024-02-05T04:54:18.085453+0000","last_peered":"2024-02-05T04:54:18.085453+0000","last_clean":"2024-02-05T04:54:18.085453+0000","last_became_active":"2024-02-05T04:53:59.956704+0000","last_became_peered":"2024-02-05T04:53:59.956704+0000","last_unstale":"2024-02-05T04:54:18.085453+0000","last_undegraded":"2024-02-05T04:54:18.085453+0000","last_fullsized":"2024-02-05T04:54:18.085453+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":5,"last_scrub":"71'59","last_scrub_stamp":"2024-02-05T04:54:03.684079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:54:03.684079+0000","objects_scrubbed":4,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:55:07.277570+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.0010554589999999999,"stat_sum":{"num_bytes":1277952,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":9,"num_read_kb":160,"num_write":1,"num_write_kb":1248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5,6,11,3,15],"acting":[2,7,5,6,11,3,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"7","length":"1"},{"start":"b","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"15","length":"1"},{"start":"1a","length":"2"}]},{"pgid":"3.13","version":"81'52","reported_seq":282,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-05T04:54:18.085889+0000","last_change":"2024-02-05T04:54:15.085313+0000","last_active":"2024-02-05T04:54:18.085889+0000","last_peered":"2024-02-05T04:54:18.085889+0000","last_clean":"2024-02-05T04:54:18.085889+0000","last_became_active":"2024-02-05T04:53:59.956458+0000","last_became_peered":"2024-02-05T04:53:59.956458+0000","last_unstale":"2024-02-05T04:54:18.085889+0000","last_undegraded":"2024-02-05T04:54:18.085889+0000","last_fullsized":"2024-02-05T04:54:18.085889+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":5,"last_scrub":"72'46","last_scrub_stamp":"2024-02-05T04:54:04.649735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T04:53:23.066897+0000","last_clean_scrub_stamp":"2024-02-05T04:54:04.649735+0000","objects_scrubbed":3,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T04:55:08.729520+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0039922969999999997,"stat_sum":{"num_bytes":2555904,"num_objects":3,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,15,1,5,0,10],"acting":[11,7,15,1,5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"7","length":"1"},{"start":"b","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"15","length":"1"},{"start":"1a","length":"2"}]}],"pool_stats":[{"poolid":3,"num_pg":26,"stat_sum":{"num_bytes":135102464,"num_objects":129,"num_object_clones":82,"num_object_copies":903,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":129,"num_whiteouts":11,"num_read":233,"num_read_kb":105768,"num_write":340,"num_write_kb":85792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":94978048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":295047168,"data_stored":427462656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1445,"ondisk_log_size":1445,"up":182,"acting":182,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":12,"up_from":23,"seq":98784247825,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51776,"kb_used_data":20588,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94320064,"statfs":{"total":96636764160,"available":96583745536,"internally_reserved":0,"allocated":21082112,"data_stored":28452118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,9,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32500000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39700000000000002}]},{"osd":2,"last update":"Mon Feb 5 04:54:13 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":1.7689999999999999,"5min":1.7689999999999999,"15min":1.7689999999999999},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.46500000000000002}]},{"osd":3,"last update":"Mon Feb 5 04:54:13 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.8859999999999999,"5min":1.8859999999999999,"15min":1.8859999999999999},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.625}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.375}]},{"osd":5,"last update":"Mon Feb 5 04:54:13 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.42199999999999999}]},{"osd":6,"last update":"Mon Feb 5 04:54:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":0.49099999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35399999999999998}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.441}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52700000000000002}]},{"osd":11,"last update":"Mon Feb 5 04:54:13 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.9099999999999999,"5min":1.9099999999999999,"15min":1.9099999999999999},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.3799999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":0.70199999999999996}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70499999999999996}]},{"osd":14,"last update":"Mon Feb 5 04:54:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.6639999999999999,"5min":1.6639999999999999,"15min":1.6639999999999999},"last":0.56200000000000006}]},{"osd":15,"last update":"Mon Feb 5 04:54:13 2024","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":2.1299999999999999,"5min":2.1299999999999999,"15min":2.1299999999999999},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":1.827,"5min":1.827,"15min":1.827},"last":0.67500000000000004}]}]},{"osd":11,"up_from":23,"seq":98784247827,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48748,"kb_used_data":17560,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94323092,"statfs":{"total":96636764160,"available":96586846208,"internally_reserved":0,"allocated":17981440,"data_stored":26285334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,4,5,6,7,8,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 04:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.46200000000000002}]},{"osd":1,"last update":"Mon Feb 5 04:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.623}]},{"osd":2,"last update":"Mon Feb 5 04:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.755},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.58999999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41599999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41999999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77000000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70999999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40400000000000003}]},{"osd":8,"last update":"Mon Feb 5 04:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.38100000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69699999999999995}]},{"osd":10,"last update":"Mon Feb 5 04:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.149,"5min":1.149,"15min":1.149},"last":0.57099999999999995}]},{"osd":12,"last update":"Mon Feb 5 04:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.016,"5min":1.016,"15min":1.016},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.52900000000000003}]},{"osd":13,"last update":"Mon Feb 5 04:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.357,"5min":1.357,"15min":1.357},"last":0.879},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.167,"5min":1.167,"15min":1.167},"last":0.82599999999999996}]},{"osd":14,"last update":"Mon Feb 5 04:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.3120000000000001,"5min":1.3120000000000001,"15min":1.3120000000000001},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"min":{"1min":0.629,"5min":0.629,"15min":0.629},"max":{"1min":1.3109999999999999,"5min":1.3109999999999999,"15min":1.3109999999999999},"last":0.52200000000000002}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54100000000000004}]}]},{"osd":10,"up_from":23,"seq":98784247827,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54932,"kb_used_data":23744,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94316908,"statfs":{"total":96636764160,"available":96580513792,"internally_reserved":0,"allocated":24313856,"data_stored":33760534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,4,5,8,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 04:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.51500000000000001}]},{"osd":1,"last update":"Mon Feb 5 04:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.66600000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.503}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]},{"osd":8,"last update":"Mon Feb 5 04:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.432},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.48999999999999999}]},{"osd":9,"last update":"Mon Feb 5 04:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.48,"5min":1.48,"15min":1.48},"last":0.69299999999999995}]},{"osd":11,"last update":"Mon Feb 5 04:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.72099999999999997}]},{"osd":12,"last update":"Mon Feb 5 04:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.45100000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88500000000000001}]},{"osd":14,"last update":"Mon Feb 5 04:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.378}]},{"osd":15,"last update":"Mon Feb 5 04:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.2669999999999999,"5min":1.2669999999999999,"15min":1.2669999999999999},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":2.3759999999999999,"5min":2.3759999999999999,"15min":2.3759999999999999},"last":0.64400000000000002}]}]},{"osd":3,"up_from":21,"seq":90194313234,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49176,"kb_used_data":17992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322664,"statfs":{"total":96636764160,"available":96586407936,"internally_reserved":0,"allocated":18423808,"data_stored":21730612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 04:54:13 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.40799999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63300000000000001}]},{"osd":2,"last update":"Mon Feb 5 04:54:13 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.53900000000000003}]},{"osd":4,"last update":"Mon Feb 5 04:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.45800000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]},{"osd":6,"last update":"Mon Feb 5 04:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.64600000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.504}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48899999999999999}]},{"osd":9,"last update":"Mon Feb 5 04:54:13 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.101,"5min":1.101,"15min":1.101},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.68899999999999995}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66000000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.623}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71999999999999997}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":14,"last update":"Mon Feb 5 04:54:13 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.119,"5min":1.119,"15min":1.119},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.3080000000000001,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.78800000000000003}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69999999999999996}]}]},{"osd":4,"up_from":21,"seq":90194313233,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43272,"kb_used_data":16244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94328568,"statfs":{"total":96636764160,"available":96592453632,"internally_reserved":0,"allocated":16633856,"data_stored":27036118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,9,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39600000000000002}]},{"osd":1,"last update":"Mon Feb 5 04:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.35199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.373}]},{"osd":3,"last update":"Mon Feb 5 04:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.496},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.41799999999999998}]},{"osd":5,"last update":"Mon Feb 5 04:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":2.9079999999999999,"5min":2.9079999999999999,"15min":2.9079999999999999},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.171,"5min":1.171,"15min":1.171},"last":0.54200000000000004}]},{"osd":6,"last update":"Mon Feb 5 04:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":3.0179999999999998,"5min":3.0179999999999998,"15min":3.0179999999999998},"last":0.46899999999999997}]},{"osd":7,"last update":"Mon Feb 5 04:54:13 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.44},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":2.7890000000000001,"5min":2.7890000000000001,"15min":2.7890000000000001},"last":0.42899999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51000000000000001}]},{"osd":9,"last update":"Mon Feb 5 04:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":2.9849999999999999,"5min":2.9849999999999999,"15min":2.9849999999999999},"last":0.51100000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":14,"last update":"Mon Feb 5 04:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.073,"5min":1.073,"15min":1.073},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":3.0470000000000002,"5min":3.0470000000000002,"15min":3.0470000000000002},"last":0.55600000000000005}]},{"osd":15,"last update":"Mon Feb 5 04:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.903,"5min":1.903,"15min":1.903},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.57099999999999995}]}]},{"osd":5,"up_from":21,"seq":90194313233,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54736,"kb_used_data":23552,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94317104,"statfs":{"total":96636764160,"available":96580714496,"internally_reserved":0,"allocated":24117248,"data_stored":33075988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,6,7,9,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 04:54:11 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"last":0.53600000000000003}]},{"osd":3,"last update":"Mon Feb 5 04:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.59299999999999997}]},{"osd":4,"last update":"Mon Feb 5 04:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.73199999999999998}]},{"osd":6,"last update":"Mon Feb 5 04:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.57399999999999995}]},{"osd":7,"last update":"Mon Feb 5 04:54:11 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.66600000000000004}]},{"osd":9,"last update":"Mon Feb 5 04:54:11 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.753},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.79500000000000004}]},{"osd":12,"last update":"Mon Feb 5 04:54:11 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.76900000000000002}]},{"osd":13,"last update":"Mon Feb 5 04:54:11 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.57099999999999995}]},{"osd":14,"last update":"Mon Feb 5 04:54:11 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.069,"5min":1.069,"15min":1.069},"last":0.78400000000000003}]},{"osd":15,"last update":"Mon Feb 5 04:54:11 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.80900000000000005}]}]},{"osd":6,"up_from":21,"seq":90194313234,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52248,"kb_used_data":21064,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94319592,"statfs":{"total":96636764160,"available":96583262208,"internally_reserved":0,"allocated":21569536,"data_stored":31568660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57999999999999996}]},{"osd":1,"last update":"Mon Feb 5 04:54:08 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.68700000000000006}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61899999999999999}]},{"osd":3,"last update":"Mon Feb 5 04:54:08 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.753},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.77800000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90000000000000002}]},{"osd":5,"last update":"Mon Feb 5 04:54:08 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.98599999999999999},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.88600000000000001}]},{"osd":7,"last update":"Mon Feb 5 04:54:11 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.94399999999999995},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.86699999999999999}]},{"osd":8,"last update":"Mon Feb 5 04:54:11 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.80000000000000004}]},{"osd":9,"last update":"Mon Feb 5 04:54:08 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.93100000000000005},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.96599999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95799999999999996}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.012}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59499999999999997}]},{"osd":14,"last update":"Mon Feb 5 04:54:08 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.91100000000000003},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.85099999999999998}]},{"osd":15,"last update":"Mon Feb 5 04:54:08 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.097,"5min":1.097,"15min":1.097},"last":1.075},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":1.044}]}]},{"osd":1,"up_from":22,"seq":94489280530,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57388,"kb_used_data":26208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94314452,"statfs":{"total":96636764160,"available":96577998848,"internally_reserved":0,"allocated":26836992,"data_stored":39420692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,4,5,8,9,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 04:54:08 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.41599999999999998}]},{"osd":2,"last update":"Mon Feb 5 04:54:08 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.51500000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53300000000000003}]},{"osd":5,"last update":"Mon Feb 5 04:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.68300000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.626}]},{"osd":8,"last update":"Mon Feb 5 04:54:08 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.80500000000000005}]},{"osd":9,"last update":"Mon Feb 5 04:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.62},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.59099999999999997}]},{"osd":10,"last update":"Mon Feb 5 04:54:08 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.56999999999999995}]},{"osd":13,"last update":"Mon Feb 5 04:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.89200000000000002}]},{"osd":14,"last update":"Mon Feb 5 04:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.075,"5min":1.075,"15min":1.075},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.69999999999999996}]},{"osd":15,"last update":"Mon Feb 5 04:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.93100000000000005},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.79100000000000004}]}]},{"osd":14,"up_from":22,"seq":94489280529,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48664,"kb_used_data":21640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94323176,"statfs":{"total":96636764160,"available":96586932224,"internally_reserved":0,"allocated":22159360,"data_stored":26660037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36899999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48799999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35099999999999998}]},{"osd":3,"last update":"Mon Feb 5 04:54:08 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.57199999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42799999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52800000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47799999999999998}]},{"osd":8,"last update":"Mon Feb 5 04:54:14 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.55700000000000005}]},{"osd":9,"last update":"Mon Feb 5 04:54:08 2024","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.63200000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59699999999999998}]},{"osd":11,"last update":"Mon Feb 5 04:54:14 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.68000000000000005}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70799999999999996}]},{"osd":13,"last update":"Mon Feb 5 04:54:08 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.181,"5min":1.181,"15min":1.181},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.1699999999999999},"last":0.81699999999999995}]},{"osd":15,"last update":"Mon Feb 5 04:54:08 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.746,"5min":0.746,"15min":0.746},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.65400000000000003}]}]},{"osd":9,"up_from":22,"seq":94489280529,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39548,"kb_used_data":12524,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94332292,"statfs":{"total":96636764160,"available":96596267008,"internally_reserved":0,"allocated":12824576,"data_stored":16374981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,4,6,7,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44700000000000001}]},{"osd":1,"last update":"Mon Feb 5 04:54:09 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.372,"15min":0.372},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.26300000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]},{"osd":4,"last update":"Mon Feb 5 04:54:14 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.77800000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30599999999999999}]},{"osd":6,"last update":"Mon Feb 5 04:54:09 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":1.492},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.63100000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76800000000000002}]},{"osd":8,"last update":"Mon Feb 5 04:54:09 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.83799999999999997}]},{"osd":10,"last update":"Mon Feb 5 04:54:09 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":1.931}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4750000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81100000000000005}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77600000000000002}]},{"osd":14,"last update":"Mon Feb 5 04:54:09 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.86199999999999999}]},{"osd":15,"last update":"Mon Feb 5 04:54:09 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":1.6299999999999999},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":1.7170000000000001}]}]},{"osd":13,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":23,"seq":98784247826,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49812,"kb_used_data":18624,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322028,"statfs":{"total":96636764160,"available":96585756672,"internally_reserved":0,"allocated":19070976,"data_stored":25298198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 04:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":1.4259999999999999,"5min":1.4259999999999999,"15min":1.4259999999999999},"last":0.371},{"interface":"front","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.40799999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47299999999999998}]},{"osd":3,"last update":"Mon Feb 5 04:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.6759999999999999,"5min":1.6759999999999999,"15min":1.6759999999999999},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.42199999999999999}]},{"osd":4,"last update":"Mon Feb 5 04:54:08 2024","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.317}]},{"osd":5,"last update":"Mon Feb 5 04:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":1.5660000000000001,"5min":1.5660000000000001,"15min":1.5660000000000001},"last":0.33300000000000002}]},{"osd":6,"last update":"Mon Feb 5 04:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":2.4929999999999999,"5min":2.4929999999999999,"15min":2.4929999999999999},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.284,"5min":1.284,"15min":1.284},"last":0.56100000000000005}]},{"osd":7,"last update":"Mon Feb 5 04:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.7549999999999999,"5min":1.7549999999999999,"15min":1.7549999999999999},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":2.7629999999999999,"5min":2.7629999999999999,"15min":2.7629999999999999},"last":0.39100000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.376}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38600000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57999999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42999999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60199999999999998}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77700000000000002}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.625}]},{"osd":15,"last update":"Mon Feb 5 04:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.7150000000000001,"5min":1.7150000000000001,"15min":1.7150000000000001},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.351,"5min":1.351,"15min":1.351},"last":0.51500000000000001}]}]},{"osd":15,"up_from":22,"seq":94489280530,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42996,"kb_used_data":15972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94328844,"statfs":{"total":96636764160,"available":96592736256,"internally_reserved":0,"allocated":16355328,"data_stored":23375045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7,8,10,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 04:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":1.52,"5min":1.52,"15min":1.52},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.35899999999999999}]},{"osd":1,"last update":"Mon Feb 5 04:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.4850000000000001,"5min":1.4850000000000001,"15min":1.4850000000000001},"last":0.497}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51900000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38400000000000001}]},{"osd":4,"last update":"Mon Feb 5 04:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.51600000000000001}]},{"osd":5,"last update":"Mon Feb 5 04:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.6479999999999999,"5min":1.6479999999999999,"15min":1.6479999999999999},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.53100000000000003}]},{"osd":6,"last update":"Mon Feb 5 04:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.0509999999999999,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.429,"5min":1.429,"15min":1.429},"last":0.68799999999999994}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61399999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71499999999999997}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71699999999999997}]},{"osd":13,"last update":"Mon Feb 5 04:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.1259999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.73599999999999999}]},{"osd":14,"last update":"Mon Feb 5 04:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.198,"5min":1.198,"15min":1.198},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.119,"5min":1.119,"15min":1.119},"last":0.70199999999999996}]}]},{"osd":2,"up_from":23,"seq":98784247827,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50820,"kb_used_data":19636,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94321020,"statfs":{"total":96636764160,"available":96584724480,"internally_reserved":0,"allocated":20107264,"data_stored":29116214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 04:54:08 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.436},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.1799999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.495}]},{"osd":3,"last update":"Mon Feb 5 04:54:08 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.51100000000000001}]},{"osd":4,"last update":"Mon Feb 5 04:54:08 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.39100000000000001}]},{"osd":5,"last update":"Mon Feb 5 04:54:08 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.36599999999999999}]},{"osd":6,"last update":"Mon Feb 5 04:54:08 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.016,"5min":1.016,"15min":1.016},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.313}]},{"osd":7,"last update":"Mon Feb 5 04:54:08 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":1.224,"5min":1.224,"15min":1.224},"last":0.60099999999999998}]},{"osd":8,"last update":"Mon Feb 5 04:54:08 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.44900000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78800000000000003}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38200000000000001}]},{"osd":11,"last update":"Mon Feb 5 04:54:08 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.05,"5min":1.05,"15min":1.05},"last":0.622}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60599999999999998}]},{"osd":13,"last update":"Mon Feb 5 04:54:08 2024","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.2869999999999999,"5min":1.2869999999999999,"15min":1.2869999999999999},"last":1.2869999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52500000000000002}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65100000000000002}]}]},{"osd":7,"up_from":23,"seq":98784247827,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52696,"kb_used_data":21508,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94319144,"statfs":{"total":96636764160,"available":96582803456,"internally_reserved":0,"allocated":22024192,"data_stored":32249110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 04:54:09 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.32000000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.376}]},{"osd":2,"last update":"Mon Feb 5 04:54:09 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.42599999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36099999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90400000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1100000000000001}]},{"osd":6,"last update":"Mon Feb 5 04:54:09 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.4630000000000001,"5min":1.4630000000000001,"15min":1.4630000000000001},"last":1.1830000000000001},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":1.2150000000000001}]},{"osd":8,"last update":"Mon Feb 5 04:54:09 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.98799999999999999}]},{"osd":9,"last update":"Mon Feb 5 04:54:09 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":1.0580000000000001},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.165,"5min":1.165,"15min":1.165},"last":1.1519999999999999}]},{"osd":10,"last update":"Mon Feb 5 04:54:09 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.93700000000000006},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.68700000000000006}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.23}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67600000000000005}]},{"osd":13,"last update":"Mon Feb 5 04:54:09 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.224,"5min":1.224,"15min":1.224},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.05,"5min":1.05,"15min":1.05},"last":0.84899999999999998}]},{"osd":14,"last update":"Mon Feb 5 04:54:09 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":1.137,"5min":1.137,"15min":1.137},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.5069999999999999,"5min":1.5069999999999999,"15min":1.5069999999999999},"last":1.024}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.623}]}]},{"osd":8,"up_from":0,"seq":98784247824,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":17825792,"data_stored":24580096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":25624576,"data_stored":38723584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":18329600,"data_stored":27869184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16678912,"data_stored":20504576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16621568,"data_stored":27951104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":22904832,"data_stored":32378880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20357120,"data_stored":30871552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20779008,"data_stored":31531008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":9826304,"data_stored":12816384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":11595776,"data_stored":15667200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":23068672,"data_stored":33042432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16736256,"data_stored":25567232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":19836928,"data_stored":27734016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":18804736,"data_stored":29605888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":20930560,"data_stored":25952256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":15126528,"data_stored":22667264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T04:54:20.974 INFO:tasks.ceph.ceph_manager.ceph:removing pool_name unique_pool_0 2024-02-05T04:54:20.975 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-02-05T04:54:21.046 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-02-05T04:54:21.046 INFO:teuthology.orchestra.run.smithi062.stdout: "size": 20, 2024-02-05T04:54:21.046 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 600, 2024-02-05T04:54:21.046 INFO:teuthology.orchestra.run.smithi062.stdout: "ops": [ 2024-02-05T04:54:21.046 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.046 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.bs3 52/31 ECSubWrite(tid=202, reqid=osd.13.0:200, at_version=0'0, trim_to=0'0, roll_forward_to=52'50))", 2024-02-05T04:54:21.046 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:43.959201+0000", 2024-02-05T04:54:21.046 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 37.084152005, 2024-02-05T04:54:21.046 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0055030230000000001, 2024-02-05T04:54:21.047 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:21.047 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:54:21.047 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:21.047 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.047 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:21.047 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.959201+0000", 2024-02-05T04:54:21.047 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:21.047 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.047 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.047 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:21.047 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.959201+0000", 2024-02-05T04:54:21.047 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:21.047 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.047 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.048 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:21.048 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.959202+0000", 2024-02-05T04:54:21.048 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.4899999999999995e-07 2024-02-05T04:54:21.048 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.048 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.048 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:21.048 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.959211+0000", 2024-02-05T04:54:21.048 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.4350000000000003e-06 2024-02-05T04:54:21.048 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.048 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.048 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:21.048 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.959221+0000", 2024-02-05T04:54:21.048 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.8700000000000004e-06 2024-02-05T04:54:21.048 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.049 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.049 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:21.049 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.959227+0000", 2024-02-05T04:54:21.049 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.1349999999999997e-06 2024-02-05T04:54:21.049 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.049 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.049 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:21.049 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.959613+0000", 2024-02-05T04:54:21.049 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00038529800000000001 2024-02-05T04:54:21.049 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.049 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.049 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:21.049 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.959622+0000", 2024-02-05T04:54:21.049 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.4339999999999998e-06 2024-02-05T04:54:21.050 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.050 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.050 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:21.050 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.964670+0000", 2024-02-05T04:54:21.050 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.005047719 2024-02-05T04:54:21.050 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.050 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.050 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:21.050 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:43.964704+0000", 2024-02-05T04:54:21.050 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.4483000000000002e-05 2024-02-05T04:54:21.050 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.050 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:21.050 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.050 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.050 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.051 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpRead(3.bs3 58/57 ECSubRead(tid=242, to_read={3:df6528f1:::smithi19639362-33:head=0,532480,0}, subchunks={3:df6528f1:::smithi19639362-33:head=[0,1]}, attrs_to_read=))", 2024-02-05T04:54:21.051 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:49.955701+0000", 2024-02-05T04:54:21.051 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 31.087651920999999, 2024-02-05T04:54:21.051 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0063665620000000001, 2024-02-05T04:54:21.051 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:21.051 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:54:21.051 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:21.051 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.051 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:21.051 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.955701+0000", 2024-02-05T04:54:21.051 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:21.051 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.051 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.051 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:21.052 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.955701+0000", 2024-02-05T04:54:21.052 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:21.052 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.052 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.052 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:21.052 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.955702+0000", 2024-02-05T04:54:21.052 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.0900000000000001e-07 2024-02-05T04:54:21.052 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.052 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.052 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:21.052 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.955710+0000", 2024-02-05T04:54:21.052 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.2220000000000002e-06 2024-02-05T04:54:21.052 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.052 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.053 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:21.053 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.955732+0000", 2024-02-05T04:54:21.053 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.2018000000000002e-05 2024-02-05T04:54:21.053 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.053 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.053 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:21.053 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.955745+0000", 2024-02-05T04:54:21.053 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.3025e-05 2024-02-05T04:54:21.053 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.053 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.053 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:21.053 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.955862+0000", 2024-02-05T04:54:21.053 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00011653400000000001 2024-02-05T04:54:21.053 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.054 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.054 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:21.054 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.962068+0000", 2024-02-05T04:54:21.054 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0062060539999999999 2024-02-05T04:54:21.054 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.054 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:21.054 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.054 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.054 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.054 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.4s3 58/57 ECSubWrite(tid=107, reqid=osd.1.0:107, at_version=58'59, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:21.054 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:49.956038+0000", 2024-02-05T04:54:21.054 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 31.087315062999998, 2024-02-05T04:54:21.054 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.005678163, 2024-02-05T04:54:21.054 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:21.055 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:54:21.055 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:21.055 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.055 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:21.055 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956038+0000", 2024-02-05T04:54:21.055 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:21.055 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.055 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.055 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:21.055 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956038+0000", 2024-02-05T04:54:21.055 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:21.055 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.055 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.055 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:21.056 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956039+0000", 2024-02-05T04:54:21.056 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.2200000000000003e-07 2024-02-05T04:54:21.056 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.056 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.056 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:21.056 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956068+0000", 2024-02-05T04:54:21.056 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.9013000000000002e-05 2024-02-05T04:54:21.056 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.056 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.056 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:21.056 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956082+0000", 2024-02-05T04:54:21.056 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.4066999999999999e-05 2024-02-05T04:54:21.056 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.056 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.057 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:21.057 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956091+0000", 2024-02-05T04:54:21.057 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.9539999999999993e-06 2024-02-05T04:54:21.057 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.057 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.057 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:21.057 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956177+0000", 2024-02-05T04:54:21.057 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.5548000000000002e-05 2024-02-05T04:54:21.057 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.057 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.057 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:21.057 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956187+0000", 2024-02-05T04:54:21.057 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0352e-05 2024-02-05T04:54:21.057 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.058 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.058 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:21.058 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.961666+0000", 2024-02-05T04:54:21.058 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0054792979999999996 2024-02-05T04:54:21.058 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.058 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.058 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:21.058 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.961716+0000", 2024-02-05T04:54:21.058 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.0108999999999997e-05 2024-02-05T04:54:21.058 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.058 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:21.058 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.058 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.058 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.058 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.4s3 58/57 ECSubWrite(tid=108, reqid=osd.1.0:108, at_version=58'61, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:21.059 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:49.956139+0000", 2024-02-05T04:54:21.059 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 31.087214488000001, 2024-02-05T04:54:21.059 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0079487310000000005, 2024-02-05T04:54:21.059 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:21.059 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:54:21.059 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:21.059 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.059 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:21.059 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956139+0000", 2024-02-05T04:54:21.059 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:21.059 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.059 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.059 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:21.059 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956139+0000", 2024-02-05T04:54:21.060 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:21.060 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.060 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.060 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:21.060 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956139+0000", 2024-02-05T04:54:21.060 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.0999999999999999e-07 2024-02-05T04:54:21.060 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.060 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.060 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:21.060 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956165+0000", 2024-02-05T04:54:21.060 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.5202999999999999e-05 2024-02-05T04:54:21.060 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.060 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.061 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:21.061 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956176+0000", 2024-02-05T04:54:21.061 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1523e-05 2024-02-05T04:54:21.061 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.061 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.061 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:21.061 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956184+0000", 2024-02-05T04:54:21.061 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.0940000000000008e-06 2024-02-05T04:54:21.061 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.061 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.061 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:21.061 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956668+0000", 2024-02-05T04:54:21.061 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000483535 2024-02-05T04:54:21.061 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.061 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.062 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:21.062 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956681+0000", 2024-02-05T04:54:21.062 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.2958e-05 2024-02-05T04:54:21.062 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.062 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.062 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:21.062 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.964043+0000", 2024-02-05T04:54:21.062 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0073623769999999998 2024-02-05T04:54:21.062 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.062 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.062 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:21.062 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.964088+0000", 2024-02-05T04:54:21.062 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.4530999999999998e-05 2024-02-05T04:54:21.063 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.063 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:21.063 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.063 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.063 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.063 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpRead(3.bs3 58/57 ECSubRead(tid=243, to_read={3:db93bf34:::smithi19639362-24:head=0,98304,0}, subchunks={3:db93bf34:::smithi19639362-24:head=[0,1]}, attrs_to_read=))", 2024-02-05T04:54:21.063 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:49.956639+0000", 2024-02-05T04:54:21.063 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 31.086713824, 2024-02-05T04:54:21.063 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0056018429999999996, 2024-02-05T04:54:21.063 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:21.063 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:54:21.063 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:21.063 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.063 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:21.064 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956639+0000", 2024-02-05T04:54:21.064 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:21.064 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.064 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.064 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:21.064 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956639+0000", 2024-02-05T04:54:21.064 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:21.064 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.064 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.064 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:21.064 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956640+0000", 2024-02-05T04:54:21.064 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.6800000000000009e-07 2024-02-05T04:54:21.064 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.065 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.065 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:21.065 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956650+0000", 2024-02-05T04:54:21.065 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.9739999999999997e-06 2024-02-05T04:54:21.065 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.065 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.065 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:21.065 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956681+0000", 2024-02-05T04:54:21.065 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.0655999999999999e-05 2024-02-05T04:54:21.065 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.065 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.065 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:21.065 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.956707+0000", 2024-02-05T04:54:21.066 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.5469999999999998e-05 2024-02-05T04:54:21.066 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.066 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.066 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:21.066 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.962076+0000", 2024-02-05T04:54:21.066 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0053699300000000002 2024-02-05T04:54:21.066 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.066 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.066 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:21.066 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:49.962241+0000", 2024-02-05T04:54:21.066 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000164845 2024-02-05T04:54:21.066 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.066 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:21.066 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.067 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.067 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.067 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.14s3 68/67 ECSubWrite(tid=17, reqid=osd.3.0:16, at_version=0'0, trim_to=0'0, roll_forward_to=68'77))", 2024-02-05T04:54:21.067 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:59.974045+0000", 2024-02-05T04:54:21.067 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 21.069307983000002, 2024-02-05T04:54:21.067 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0081087980000000004, 2024-02-05T04:54:21.067 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:21.067 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:54:21.067 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:21.067 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.067 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:21.067 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.974045+0000", 2024-02-05T04:54:21.067 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:21.067 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.068 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.068 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:21.068 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.974045+0000", 2024-02-05T04:54:21.068 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:21.068 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.068 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.068 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:21.068 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.974046+0000", 2024-02-05T04:54:21.068 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.5400000000000001e-07 2024-02-05T04:54:21.068 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.068 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.068 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:21.068 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.974056+0000", 2024-02-05T04:54:21.069 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.9960000000000005e-06 2024-02-05T04:54:21.069 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.069 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.069 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:21.069 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.974068+0000", 2024-02-05T04:54:21.069 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.2502e-05 2024-02-05T04:54:21.069 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.069 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.069 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:21.069 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.974078+0000", 2024-02-05T04:54:21.069 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.1889999999999996e-06 2024-02-05T04:54:21.069 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.069 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.070 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:21.070 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.974140+0000", 2024-02-05T04:54:21.070 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.2535999999999999e-05 2024-02-05T04:54:21.070 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.070 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.070 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:21.070 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.974151+0000", 2024-02-05T04:54:21.070 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1263e-05 2024-02-05T04:54:21.070 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.070 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.070 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:21.070 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.982119+0000", 2024-02-05T04:54:21.070 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0079679380000000008 2024-02-05T04:54:21.070 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.071 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.071 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:21.071 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:59.982154+0000", 2024-02-05T04:54:21.071 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.472e-05 2024-02-05T04:54:21.071 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.071 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:21.071 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.071 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.071 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.071 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.bs3 74/57 ECSubWrite(tid=340, reqid=client.4634.0:687, at_version=74'92, trim_to=0'0, roll_forward_to=74'88))", 2024-02-05T04:54:21.071 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:54:06.054652+0000", 2024-02-05T04:54:21.071 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 14.988701317, 2024-02-05T04:54:21.071 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0058186510000000002, 2024-02-05T04:54:21.071 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:21.072 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:54:21.072 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:21.072 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.072 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:21.072 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.054652+0000", 2024-02-05T04:54:21.072 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:21.072 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.072 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.072 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:21.072 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.054652+0000", 2024-02-05T04:54:21.072 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:21.072 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.072 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.072 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:21.072 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.054653+0000", 2024-02-05T04:54:21.073 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.5000000000000002e-07 2024-02-05T04:54:21.073 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.073 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.073 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:21.073 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.055108+0000", 2024-02-05T04:54:21.073 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00045540499999999998 2024-02-05T04:54:21.073 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.073 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.073 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:21.073 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.055119+0000", 2024-02-05T04:54:21.073 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1286e-05 2024-02-05T04:54:21.073 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.073 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.073 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:21.074 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.055127+0000", 2024-02-05T04:54:21.074 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.61e-06 2024-02-05T04:54:21.074 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.074 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.074 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:21.074 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.055190+0000", 2024-02-05T04:54:21.074 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.2718999999999994e-05 2024-02-05T04:54:21.074 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.074 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.074 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:21.074 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.055200+0000", 2024-02-05T04:54:21.075 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.9720000000000004e-06 2024-02-05T04:54:21.075 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.075 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.075 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:21.075 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.060431+0000", 2024-02-05T04:54:21.075 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0052310940000000004 2024-02-05T04:54:21.075 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.075 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.075 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:21.075 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.060471+0000", 2024-02-05T04:54:21.075 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.9814999999999997e-05 2024-02-05T04:54:21.075 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.075 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:21.076 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.076 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.076 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.076 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:693 3.17s0 3:e8f6ce71:::smithi19639362-28:head [append 0~540672] snapc 18=[18,17,16,15,12,8,7] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-02-05T04:54:21.076 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:54:06.073867+0000", 2024-02-05T04:54:21.076 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 14.969486295999999, 2024-02-05T04:54:21.076 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.013133568999999999, 2024-02-05T04:54:21.076 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:21.076 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:21.076 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:54:21.076 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:54:21.076 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:21.076 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 693 2024-02-05T04:54:21.076 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.077 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:21.077 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.077 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:21.077 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.073867+0000", 2024-02-05T04:54:21.077 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:21.077 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.077 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.077 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:21.077 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.073867+0000", 2024-02-05T04:54:21.077 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:21.077 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.077 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.077 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:21.077 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.073869+0000", 2024-02-05T04:54:21.077 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.59e-06 2024-02-05T04:54:21.078 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.078 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.078 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:21.078 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.074544+0000", 2024-02-05T04:54:21.078 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00067502299999999996 2024-02-05T04:54:21.078 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.078 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.078 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:21.078 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.074555+0000", 2024-02-05T04:54:21.078 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1780000000000001e-05 2024-02-05T04:54:21.078 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.078 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.078 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:21.078 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.074561+0000", 2024-02-05T04:54:21.078 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.8080000000000001e-06 2024-02-05T04:54:21.079 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.079 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.079 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:21.079 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.074606+0000", 2024-02-05T04:54:21.079 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.4468000000000002e-05 2024-02-05T04:54:21.079 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.079 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.079 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:54:21.079 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.074688+0000", 2024-02-05T04:54:21.079 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.2262999999999995e-05 2024-02-05T04:54:21.079 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.079 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.079 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:21.079 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.075525+0000", 2024-02-05T04:54:21.080 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000837023 2024-02-05T04:54:21.080 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.080 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.080 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:21.080 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.079967+0000", 2024-02-05T04:54:21.080 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0044417930000000003 2024-02-05T04:54:21.080 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.080 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.080 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:54:21.080 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.086948+0000", 2024-02-05T04:54:21.080 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0069808910000000004 2024-02-05T04:54:21.080 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.080 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.080 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:21.081 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.087001+0000", 2024-02-05T04:54:21.081 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.2930000000000003e-05 2024-02-05T04:54:21.081 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.081 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:21.081 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.081 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.081 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.081 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.bs3 74/57 ECSubWrite(tid=355, reqid=osd.13.0:355, at_version=74'100, trim_to=0'0, roll_forward_to=74'96))", 2024-02-05T04:54:21.081 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:54:06.077090+0000", 2024-02-05T04:54:21.081 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 14.966263421000001, 2024-02-05T04:54:21.081 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0086224860000000004, 2024-02-05T04:54:21.081 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:21.081 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:54:21.081 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:21.081 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.081 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:21.082 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.077090+0000", 2024-02-05T04:54:21.082 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:21.082 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.082 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.082 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:21.082 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.077090+0000", 2024-02-05T04:54:21.082 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:21.082 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.082 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.082 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:21.082 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.077090+0000", 2024-02-05T04:54:21.082 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.2300000000000001e-07 2024-02-05T04:54:21.082 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.082 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.082 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:21.083 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.077103+0000", 2024-02-05T04:54:21.083 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.3005e-05 2024-02-05T04:54:21.083 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.083 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.083 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:21.083 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.077115+0000", 2024-02-05T04:54:21.083 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1331e-05 2024-02-05T04:54:21.083 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.083 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.083 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:21.083 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.077120+0000", 2024-02-05T04:54:21.083 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.6799999999999998e-06 2024-02-05T04:54:21.083 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.083 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.083 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:21.084 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.077280+0000", 2024-02-05T04:54:21.084 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000159588 2024-02-05T04:54:21.084 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.084 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.084 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:21.084 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.077291+0000", 2024-02-05T04:54:21.084 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1207e-05 2024-02-05T04:54:21.084 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.084 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.084 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:21.084 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.085681+0000", 2024-02-05T04:54:21.084 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0083902419999999991 2024-02-05T04:54:21.084 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.084 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.085 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:21.085 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.085712+0000", 2024-02-05T04:54:21.085 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.0910000000000001e-05 2024-02-05T04:54:21.085 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.085 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:21.085 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.085 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.085 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.085 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:694 3.17s0 3:e8f6ce71:::smithi19639362-28:head [append 0~720896] snapc 18=[18,17,16,15,12,8,7] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-02-05T04:54:21.085 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:54:06.077837+0000", 2024-02-05T04:54:21.085 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 14.965516437, 2024-02-05T04:54:21.085 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0092709699999999999, 2024-02-05T04:54:21.085 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:21.085 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:21.085 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:54:21.086 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:54:21.086 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:21.086 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 694 2024-02-05T04:54:21.086 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.086 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:21.086 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.086 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:21.086 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.077837+0000", 2024-02-05T04:54:21.086 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:21.086 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.086 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.086 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:21.086 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.077837+0000", 2024-02-05T04:54:21.086 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:21.086 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.087 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.087 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:21.087 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.077838+0000", 2024-02-05T04:54:21.087 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.6700000000000002e-07 2024-02-05T04:54:21.087 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.087 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.087 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:21.087 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.078520+0000", 2024-02-05T04:54:21.087 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00068213300000000002 2024-02-05T04:54:21.087 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.087 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.087 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:21.087 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.078532+0000", 2024-02-05T04:54:21.087 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1907e-05 2024-02-05T04:54:21.088 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.088 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.088 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:21.088 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.078538+0000", 2024-02-05T04:54:21.088 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.6690000000000003e-06 2024-02-05T04:54:21.088 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.088 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.088 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:21.088 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.078589+0000", 2024-02-05T04:54:21.088 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.0983999999999998e-05 2024-02-05T04:54:21.088 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.088 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.088 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:54:21.088 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.078656+0000", 2024-02-05T04:54:21.089 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.7088000000000005e-05 2024-02-05T04:54:21.089 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.089 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.089 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:21.089 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.079247+0000", 2024-02-05T04:54:21.089 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00059124199999999996 2024-02-05T04:54:21.089 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.089 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.089 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:21.089 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.080425+0000", 2024-02-05T04:54:21.089 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0011778680000000001 2024-02-05T04:54:21.089 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.089 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.089 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:54:21.089 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.087070+0000", 2024-02-05T04:54:21.090 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0066450069999999997 2024-02-05T04:54:21.090 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.090 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.090 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:21.090 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.087108+0000", 2024-02-05T04:54:21.090 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.8105000000000002e-05 2024-02-05T04:54:21.090 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.090 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:21.090 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.090 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.090 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.090 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:695 3.17s0 3:e8f6ce71:::smithi19639362-28:head [append 0~786432] snapc 18=[18,17,16,15,12,8,7] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-02-05T04:54:21.090 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:54:06.082921+0000", 2024-02-05T04:54:21.090 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 14.960432257000001, 2024-02-05T04:54:21.090 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0059995420000000001, 2024-02-05T04:54:21.091 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:21.091 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:21.091 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:54:21.091 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:54:21.091 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:21.091 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 695 2024-02-05T04:54:21.091 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.091 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:21.091 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.091 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:21.091 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.082921+0000", 2024-02-05T04:54:21.091 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:21.091 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.091 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.092 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:21.092 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.082921+0000", 2024-02-05T04:54:21.092 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:21.092 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.092 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.092 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:21.092 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.082922+0000", 2024-02-05T04:54:21.092 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.215e-06 2024-02-05T04:54:21.092 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.092 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.092 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:21.092 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.083627+0000", 2024-02-05T04:54:21.092 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00070471499999999996 2024-02-05T04:54:21.092 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.093 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.093 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:21.093 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.083639+0000", 2024-02-05T04:54:21.093 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1691e-05 2024-02-05T04:54:21.093 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.093 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.093 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:21.093 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.083644+0000", 2024-02-05T04:54:21.093 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.6930000000000004e-06 2024-02-05T04:54:21.093 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.093 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.093 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:21.093 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.083686+0000", 2024-02-05T04:54:21.093 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.1567999999999999e-05 2024-02-05T04:54:21.093 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.094 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.094 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:54:21.094 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.083758+0000", 2024-02-05T04:54:21.094 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.2107999999999995e-05 2024-02-05T04:54:21.094 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.094 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.094 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:21.094 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.084390+0000", 2024-02-05T04:54:21.094 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00063148200000000005 2024-02-05T04:54:21.094 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.094 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.094 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:21.094 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.087118+0000", 2024-02-05T04:54:21.094 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0027284560000000002 2024-02-05T04:54:21.094 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.094 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.095 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:54:21.095 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.088858+0000", 2024-02-05T04:54:21.095 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0017397090000000001 2024-02-05T04:54:21.095 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.095 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.095 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:21.095 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.088921+0000", 2024-02-05T04:54:21.095 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.2904999999999995e-05 2024-02-05T04:54:21.095 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.095 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:21.095 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.095 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.095 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.095 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:696 3.17s0 3:e8f6ce71:::smithi19639362-28:head [append 0~49152] snapc 18=[18,17,16,15,12,8,7] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-02-05T04:54:21.095 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:54:06.083659+0000", 2024-02-05T04:54:21.096 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 14.959694627999999, 2024-02-05T04:54:21.096 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0082693929999999999, 2024-02-05T04:54:21.096 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:21.096 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:21.096 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:54:21.096 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:54:21.096 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:21.096 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 696 2024-02-05T04:54:21.096 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.096 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:21.096 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.097 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:21.097 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.083659+0000", 2024-02-05T04:54:21.097 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:21.097 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.097 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.097 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:21.097 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.083659+0000", 2024-02-05T04:54:21.097 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:21.097 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.097 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.097 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:21.097 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.083659+0000", 2024-02-05T04:54:21.097 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.75e-07 2024-02-05T04:54:21.097 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.097 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.098 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:21.098 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.083683+0000", 2024-02-05T04:54:21.098 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.3626999999999999e-05 2024-02-05T04:54:21.098 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.098 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.098 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:21.098 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.083690+0000", 2024-02-05T04:54:21.098 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.4370000000000001e-06 2024-02-05T04:54:21.098 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.098 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.098 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:21.098 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.083695+0000", 2024-02-05T04:54:21.098 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.5029999999999999e-06 2024-02-05T04:54:21.098 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.098 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.099 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:21.099 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.085844+0000", 2024-02-05T04:54:21.099 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0021489119999999998 2024-02-05T04:54:21.099 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.099 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.099 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:54:21.099 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.085950+0000", 2024-02-05T04:54:21.099 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000106159 2024-02-05T04:54:21.099 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.099 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.099 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:21.099 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.086449+0000", 2024-02-05T04:54:21.099 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00049940300000000004 2024-02-05T04:54:21.099 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.099 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.100 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:21.100 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.091749+0000", 2024-02-05T04:54:21.100 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0053000180000000001 2024-02-05T04:54:21.100 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.100 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.100 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:54:21.100 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.091890+0000", 2024-02-05T04:54:21.100 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00014029299999999999 2024-02-05T04:54:21.100 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.100 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.100 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:21.100 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.091928+0000", 2024-02-05T04:54:21.100 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.8365999999999998e-05 2024-02-05T04:54:21.100 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.101 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:21.101 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.101 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.101 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.101 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:697 3.17s0 3:e8f6ce71:::smithi19639362-28:head [setxattr _header (57) in=7b] snapc 18=[18,17,16,15,12,8,7] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-02-05T04:54:21.101 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:54:06.083703+0000", 2024-02-05T04:54:21.101 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 14.959649857, 2024-02-05T04:54:21.101 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0084306329999999999, 2024-02-05T04:54:21.101 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:21.101 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:21.101 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:54:21.101 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:54:21.101 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:21.101 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 697 2024-02-05T04:54:21.101 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.102 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:21.102 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.102 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:21.102 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.083703+0000", 2024-02-05T04:54:21.102 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:21.102 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.102 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.102 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:21.102 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.083703+0000", 2024-02-05T04:54:21.102 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:21.102 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.102 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.102 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:21.102 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.083704+0000", 2024-02-05T04:54:21.102 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.6000000000000004e-07 2024-02-05T04:54:21.102 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.103 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.103 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:21.103 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.083708+0000", 2024-02-05T04:54:21.103 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.6610000000000002e-06 2024-02-05T04:54:21.103 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.103 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.103 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:21.103 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.083714+0000", 2024-02-05T04:54:21.103 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.4309999999999999e-06 2024-02-05T04:54:21.103 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.103 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.103 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:21.103 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.083719+0000", 2024-02-05T04:54:21.103 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.9049999999999996e-06 2024-02-05T04:54:21.103 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.104 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.104 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:21.104 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.087267+0000", 2024-02-05T04:54:21.104 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0035483390000000002 2024-02-05T04:54:21.104 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.104 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.104 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:54:21.104 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.087324+0000", 2024-02-05T04:54:21.104 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.6459999999999998e-05 2024-02-05T04:54:21.104 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.104 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.104 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:21.104 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.087567+0000", 2024-02-05T04:54:21.104 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000243651 2024-02-05T04:54:21.105 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.105 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.105 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:21.105 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.091770+0000", 2024-02-05T04:54:21.105 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0042029190000000003 2024-02-05T04:54:21.105 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.105 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.105 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:54:21.105 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.092049+0000", 2024-02-05T04:54:21.105 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00027828100000000001 2024-02-05T04:54:21.105 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.105 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.105 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:21.105 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.092134+0000", 2024-02-05T04:54:21.105 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.5426000000000005e-05 2024-02-05T04:54:21.106 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.106 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:21.106 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.106 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.106 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.106 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:698 3.17s0 3:e8f6ce71:::smithi19639362-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e74)", 2024-02-05T04:54:21.106 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:54:06.083731+0000", 2024-02-05T04:54:21.106 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 14.959621987, 2024-02-05T04:54:21.106 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0094381120000000002, 2024-02-05T04:54:21.106 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:21.106 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "started", 2024-02-05T04:54:21.106 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:54:21.106 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:54:21.106 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:21.106 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 698 2024-02-05T04:54:21.107 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.107 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:21.107 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.107 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:21.107 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.083731+0000", 2024-02-05T04:54:21.107 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:21.107 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.107 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.107 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:21.107 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.083731+0000", 2024-02-05T04:54:21.107 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:21.107 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.107 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.107 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:21.107 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.083732+0000", 2024-02-05T04:54:21.107 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.6199999999999999e-07 2024-02-05T04:54:21.108 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.108 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.108 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:21.108 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.083733+0000", 2024-02-05T04:54:21.108 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.6869999999999999e-06 2024-02-05T04:54:21.108 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.108 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.108 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:21.108 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.083739+0000", 2024-02-05T04:54:21.108 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.48e-06 2024-02-05T04:54:21.108 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.108 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.108 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:21.109 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.083743+0000", 2024-02-05T04:54:21.109 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.8120000000000001e-06 2024-02-05T04:54:21.109 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.109 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.109 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:21.109 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.087790+0000", 2024-02-05T04:54:21.109 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.004047246 2024-02-05T04:54:21.109 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.109 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.109 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for rw locks", 2024-02-05T04:54:21.109 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.087843+0000", 2024-02-05T04:54:21.109 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.3545999999999999e-05 2024-02-05T04:54:21.109 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.109 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.110 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:21.110 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.092445+0000", 2024-02-05T04:54:21.110 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0046010909999999999 2024-02-05T04:54:21.110 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.110 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.110 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:54:21.110 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.092504+0000", 2024-02-05T04:54:21.110 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.9431000000000001e-05 2024-02-05T04:54:21.110 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.110 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.110 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:21.110 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.093169+0000", 2024-02-05T04:54:21.110 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00066545700000000005 2024-02-05T04:54:21.111 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.111 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:21.111 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.111 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.111 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.111 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.14s3 84/83 ECSubWrite(tid=27, reqid=client.4634.0:850, at_version=84'89, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:21.111 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:54:16.090847+0000", 2024-02-05T04:54:21.111 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 4.9525058980000001, 2024-02-05T04:54:21.111 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0060448100000000003, 2024-02-05T04:54:21.111 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:21.111 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:54:21.111 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:21.111 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.111 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:21.111 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:16.090847+0000", 2024-02-05T04:54:21.111 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:21.112 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.112 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.112 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:21.112 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:16.090847+0000", 2024-02-05T04:54:21.112 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:21.112 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.112 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.112 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:21.112 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:16.090848+0000", 2024-02-05T04:54:21.112 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.3300000000000002e-07 2024-02-05T04:54:21.112 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.112 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.112 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:21.112 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:16.090870+0000", 2024-02-05T04:54:21.112 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.1722e-05 2024-02-05T04:54:21.113 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.113 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.113 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:21.113 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:16.090879+0000", 2024-02-05T04:54:21.113 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.4290000000000007e-06 2024-02-05T04:54:21.113 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.113 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.113 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:21.113 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:16.090886+0000", 2024-02-05T04:54:21.113 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.3490000000000003e-06 2024-02-05T04:54:21.113 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.113 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.113 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:21.113 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:16.090941+0000", 2024-02-05T04:54:21.113 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.4611000000000003e-05 2024-02-05T04:54:21.113 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.114 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.114 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:21.114 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:16.090953+0000", 2024-02-05T04:54:21.114 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.206e-05 2024-02-05T04:54:21.114 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.114 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.114 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:21.114 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:16.096850+0000", 2024-02-05T04:54:21.114 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0058969959999999998 2024-02-05T04:54:21.114 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.114 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.114 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:21.114 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:16.096892+0000", 2024-02-05T04:54:21.114 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.2110000000000002e-05 2024-02-05T04:54:21.114 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.115 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:21.115 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.115 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.115 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.115 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:902 3.17s0 3:e8f6ce71:::smithi19639362-28:19 [read 0~1277952,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-02-05T04:54:21.115 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:54:16.191832+0000", 2024-02-05T04:54:21.115 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 4.8515216429999999, 2024-02-05T04:54:21.115 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0058202159999999996, 2024-02-05T04:54:21.115 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:21.115 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "started", 2024-02-05T04:54:21.115 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:54:21.115 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:54:21.115 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:21.115 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 902 2024-02-05T04:54:21.116 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.116 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:21.116 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.116 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:21.116 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:16.191832+0000", 2024-02-05T04:54:21.116 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:21.116 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.116 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.116 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:21.116 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:16.191832+0000", 2024-02-05T04:54:21.116 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:21.116 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.116 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.116 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:21.116 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:16.191832+0000", 2024-02-05T04:54:21.117 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.1899999999999998e-07 2024-02-05T04:54:21.117 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.117 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.117 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:21.117 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:16.191834+0000", 2024-02-05T04:54:21.117 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.9649999999999998e-06 2024-02-05T04:54:21.117 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.117 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.117 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:21.117 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:16.191838+0000", 2024-02-05T04:54:21.117 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.0010000000000003e-06 2024-02-05T04:54:21.117 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.117 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.117 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:21.118 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:16.191842+0000", 2024-02-05T04:54:21.118 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.8739999999999997e-06 2024-02-05T04:54:21.118 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.118 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.118 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:21.118 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:16.192646+0000", 2024-02-05T04:54:21.118 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00080379300000000002 2024-02-05T04:54:21.118 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.118 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.118 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:54:21.118 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:16.192750+0000", 2024-02-05T04:54:21.118 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000104629 2024-02-05T04:54:21.118 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.118 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.119 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:21.119 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:16.197652+0000", 2024-02-05T04:54:21.119 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0049016349999999997 2024-02-05T04:54:21.119 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.119 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:21.119 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.119 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.119 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.119 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:903 3.17s0 3:e8f6ce71:::smithi19639362-28:19 [read 0~1277952,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-02-05T04:54:21.119 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:54:16.191868+0000", 2024-02-05T04:54:21.119 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 4.8514856159999997, 2024-02-05T04:54:21.119 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0066385460000000004, 2024-02-05T04:54:21.119 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:21.119 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "started", 2024-02-05T04:54:21.119 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:54:21.119 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:54:21.120 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:21.120 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 903 2024-02-05T04:54:21.120 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.120 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:21.120 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.120 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:21.120 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:16.191868+0000", 2024-02-05T04:54:21.120 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:21.120 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.120 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.120 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:21.120 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:16.191868+0000", 2024-02-05T04:54:21.120 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:21.121 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.121 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.121 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:21.121 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:16.191868+0000", 2024-02-05T04:54:21.121 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.2899999999999999e-07 2024-02-05T04:54:21.121 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.121 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.121 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:21.121 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:16.191873+0000", 2024-02-05T04:54:21.121 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.7199999999999997e-06 2024-02-05T04:54:21.121 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.121 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.121 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:21.121 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:16.191881+0000", 2024-02-05T04:54:21.121 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.3129999999999999e-06 2024-02-05T04:54:21.121 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.122 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.122 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:21.122 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:16.191887+0000", 2024-02-05T04:54:21.122 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.0349999999999998e-06 2024-02-05T04:54:21.122 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.122 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.122 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:21.122 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:16.194603+0000", 2024-02-05T04:54:21.122 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0027156799999999998 2024-02-05T04:54:21.122 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.122 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.122 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:54:21.122 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:16.194664+0000", 2024-02-05T04:54:21.122 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.1487000000000003e-05 2024-02-05T04:54:21.122 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.122 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.123 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:21.123 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:16.198506+0000", 2024-02-05T04:54:21.123 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.003841882 2024-02-05T04:54:21.123 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.123 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:21.123 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.123 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.123 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.123 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.bs3 88/85 ECSubWrite(tid=24, reqid=osd.5.0:24, at_version=88'173, trim_to=0'0, roll_forward_to=87'169))", 2024-02-05T04:54:21.123 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:54:20.103446+0000", 2024-02-05T04:54:21.123 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 0.93990707799999995, 2024-02-05T04:54:21.123 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0055163850000000004, 2024-02-05T04:54:21.123 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:21.123 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:54:21.123 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:21.124 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.124 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:21.124 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:20.103446+0000", 2024-02-05T04:54:21.124 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:21.124 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.124 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.124 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:21.124 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:20.103446+0000", 2024-02-05T04:54:21.124 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:21.124 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.124 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.124 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:21.124 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:20.103447+0000", 2024-02-05T04:54:21.124 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.9400000000000005e-07 2024-02-05T04:54:21.124 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.125 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.125 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:21.125 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:20.103465+0000", 2024-02-05T04:54:21.125 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.8094999999999999e-05 2024-02-05T04:54:21.125 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.125 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.125 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:21.125 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:20.103477+0000", 2024-02-05T04:54:21.125 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1809e-05 2024-02-05T04:54:21.125 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.125 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.125 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:21.125 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:20.103484+0000", 2024-02-05T04:54:21.125 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.3819999999999998e-06 2024-02-05T04:54:21.125 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.125 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.126 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:21.126 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:20.103534+0000", 2024-02-05T04:54:21.126 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.9397000000000002e-05 2024-02-05T04:54:21.126 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.126 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.126 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:21.126 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:20.103543+0000", 2024-02-05T04:54:21.126 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.1419999999999999e-06 2024-02-05T04:54:21.126 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.126 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.126 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:21.126 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:20.108922+0000", 2024-02-05T04:54:21.126 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0053792290000000001 2024-02-05T04:54:21.126 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.127 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.127 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:21.127 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:20.108963+0000", 2024-02-05T04:54:21.127 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.0737e-05 2024-02-05T04:54:21.127 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.127 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:21.127 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.127 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.127 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.127 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.bs3 88/85 ECSubWrite(tid=29, reqid=osd.5.0:29, at_version=88'175, trim_to=0'0, roll_forward_to=88'173))", 2024-02-05T04:54:21.127 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:54:20.114033+0000", 2024-02-05T04:54:21.127 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 0.92932002400000002, 2024-02-05T04:54:21.127 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0065511079999999999, 2024-02-05T04:54:21.127 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:21.128 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:54:21.128 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:21.128 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.128 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:21.128 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:20.114033+0000", 2024-02-05T04:54:21.128 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:21.128 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.128 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.128 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:21.128 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:20.114033+0000", 2024-02-05T04:54:21.128 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:21.128 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.128 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.128 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:21.128 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:20.114034+0000", 2024-02-05T04:54:21.129 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.7800000000000001e-07 2024-02-05T04:54:21.129 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.129 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.129 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:21.131 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:20.114052+0000", 2024-02-05T04:54:21.131 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.8034e-05 2024-02-05T04:54:21.131 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.131 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.131 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:21.131 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:20.114063+0000", 2024-02-05T04:54:21.131 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0879999999999999e-05 2024-02-05T04:54:21.131 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.131 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.131 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:21.132 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:20.114073+0000", 2024-02-05T04:54:21.132 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0414000000000001e-05 2024-02-05T04:54:21.132 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.132 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.132 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:21.132 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:20.114121+0000", 2024-02-05T04:54:21.132 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.7852999999999997e-05 2024-02-05T04:54:21.132 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.132 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.132 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:21.132 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:20.114127+0000", 2024-02-05T04:54:21.132 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.5389999999999998e-06 2024-02-05T04:54:21.132 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.132 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.132 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:21.133 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:20.120543+0000", 2024-02-05T04:54:21.133 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0064157570000000002 2024-02-05T04:54:21.133 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.133 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.133 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:21.133 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:20.120584+0000", 2024-02-05T04:54:21.133 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.1853000000000001e-05 2024-02-05T04:54:21.133 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.133 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:21.133 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.133 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.133 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.133 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.bs3 88/85 ECSubWrite(tid=30, reqid=osd.5.0:30, at_version=88'177, trim_to=0'0, roll_forward_to=88'173))", 2024-02-05T04:54:21.133 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:54:20.115413+0000", 2024-02-05T04:54:21.133 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 0.92794053700000001, 2024-02-05T04:54:21.133 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0054501480000000001, 2024-02-05T04:54:21.134 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:21.134 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:54:21.134 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:21.134 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.134 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:21.134 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:20.115413+0000", 2024-02-05T04:54:21.134 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:21.134 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.134 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.134 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:21.134 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:20.115413+0000", 2024-02-05T04:54:21.134 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:21.134 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.134 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.134 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:21.135 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:20.115413+0000", 2024-02-05T04:54:21.135 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.2399999999999997e-07 2024-02-05T04:54:21.135 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.135 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.135 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:21.135 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:20.115443+0000", 2024-02-05T04:54:21.135 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.9848999999999998e-05 2024-02-05T04:54:21.135 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.135 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.135 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:21.135 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:20.115455+0000", 2024-02-05T04:54:21.135 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1178999999999999e-05 2024-02-05T04:54:21.135 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.135 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.135 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:21.135 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:20.115464+0000", 2024-02-05T04:54:21.135 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.0659999999999993e-06 2024-02-05T04:54:21.136 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.136 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.136 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:21.136 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:20.115535+0000", 2024-02-05T04:54:21.136 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.1243999999999998e-05 2024-02-05T04:54:21.136 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.136 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.136 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:21.136 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:20.115546+0000", 2024-02-05T04:54:21.136 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1104000000000001e-05 2024-02-05T04:54:21.136 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.136 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.136 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:21.136 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:20.120825+0000", 2024-02-05T04:54:21.137 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0052788109999999996 2024-02-05T04:54:21.137 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.137 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.137 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:21.137 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:20.120863+0000", 2024-02-05T04:54:21.137 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.8170999999999998e-05 2024-02-05T04:54:21.137 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.137 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:21.137 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.137 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.137 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:21.137 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-02-05T04:54:21.140 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-02-05T04:54:21.141 INFO:teuthology.orchestra.run.smithi062.stderr:noscrub is set 2024-02-05T04:54:21.141 INFO:tasks.ceph.osd.5.smithi091.stderr:2024-02-05T04:54:21.137+0000 7f63983b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34277) UID: 0 2024-02-05T04:54:21.239 INFO:tasks.ceph.osd.9.smithi091.stderr:2024-02-05T04:54:21.238+0000 7efc18827640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 34279) UID: 0 2024-02-05T04:54:21.286 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-02-05T04:54:21.286 INFO:teuthology.orchestra.run.smithi062.stdout: "ops": [], 2024-02-05T04:54:21.286 INFO:teuthology.orchestra.run.smithi062.stdout: "num_ops": 0 2024-02-05T04:54:21.286 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-02-05T04:54:21.295 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-02-05T04:54:21.341 INFO:tasks.ceph.osd.1.smithi091.stderr:2024-02-05T04:54:21.339+0000 7ff9dc283640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34275) UID: 0 2024-02-05T04:54:21.402 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-02-05T04:54:21.402 INFO:teuthology.orchestra.run.smithi062.stdout: "ops": [], 2024-02-05T04:54:21.402 INFO:teuthology.orchestra.run.smithi062.stdout: "complaint_time": 30, 2024-02-05T04:54:21.402 INFO:teuthology.orchestra.run.smithi062.stdout: "num_blocked_ops": 0 2024-02-05T04:54:21.402 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-02-05T04:54:21.411 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-02-05T04:54:21.436 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T04:54:21.442 INFO:tasks.ceph.osd.11.smithi196.stderr:2024-02-05T04:54:21.440+0000 7f85a8d86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 34609) UID: 0 2024-02-05T04:54:21.521 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-02-05T04:54:21.521 INFO:teuthology.orchestra.run.smithi062.stdout: "size": 20, 2024-02-05T04:54:21.521 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 600, 2024-02-05T04:54:21.521 INFO:teuthology.orchestra.run.smithi062.stdout: "ops": [ 2024-02-05T04:54:21.521 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.521 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:218 3.3s0 3:ce48de55:::smithi19639362-16:head [append 0~638976] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:54:21.521 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:29.132294+0000", 2024-02-05T04:54:21.521 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 52.386433027000002, 2024-02-05T04:54:21.521 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0081869909999999994, 2024-02-05T04:54:21.522 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:21.522 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:21.522 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:54:21.522 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:54:21.522 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:21.522 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 218 2024-02-05T04:54:21.522 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.522 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:21.522 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.522 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:21.522 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.132294+0000", 2024-02-05T04:54:21.522 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:21.522 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.522 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.522 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:21.523 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.132294+0000", 2024-02-05T04:54:21.523 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:21.523 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.523 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.523 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:21.523 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.132296+0000", 2024-02-05T04:54:21.523 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.886e-06 2024-02-05T04:54:21.523 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.523 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.523 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:21.523 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.133151+0000", 2024-02-05T04:54:21.523 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00085535699999999995 2024-02-05T04:54:21.523 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.523 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.523 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:21.524 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.133165+0000", 2024-02-05T04:54:21.524 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.396e-05 2024-02-05T04:54:21.524 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.524 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.524 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:21.524 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.133182+0000", 2024-02-05T04:54:21.524 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.6787e-05 2024-02-05T04:54:21.524 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.524 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.524 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:21.524 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.133229+0000", 2024-02-05T04:54:21.524 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.7509999999999997e-05 2024-02-05T04:54:21.524 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.525 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.525 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:54:21.525 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.133324+0000", 2024-02-05T04:54:21.525 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.4950000000000004e-05 2024-02-05T04:54:21.525 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.525 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.525 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:21.525 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.134392+0000", 2024-02-05T04:54:21.525 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0010674219999999999 2024-02-05T04:54:21.525 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.525 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.525 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:21.525 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.138240+0000", 2024-02-05T04:54:21.525 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0038487930000000001 2024-02-05T04:54:21.526 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.526 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.526 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:54:21.526 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.140400+0000", 2024-02-05T04:54:21.526 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0021592690000000001 2024-02-05T04:54:21.526 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.526 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.526 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:21.526 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.140481+0000", 2024-02-05T04:54:21.526 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.1057000000000007e-05 2024-02-05T04:54:21.526 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.526 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:21.526 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.526 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.526 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.527 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:219 3.3s0 3:ce48de55:::smithi19639362-16:head [append 0~376832] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:54:21.527 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:29.134822+0000", 2024-02-05T04:54:21.527 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 52.383904350999998, 2024-02-05T04:54:21.527 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0088602019999999993, 2024-02-05T04:54:21.527 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:21.527 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:21.527 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:54:21.527 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:54:21.527 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:21.527 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 219 2024-02-05T04:54:21.527 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.527 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:21.527 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.528 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:21.528 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.134822+0000", 2024-02-05T04:54:21.528 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:21.528 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.528 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.528 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:21.528 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.134822+0000", 2024-02-05T04:54:21.528 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:21.528 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.528 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.529 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:21.529 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.134825+0000", 2024-02-05T04:54:21.529 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.1229999999999998e-06 2024-02-05T04:54:21.529 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.529 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.529 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:21.530 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.135417+0000", 2024-02-05T04:54:21.530 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00059254900000000005 2024-02-05T04:54:21.530 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.530 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.530 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:21.530 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.135430+0000", 2024-02-05T04:54:21.530 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.2663000000000001e-05 2024-02-05T04:54:21.530 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.530 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.530 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:21.530 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.135436+0000", 2024-02-05T04:54:21.530 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.6959999999999999e-06 2024-02-05T04:54:21.530 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.530 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.530 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:21.531 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.135834+0000", 2024-02-05T04:54:21.531 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000397382 2024-02-05T04:54:21.531 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.531 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.531 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:54:21.531 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.135967+0000", 2024-02-05T04:54:21.531 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00013333 2024-02-05T04:54:21.531 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.531 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.531 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:21.531 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.136753+0000", 2024-02-05T04:54:21.531 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00078585700000000005 2024-02-05T04:54:21.531 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.531 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.532 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:21.532 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.138286+0000", 2024-02-05T04:54:21.532 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0015333390000000001 2024-02-05T04:54:21.532 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.532 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.532 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:54:21.532 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.143611+0000", 2024-02-05T04:54:21.532 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0053249639999999997 2024-02-05T04:54:21.532 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.532 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.532 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:21.532 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.143683+0000", 2024-02-05T04:54:21.532 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.1298999999999996e-05 2024-02-05T04:54:21.532 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.532 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:21.533 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.533 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.533 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.533 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:220 3.3s0 3:ce48de55:::smithi19639362-16:head [setxattr _header (56) in=7b] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:54:21.533 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:29.135450+0000", 2024-02-05T04:54:21.533 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 52.383276428999999, 2024-02-05T04:54:21.533 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0084937420000000003, 2024-02-05T04:54:21.533 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:21.533 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:21.533 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:54:21.533 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:54:21.533 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:21.533 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 220 2024-02-05T04:54:21.533 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.534 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:21.534 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.534 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:21.534 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.135450+0000", 2024-02-05T04:54:21.534 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:21.534 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.534 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.534 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:21.534 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.135450+0000", 2024-02-05T04:54:21.534 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:21.534 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.534 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.534 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:21.534 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.135451+0000", 2024-02-05T04:54:21.534 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.4799999999999997e-07 2024-02-05T04:54:21.534 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.535 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.535 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:21.535 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.135454+0000", 2024-02-05T04:54:21.535 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.3539999999999998e-06 2024-02-05T04:54:21.535 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.535 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.535 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:21.535 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.135458+0000", 2024-02-05T04:54:21.535 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.1509999999999997e-06 2024-02-05T04:54:21.535 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.535 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.535 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:21.535 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.135461+0000", 2024-02-05T04:54:21.536 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.1650000000000002e-06 2024-02-05T04:54:21.536 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.536 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.536 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:21.536 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.137469+0000", 2024-02-05T04:54:21.536 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0020076410000000001 2024-02-05T04:54:21.536 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.536 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.536 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:54:21.536 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.137586+0000", 2024-02-05T04:54:21.536 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000117599 2024-02-05T04:54:21.536 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.536 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.536 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:21.537 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.138018+0000", 2024-02-05T04:54:21.537 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00043188200000000002 2024-02-05T04:54:21.537 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.537 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.537 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:21.537 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.140234+0000", 2024-02-05T04:54:21.537 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0022155159999999998 2024-02-05T04:54:21.537 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.537 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.537 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:54:21.537 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.143857+0000", 2024-02-05T04:54:21.537 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0036236749999999998 2024-02-05T04:54:21.537 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.537 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.537 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:21.538 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.143944+0000", 2024-02-05T04:54:21.538 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.6910999999999995e-05 2024-02-05T04:54:21.538 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.538 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:21.538 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.538 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.538 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.538 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:221 3.3s0 3:ce48de55:::smithi19639362-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e37)", 2024-02-05T04:54:21.538 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:29.135468+0000", 2024-02-05T04:54:21.538 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 52.383259291000002, 2024-02-05T04:54:21.538 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.010019444000000001, 2024-02-05T04:54:21.538 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:21.538 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "started", 2024-02-05T04:54:21.538 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:54:21.539 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:54:21.539 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:21.539 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 221 2024-02-05T04:54:21.539 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.539 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:21.539 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.539 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:21.539 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.135468+0000", 2024-02-05T04:54:21.539 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:21.539 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.539 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.539 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:21.539 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.135468+0000", 2024-02-05T04:54:21.540 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:21.540 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.540 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.540 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:21.540 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.135468+0000", 2024-02-05T04:54:21.540 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.4299999999999999e-07 2024-02-05T04:54:21.540 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.540 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.540 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:21.540 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.135469+0000", 2024-02-05T04:54:21.540 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.457e-06 2024-02-05T04:54:21.540 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.540 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.541 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:21.541 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.135475+0000", 2024-02-05T04:54:21.541 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.9939999999999997e-06 2024-02-05T04:54:21.541 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.541 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.541 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:21.543 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.135480+0000", 2024-02-05T04:54:21.543 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.0989999999999998e-06 2024-02-05T04:54:21.543 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.544 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.544 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:21.544 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.138320+0000", 2024-02-05T04:54:21.544 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0028394739999999998 2024-02-05T04:54:21.544 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.544 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.544 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for rw locks", 2024-02-05T04:54:21.544 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.138401+0000", 2024-02-05T04:54:21.544 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.0966999999999999e-05 2024-02-05T04:54:21.544 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.544 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.544 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:21.544 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.144381+0000", 2024-02-05T04:54:21.544 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0059798389999999998 2024-02-05T04:54:21.545 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.545 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.545 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:54:21.545 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.144442+0000", 2024-02-05T04:54:21.545 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.1311999999999998e-05 2024-02-05T04:54:21.545 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.545 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.545 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:21.545 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:29.145487+0000", 2024-02-05T04:54:21.545 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0010450590000000001 2024-02-05T04:54:21.545 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.545 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:21.545 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.545 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.545 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.546 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:390 3.8s0 3:170878b2:::smithi19639362-30:head [copy-from ver 20] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-02-05T04:54:21.546 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:34.171935+0000", 2024-02-05T04:54:21.546 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 47.346791330999999, 2024-02-05T04:54:21.546 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.014076916, 2024-02-05T04:54:21.546 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:21.546 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:21.546 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:54:21.546 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:54:21.546 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:21.546 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 390 2024-02-05T04:54:21.546 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.546 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:21.546 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.546 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:21.547 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.171935+0000", 2024-02-05T04:54:21.547 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:21.547 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.547 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.547 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:21.547 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.171935+0000", 2024-02-05T04:54:21.547 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:21.547 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.547 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.547 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:21.547 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.171937+0000", 2024-02-05T04:54:21.547 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.8959999999999999e-06 2024-02-05T04:54:21.547 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.547 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.547 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:21.548 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.171944+0000", 2024-02-05T04:54:21.548 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.5729999999999996e-06 2024-02-05T04:54:21.548 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.548 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.548 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:21.548 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.171956+0000", 2024-02-05T04:54:21.548 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.2499e-05 2024-02-05T04:54:21.548 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.548 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.548 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:21.548 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.171964+0000", 2024-02-05T04:54:21.548 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.6450000000000005e-06 2024-02-05T04:54:21.548 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.548 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.548 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:21.549 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.172025+0000", 2024-02-05T04:54:21.549 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.0713e-05 2024-02-05T04:54:21.549 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.549 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.549 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:54:21.549 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.172112+0000", 2024-02-05T04:54:21.549 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.7421000000000002e-05 2024-02-05T04:54:21.549 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.549 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.549 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:21.549 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.180774+0000", 2024-02-05T04:54:21.549 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.008661986 2024-02-05T04:54:21.549 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.549 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.549 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:21.550 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.182980+0000", 2024-02-05T04:54:21.550 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0022060389999999999 2024-02-05T04:54:21.550 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.550 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.550 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:54:21.550 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.185921+0000", 2024-02-05T04:54:21.550 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0029412179999999998 2024-02-05T04:54:21.550 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.550 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.550 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:21.550 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.186012+0000", 2024-02-05T04:54:21.550 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.0926000000000003e-05 2024-02-05T04:54:21.551 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.551 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:21.551 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.551 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.551 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.551 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:391 3.8s0 3:170878b2:::smithi19639362-30:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-02-05T04:54:21.551 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:34.172014+0000", 2024-02-05T04:54:21.551 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 47.346712848000003, 2024-02-05T04:54:21.551 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.010857311999999999, 2024-02-05T04:54:21.551 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:21.551 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "started", 2024-02-05T04:54:21.551 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:54:21.551 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:54:21.551 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:21.552 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 391 2024-02-05T04:54:21.552 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.552 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:21.552 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.552 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:21.552 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.172014+0000", 2024-02-05T04:54:21.552 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:21.552 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.552 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.552 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:21.552 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.172014+0000", 2024-02-05T04:54:21.552 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:21.552 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.552 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.553 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:21.553 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.172015+0000", 2024-02-05T04:54:21.553 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.5199999999999996e-07 2024-02-05T04:54:21.553 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.553 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.553 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:21.553 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.172018+0000", 2024-02-05T04:54:21.553 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.5319999999999998e-06 2024-02-05T04:54:21.553 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.553 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.553 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:21.553 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.172028+0000", 2024-02-05T04:54:21.553 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.5990000000000008e-06 2024-02-05T04:54:21.553 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.553 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.554 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:21.554 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.172034+0000", 2024-02-05T04:54:21.554 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.742e-06 2024-02-05T04:54:21.554 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.554 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.554 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:21.554 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.172290+0000", 2024-02-05T04:54:21.554 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00025550300000000002 2024-02-05T04:54:21.554 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.554 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.554 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for blocked object", 2024-02-05T04:54:21.554 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.172342+0000", 2024-02-05T04:54:21.554 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.1657999999999999e-05 2024-02-05T04:54:21.554 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.554 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.555 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:21.555 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.182668+0000", 2024-02-05T04:54:21.555 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.010326223000000001 2024-02-05T04:54:21.555 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.555 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.555 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:54:21.555 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.182773+0000", 2024-02-05T04:54:21.555 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00010530399999999999 2024-02-05T04:54:21.555 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.555 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.555 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:21.555 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:34.182871+0000", 2024-02-05T04:54:21.555 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.8098999999999997e-05 2024-02-05T04:54:21.555 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.555 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:21.556 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.556 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.556 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.556 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.4s5 43/31 ECSubWrite(tid=46, reqid=osd.1.0:46, at_version=43'29, trim_to=0'0, roll_forward_to=42'27))", 2024-02-05T04:54:21.556 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:35.182637+0000", 2024-02-05T04:54:21.556 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 46.336090040999999, 2024-02-05T04:54:21.556 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0075236640000000002, 2024-02-05T04:54:21.556 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:21.556 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:54:21.556 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:21.556 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.556 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:21.556 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.182637+0000", 2024-02-05T04:54:21.556 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:21.556 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.557 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.557 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:21.557 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.182637+0000", 2024-02-05T04:54:21.557 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:21.557 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.557 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.557 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:21.557 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.182637+0000", 2024-02-05T04:54:21.557 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.3000000000000002e-07 2024-02-05T04:54:21.557 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.557 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.557 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:21.557 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.182698+0000", 2024-02-05T04:54:21.557 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.1062000000000006e-05 2024-02-05T04:54:21.557 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.558 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.558 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:21.558 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.182708+0000", 2024-02-05T04:54:21.558 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0186999999999999e-05 2024-02-05T04:54:21.558 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.558 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.558 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:21.558 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.182715+0000", 2024-02-05T04:54:21.558 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.0040000000000002e-06 2024-02-05T04:54:21.558 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.558 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.558 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:21.558 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.182796+0000", 2024-02-05T04:54:21.558 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.0364000000000005e-05 2024-02-05T04:54:21.558 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.559 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.559 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:21.559 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.182805+0000", 2024-02-05T04:54:21.559 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.5990000000000008e-06 2024-02-05T04:54:21.559 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.559 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.559 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:21.559 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.190124+0000", 2024-02-05T04:54:21.559 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0073191009999999997 2024-02-05T04:54:21.559 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.559 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.559 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:21.559 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:35.190160+0000", 2024-02-05T04:54:21.560 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.6016999999999998e-05 2024-02-05T04:54:21.560 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.560 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:21.560 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.560 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.560 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.560 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:591 3.8s0 3:170878b2:::smithi19639362-30:head [append 0~0] snapc 13=[13,12,11,f,e,d,b,8,7] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-02-05T04:54:21.560 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:54.007323+0000", 2024-02-05T04:54:21.560 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 27.511404204000002, 2024-02-05T04:54:21.560 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.99264834700000004, 2024-02-05T04:54:21.560 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:21.560 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:21.560 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:54:21.560 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:54:21.561 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:21.561 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 591 2024-02-05T04:54:21.561 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.561 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:21.561 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.561 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:21.561 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.007323+0000", 2024-02-05T04:54:21.561 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:21.561 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.561 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.561 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:21.561 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.007323+0000", 2024-02-05T04:54:21.561 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:21.561 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.561 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.562 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:21.562 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.007324+0000", 2024-02-05T04:54:21.562 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.8339999999999999e-06 2024-02-05T04:54:21.562 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.562 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.562 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:21.562 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.007333+0000", 2024-02-05T04:54:21.562 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.5790000000000004e-06 2024-02-05T04:54:21.562 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.562 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.562 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:21.562 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.007353+0000", 2024-02-05T04:54:21.562 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.9568999999999998e-05 2024-02-05T04:54:21.562 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.562 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.563 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:21.563 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.007363+0000", 2024-02-05T04:54:21.563 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0319000000000001e-05 2024-02-05T04:54:21.563 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.563 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.563 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:21.563 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.007446+0000", 2024-02-05T04:54:21.563 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.2924999999999996e-05 2024-02-05T04:54:21.563 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.563 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.563 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for peered", 2024-02-05T04:54:21.563 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.007450+0000", 2024-02-05T04:54:21.563 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.5909999999999998e-06 2024-02-05T04:54:21.563 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.563 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.564 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:21.564 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.986231+0000", 2024-02-05T04:54:21.564 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.97878106300000001 2024-02-05T04:54:21.564 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.564 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.564 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:54:21.564 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.986401+0000", 2024-02-05T04:54:21.564 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00016971600000000001 2024-02-05T04:54:21.564 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.564 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.564 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:21.564 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.987097+0000", 2024-02-05T04:54:21.564 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00069556800000000003 2024-02-05T04:54:21.564 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.565 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.565 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:21.565 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.990602+0000", 2024-02-05T04:54:21.565 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0035056549999999999 2024-02-05T04:54:21.565 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.565 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.565 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:54:21.565 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.999810+0000", 2024-02-05T04:54:21.565 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0092078690000000005 2024-02-05T04:54:21.565 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.565 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.565 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:21.565 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.999971+0000", 2024-02-05T04:54:21.565 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00016065900000000001 2024-02-05T04:54:21.565 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.566 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:21.566 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.566 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.566 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.566 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:592 3.8s0 3:170878b2:::smithi19639362-30:head [rollback 8] snapc 13=[13,12,11,f,e,d,b,8,7] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-02-05T04:54:21.566 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:54.007387+0000", 2024-02-05T04:54:21.566 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 27.511339795000001, 2024-02-05T04:54:21.566 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.99478262799999995, 2024-02-05T04:54:21.566 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:21.566 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:21.566 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:54:21.566 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:54:21.566 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:21.566 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 592 2024-02-05T04:54:21.566 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.567 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:21.567 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.567 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:21.567 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.007387+0000", 2024-02-05T04:54:21.567 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:21.567 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.567 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.567 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:21.567 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.007387+0000", 2024-02-05T04:54:21.567 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:21.567 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.567 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.567 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:21.567 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.007387+0000", 2024-02-05T04:54:21.567 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.3799999999999998e-07 2024-02-05T04:54:21.568 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.568 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.568 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:21.568 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.007390+0000", 2024-02-05T04:54:21.568 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.4030000000000001e-06 2024-02-05T04:54:21.568 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.568 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.568 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:21.568 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.007397+0000", 2024-02-05T04:54:21.568 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.272e-06 2024-02-05T04:54:21.568 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.568 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.568 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:21.568 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.007402+0000", 2024-02-05T04:54:21.569 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.7400000000000004e-06 2024-02-05T04:54:21.569 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.569 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.569 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:21.569 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.007491+0000", 2024-02-05T04:54:21.569 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.9462999999999994e-05 2024-02-05T04:54:21.569 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.569 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.569 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for peered", 2024-02-05T04:54:21.569 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.007493+0000", 2024-02-05T04:54:21.569 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.7540000000000001e-06 2024-02-05T04:54:21.569 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.569 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.569 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:21.569 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.988725+0000", 2024-02-05T04:54:21.570 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.98123155299999998 2024-02-05T04:54:21.570 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.570 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.570 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:54:21.570 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.988828+0000", 2024-02-05T04:54:21.570 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000103463 2024-02-05T04:54:21.570 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.570 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.570 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:21.570 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.989372+0000", 2024-02-05T04:54:21.570 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00054408600000000005 2024-02-05T04:54:21.570 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.570 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.570 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:21.570 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.994357+0000", 2024-02-05T04:54:21.571 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0049851440000000004 2024-02-05T04:54:21.571 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.571 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.571 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:54:21.571 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:55.002096+0000", 2024-02-05T04:54:21.571 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.007738389 2024-02-05T04:54:21.571 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.571 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.571 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:21.571 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:55.002170+0000", 2024-02-05T04:54:21.571 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.3923000000000004e-05 2024-02-05T04:54:21.571 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.571 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:21.571 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.571 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.572 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.572 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:593 3.8s0 3:170878b2:::smithi19639362-30:head [append 0~0] snapc 13=[13,12,11,f,e,d,b,8,7] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-02-05T04:54:21.572 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:53:54.007407+0000", 2024-02-05T04:54:21.572 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 27.511319308000001, 2024-02-05T04:54:21.572 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.99488064899999995, 2024-02-05T04:54:21.572 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:21.572 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:21.572 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:54:21.572 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:54:21.572 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:21.572 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 593 2024-02-05T04:54:21.572 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.572 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:21.572 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.572 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:21.573 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.007407+0000", 2024-02-05T04:54:21.573 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:21.573 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.573 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.573 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:21.573 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.007407+0000", 2024-02-05T04:54:21.573 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:21.573 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.573 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.573 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:21.573 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.007408+0000", 2024-02-05T04:54:21.573 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.0400000000000002e-07 2024-02-05T04:54:21.573 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.573 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.573 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:21.574 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.007413+0000", 2024-02-05T04:54:21.574 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.9790000000000001e-06 2024-02-05T04:54:21.574 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.574 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.574 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:21.574 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.007418+0000", 2024-02-05T04:54:21.574 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.8180000000000003e-06 2024-02-05T04:54:21.574 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.574 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.574 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:21.574 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.007422+0000", 2024-02-05T04:54:21.574 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.1930000000000003e-06 2024-02-05T04:54:21.574 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.574 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.574 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:21.575 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.007524+0000", 2024-02-05T04:54:21.575 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000101808 2024-02-05T04:54:21.575 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.575 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.575 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for peered", 2024-02-05T04:54:21.575 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.007525+0000", 2024-02-05T04:54:21.575 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.2330000000000001e-06 2024-02-05T04:54:21.575 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.575 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.575 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:21.575 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.989757+0000", 2024-02-05T04:54:21.575 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.98223166900000003 2024-02-05T04:54:21.575 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.575 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.576 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:54:21.576 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.989846+0000", 2024-02-05T04:54:21.576 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.9580999999999995e-05 2024-02-05T04:54:21.576 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.576 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.576 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:21.576 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.990320+0000", 2024-02-05T04:54:21.576 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00047333 2024-02-05T04:54:21.576 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.576 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.576 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:21.576 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:54.994359+0000", 2024-02-05T04:54:21.576 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0040392400000000004 2024-02-05T04:54:21.576 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.576 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.577 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:54:21.577 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:55.002244+0000", 2024-02-05T04:54:21.577 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0078852279999999993 2024-02-05T04:54:21.577 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.577 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.577 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:21.577 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:53:55.002288+0000", 2024-02-05T04:54:21.577 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.4165999999999997e-05 2024-02-05T04:54:21.577 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.577 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:21.577 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.577 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.577 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.577 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:629 3.3s0 3:c46da155:::smithi19639362-15:head [append 0~770048] snapc 16=[16,15,12,11,f,b,8,7] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-02-05T04:54:21.577 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:54:00.957118+0000", 2024-02-05T04:54:21.578 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 20.561609206, 2024-02-05T04:54:21.578 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.016343434, 2024-02-05T04:54:21.578 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:21.578 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:21.578 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:54:21.578 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:54:21.578 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:21.578 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 629 2024-02-05T04:54:21.578 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.578 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:21.578 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.578 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:21.578 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.957118+0000", 2024-02-05T04:54:21.578 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:21.579 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.579 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.579 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:21.579 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.957118+0000", 2024-02-05T04:54:21.579 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:21.579 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.579 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.579 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:21.579 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.957121+0000", 2024-02-05T04:54:21.579 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.0910000000000001e-06 2024-02-05T04:54:21.579 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.579 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.579 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:21.579 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.958839+0000", 2024-02-05T04:54:21.579 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.001718682 2024-02-05T04:54:21.580 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.580 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.580 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:21.580 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.958876+0000", 2024-02-05T04:54:21.580 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.6293e-05 2024-02-05T04:54:21.580 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.580 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.580 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:21.580 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.958889+0000", 2024-02-05T04:54:21.580 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.3664e-05 2024-02-05T04:54:21.580 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.580 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.580 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:21.580 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.958976+0000", 2024-02-05T04:54:21.580 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.6207999999999999e-05 2024-02-05T04:54:21.581 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.581 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.581 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "op must wait for map", 2024-02-05T04:54:21.581 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.958983+0000", 2024-02-05T04:54:21.581 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.7260000000000002e-06 2024-02-05T04:54:21.581 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.581 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.581 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:21.581 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.964806+0000", 2024-02-05T04:54:21.581 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0058228869999999997 2024-02-05T04:54:21.581 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.581 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.581 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:54:21.581 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.964928+0000", 2024-02-05T04:54:21.581 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000121997 2024-02-05T04:54:21.582 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.582 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.582 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:21.582 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.965928+0000", 2024-02-05T04:54:21.582 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0010002850000000001 2024-02-05T04:54:21.582 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.582 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.582 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:21.582 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.973092+0000", 2024-02-05T04:54:21.582 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0071639190000000004 2024-02-05T04:54:21.582 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.582 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.582 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:54:21.582 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.973366+0000", 2024-02-05T04:54:21.582 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000273467 2024-02-05T04:54:21.583 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.583 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.583 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:21.583 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.973461+0000", 2024-02-05T04:54:21.583 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.5215000000000002e-05 2024-02-05T04:54:21.583 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.583 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:21.583 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.583 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.583 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.583 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:630 3.3s0 3:c46da155:::smithi19639362-15:head [append 0~704512] snapc 16=[16,15,12,11,f,b,8,7] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-02-05T04:54:21.583 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:54:00.961143+0000", 2024-02-05T04:54:21.583 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 20.557583766, 2024-02-05T04:54:21.583 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.014004825, 2024-02-05T04:54:21.583 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:21.584 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:21.584 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:54:21.584 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:54:21.584 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:21.584 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 630 2024-02-05T04:54:21.584 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.584 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:21.584 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.584 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:21.584 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.961143+0000", 2024-02-05T04:54:21.584 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:21.584 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.584 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.584 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:21.585 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.961143+0000", 2024-02-05T04:54:21.585 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:21.585 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.585 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.585 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:21.585 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.961145+0000", 2024-02-05T04:54:21.585 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.716e-06 2024-02-05T04:54:21.585 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.585 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.585 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:21.585 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.962094+0000", 2024-02-05T04:54:21.585 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00094879700000000003 2024-02-05T04:54:21.586 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.586 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.586 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:21.586 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.962105+0000", 2024-02-05T04:54:21.586 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1209999999999999e-05 2024-02-05T04:54:21.586 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.586 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.586 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:21.586 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.962111+0000", 2024-02-05T04:54:21.586 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.9510000000000003e-06 2024-02-05T04:54:21.586 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.586 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.586 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:21.586 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.962167+0000", 2024-02-05T04:54:21.587 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.6020999999999998e-05 2024-02-05T04:54:21.587 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.587 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.587 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting_for_map not empty", 2024-02-05T04:54:21.587 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.962172+0000", 2024-02-05T04:54:21.587 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.6860000000000003e-06 2024-02-05T04:54:21.587 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.587 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.587 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:21.587 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.966743+0000", 2024-02-05T04:54:21.587 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0045707639999999997 2024-02-05T04:54:21.587 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.588 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.588 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:54:21.588 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.966868+0000", 2024-02-05T04:54:21.588 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00012479099999999999 2024-02-05T04:54:21.588 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.588 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.588 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:21.588 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.967757+0000", 2024-02-05T04:54:21.588 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00088885699999999995 2024-02-05T04:54:21.588 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.588 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.588 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:21.588 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.974750+0000", 2024-02-05T04:54:21.588 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0069934250000000002 2024-02-05T04:54:21.588 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.589 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.589 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:54:21.589 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.975055+0000", 2024-02-05T04:54:21.589 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00030481699999999998 2024-02-05T04:54:21.589 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.589 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.589 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:21.589 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.975148+0000", 2024-02-05T04:54:21.589 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.2789999999999998e-05 2024-02-05T04:54:21.589 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.589 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:21.590 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.590 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.590 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.590 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:631 3.3s0 3:c46da155:::smithi19639362-15:head [append 0~458752] snapc 16=[16,15,12,11,f,b,8,7] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-02-05T04:54:21.590 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:54:00.964145+0000", 2024-02-05T04:54:21.590 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 20.554582281999998, 2024-02-05T04:54:21.590 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.012240795, 2024-02-05T04:54:21.590 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:21.590 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:21.590 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:54:21.590 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:54:21.590 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:21.590 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 631 2024-02-05T04:54:21.591 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.591 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:21.591 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.591 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:21.591 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.964145+0000", 2024-02-05T04:54:21.591 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:21.591 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.591 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.591 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:21.591 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.964145+0000", 2024-02-05T04:54:21.591 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:21.591 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.591 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.591 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:21.592 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.964146+0000", 2024-02-05T04:54:21.592 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.5689999999999999e-06 2024-02-05T04:54:21.592 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.592 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.592 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:21.592 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.965007+0000", 2024-02-05T04:54:21.592 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00086071499999999996 2024-02-05T04:54:21.592 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.592 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.592 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:21.592 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.965017+0000", 2024-02-05T04:54:21.592 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0643e-05 2024-02-05T04:54:21.592 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.592 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.592 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:21.593 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.965024+0000", 2024-02-05T04:54:21.593 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.4659999999999996e-06 2024-02-05T04:54:21.593 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.593 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.593 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:21.593 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.970390+0000", 2024-02-05T04:54:21.593 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00536587 2024-02-05T04:54:21.593 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.593 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.593 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:54:21.593 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.970537+0000", 2024-02-05T04:54:21.593 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00014697899999999999 2024-02-05T04:54:21.593 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.593 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.593 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:21.594 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.971373+0000", 2024-02-05T04:54:21.594 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00083609699999999995 2024-02-05T04:54:21.594 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.594 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.594 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:21.594 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.975431+0000", 2024-02-05T04:54:21.594 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0040576910000000004 2024-02-05T04:54:21.594 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.594 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.594 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:54:21.594 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.976344+0000", 2024-02-05T04:54:21.594 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00091364799999999998 2024-02-05T04:54:21.594 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.594 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.595 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:21.595 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.976385+0000", 2024-02-05T04:54:21.595 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.1117e-05 2024-02-05T04:54:21.595 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.595 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:21.595 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.595 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.595 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.595 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:632 3.3s0 3:c46da155:::smithi19639362-15:head [setxattr _header (57) in=7b] snapc 16=[16,15,12,11,f,b,8,7] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-02-05T04:54:21.595 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:54:00.965035+0000", 2024-02-05T04:54:21.595 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 20.553691549, 2024-02-05T04:54:21.595 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.011543045, 2024-02-05T04:54:21.595 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:21.595 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:21.595 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:54:21.596 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:54:21.596 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:21.596 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 632 2024-02-05T04:54:21.596 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.596 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:21.596 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.596 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:21.596 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.965035+0000", 2024-02-05T04:54:21.596 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:21.596 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.596 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.596 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:21.596 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.965035+0000", 2024-02-05T04:54:21.596 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:21.596 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.597 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.597 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:21.597 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.965036+0000", 2024-02-05T04:54:21.597 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.99e-07 2024-02-05T04:54:21.597 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.597 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.597 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:21.597 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.965038+0000", 2024-02-05T04:54:21.597 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.604e-06 2024-02-05T04:54:21.597 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.597 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.597 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:21.597 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.965043+0000", 2024-02-05T04:54:21.597 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.0999999999999997e-06 2024-02-05T04:54:21.597 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.598 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.598 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:21.598 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.965046+0000", 2024-02-05T04:54:21.598 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.1410000000000001e-06 2024-02-05T04:54:21.598 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.598 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.598 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:21.598 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.972375+0000", 2024-02-05T04:54:21.598 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0073292299999999999 2024-02-05T04:54:21.598 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.598 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.598 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:54:21.598 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.972473+0000", 2024-02-05T04:54:21.598 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.8414999999999999e-05 2024-02-05T04:54:21.598 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.599 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.599 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:21.599 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.972832+0000", 2024-02-05T04:54:21.599 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00035884900000000001 2024-02-05T04:54:21.599 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.599 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.599 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:21.599 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.975432+0000", 2024-02-05T04:54:21.599 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.002600265 2024-02-05T04:54:21.599 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.599 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.599 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:54:21.599 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.976514+0000", 2024-02-05T04:54:21.599 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.001081551 2024-02-05T04:54:21.600 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.600 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.600 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:21.600 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.976578+0000", 2024-02-05T04:54:21.600 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.4290999999999998e-05 2024-02-05T04:54:21.600 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.600 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:21.600 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.600 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.600 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.600 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:633 3.3s0 3:c46da155:::smithi19639362-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e69)", 2024-02-05T04:54:21.600 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:54:00.965050+0000", 2024-02-05T04:54:21.600 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 20.553676830000001, 2024-02-05T04:54:21.600 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.013116340000000001, 2024-02-05T04:54:21.601 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:21.601 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "started", 2024-02-05T04:54:21.601 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:54:21.601 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:54:21.601 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:21.601 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 633 2024-02-05T04:54:21.601 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.601 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:21.601 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.601 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:21.601 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.965050+0000", 2024-02-05T04:54:21.601 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:21.601 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.601 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.601 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:21.602 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.965050+0000", 2024-02-05T04:54:21.602 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:21.602 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.602 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.602 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:21.602 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.965050+0000", 2024-02-05T04:54:21.602 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.6800000000000002e-07 2024-02-05T04:54:21.602 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.602 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.602 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:21.602 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.965052+0000", 2024-02-05T04:54:21.602 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.4249999999999999e-06 2024-02-05T04:54:21.602 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.602 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.602 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:21.603 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.965055+0000", 2024-02-05T04:54:21.603 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.8290000000000001e-06 2024-02-05T04:54:21.603 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.603 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.603 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:21.603 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.965058+0000", 2024-02-05T04:54:21.603 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.9560000000000002e-06 2024-02-05T04:54:21.603 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.603 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.603 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:21.603 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.973682+0000", 2024-02-05T04:54:21.603 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0086237540000000008 2024-02-05T04:54:21.603 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.603 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.603 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "waiting for rw locks", 2024-02-05T04:54:21.604 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.973747+0000", 2024-02-05T04:54:21.604 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.4393000000000005e-05 2024-02-05T04:54:21.604 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.604 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.604 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:21.604 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.976885+0000", 2024-02-05T04:54:21.604 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0031381040000000001 2024-02-05T04:54:21.604 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.604 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.604 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:54:21.604 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.976961+0000", 2024-02-05T04:54:21.604 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.6206999999999997e-05 2024-02-05T04:54:21.604 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.605 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.605 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:21.605 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:00.978166+0000", 2024-02-05T04:54:21.605 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.001205404 2024-02-05T04:54:21.605 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.605 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:21.605 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.605 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.605 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.605 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.6s2 74/62 ECSubWrite(tid=337, reqid=osd.13.0:337, at_version=74'38, trim_to=0'0, roll_forward_to=72'36))", 2024-02-05T04:54:21.605 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:54:06.043590+0000", 2024-02-05T04:54:21.605 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 15.475137199000001, 2024-02-05T04:54:21.605 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0078961069999999994, 2024-02-05T04:54:21.605 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:21.605 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:54:21.606 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:21.606 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.606 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:21.606 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.043590+0000", 2024-02-05T04:54:21.606 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:21.606 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.606 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.606 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:21.606 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.043590+0000", 2024-02-05T04:54:21.606 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:21.606 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.606 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.606 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:21.606 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.043590+0000", 2024-02-05T04:54:21.606 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.6800000000000002e-07 2024-02-05T04:54:21.607 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.607 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.607 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:21.607 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.043616+0000", 2024-02-05T04:54:21.607 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.6449e-05 2024-02-05T04:54:21.607 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.607 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.607 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:21.607 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.043626+0000", 2024-02-05T04:54:21.607 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.1900000000000001e-06 2024-02-05T04:54:21.607 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.607 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.607 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:21.607 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.043634+0000", 2024-02-05T04:54:21.607 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.4530000000000002e-06 2024-02-05T04:54:21.608 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.608 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.608 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:21.608 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.043691+0000", 2024-02-05T04:54:21.608 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.7466999999999997e-05 2024-02-05T04:54:21.608 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.608 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.608 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:21.608 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.043701+0000", 2024-02-05T04:54:21.608 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.4860000000000003e-06 2024-02-05T04:54:21.608 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.608 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.608 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:21.608 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.051445+0000", 2024-02-05T04:54:21.609 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0077437119999999998 2024-02-05T04:54:21.609 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.609 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.609 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:21.609 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.051486+0000", 2024-02-05T04:54:21.609 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.1081999999999997e-05 2024-02-05T04:54:21.609 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.609 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:21.609 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.609 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.609 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.609 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.17s3 74/67 ECSubWrite(tid=4, reqid=osd.0.0:4, at_version=74'32, trim_to=0'0, roll_forward_to=0'0))", 2024-02-05T04:54:21.609 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:54:06.045714+0000", 2024-02-05T04:54:21.609 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 15.473012375, 2024-02-05T04:54:21.609 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0091829640000000001, 2024-02-05T04:54:21.610 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:21.610 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:54:21.610 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:21.610 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.610 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:21.610 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.045714+0000", 2024-02-05T04:54:21.610 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:21.610 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.610 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.610 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:21.610 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.045714+0000", 2024-02-05T04:54:21.610 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:21.610 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.610 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.610 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:21.611 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.045715+0000", 2024-02-05T04:54:21.611 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.1699999999999998e-07 2024-02-05T04:54:21.611 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.611 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.611 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:21.611 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.045736+0000", 2024-02-05T04:54:21.611 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.0944000000000001e-05 2024-02-05T04:54:21.611 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.611 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.611 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:21.611 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.045747+0000", 2024-02-05T04:54:21.611 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.1151e-05 2024-02-05T04:54:21.611 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.611 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.611 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:21.612 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.045755+0000", 2024-02-05T04:54:21.612 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.0299999999999994e-06 2024-02-05T04:54:21.612 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.612 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.612 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:21.612 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.045838+0000", 2024-02-05T04:54:21.612 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.3269000000000005e-05 2024-02-05T04:54:21.612 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.612 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.612 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:21.612 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.045848+0000", 2024-02-05T04:54:21.612 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.1099999999999992e-06 2024-02-05T04:54:21.612 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.612 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.613 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:21.613 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.054858+0000", 2024-02-05T04:54:21.613 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0090107869999999993 2024-02-05T04:54:21.613 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.613 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.613 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:21.613 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.054897+0000", 2024-02-05T04:54:21.613 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.9056e-05 2024-02-05T04:54:21.613 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.613 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:21.613 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.613 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.613 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.613 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.es1 74/57 ECSubWrite(tid=98, reqid=osd.2.0:97, at_version=0'0, trim_to=0'0, roll_forward_to=74'68))", 2024-02-05T04:54:21.613 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:54:06.046317+0000", 2024-02-05T04:54:21.614 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 15.472409787, 2024-02-05T04:54:21.614 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0086387900000000004, 2024-02-05T04:54:21.614 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:21.614 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:54:21.614 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:21.614 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.614 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:21.614 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.046317+0000", 2024-02-05T04:54:21.614 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:21.614 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.614 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.614 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:21.614 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.046317+0000", 2024-02-05T04:54:21.614 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:21.614 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.615 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.615 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:21.615 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.046318+0000", 2024-02-05T04:54:21.615 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 7.0900000000000001e-07 2024-02-05T04:54:21.615 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.615 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.615 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:21.615 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.046328+0000", 2024-02-05T04:54:21.615 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0336000000000001e-05 2024-02-05T04:54:21.615 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.615 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.615 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:21.615 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.046345+0000", 2024-02-05T04:54:21.615 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.6697999999999998e-05 2024-02-05T04:54:21.615 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.615 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.616 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:21.616 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.046354+0000", 2024-02-05T04:54:21.616 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.9770000000000006e-06 2024-02-05T04:54:21.616 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.616 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.616 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:21.616 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.046435+0000", 2024-02-05T04:54:21.616 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.1154000000000003e-05 2024-02-05T04:54:21.616 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.616 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.616 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:21.616 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.046444+0000", 2024-02-05T04:54:21.616 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 9.4700000000000008e-06 2024-02-05T04:54:21.616 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.617 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.617 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:21.617 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.054915+0000", 2024-02-05T04:54:21.617 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0084709250000000007 2024-02-05T04:54:21.617 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.617 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.617 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:21.617 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.054956+0000", 2024-02-05T04:54:21.617 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 4.0521e-05 2024-02-05T04:54:21.617 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.617 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:21.617 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.617 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.617 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.617 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "MOSDECSubOpWrite(3.4s5 74/62 ECSubWrite(tid=132, reqid=osd.1.0:132, at_version=74'81, trim_to=0'0, roll_forward_to=72'79))", 2024-02-05T04:54:21.618 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:54:06.052059+0000", 2024-02-05T04:54:21.618 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 15.466667326, 2024-02-05T04:54:21.618 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0080075749999999994, 2024-02-05T04:54:21.618 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:21.618 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "reached pg", 2024-02-05T04:54:21.618 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:21.618 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.618 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:21.618 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.052059+0000", 2024-02-05T04:54:21.618 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:21.618 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.618 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.618 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:21.618 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.052059+0000", 2024-02-05T04:54:21.619 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:21.619 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.619 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.619 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:21.619 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.052060+0000", 2024-02-05T04:54:21.619 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.3099999999999997e-07 2024-02-05T04:54:21.619 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.619 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.619 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:21.619 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.052080+0000", 2024-02-05T04:54:21.619 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.9690999999999998e-05 2024-02-05T04:54:21.619 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.619 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.619 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:21.619 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.052092+0000", 2024-02-05T04:54:21.619 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.2381000000000001e-05 2024-02-05T04:54:21.620 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.620 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.620 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:21.620 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.052101+0000", 2024-02-05T04:54:21.620 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 8.7309999999999999e-06 2024-02-05T04:54:21.620 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.620 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.620 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:21.620 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.052161+0000", 2024-02-05T04:54:21.620 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.0084000000000003e-05 2024-02-05T04:54:21.620 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.620 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.620 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:21.620 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.052171+0000", 2024-02-05T04:54:21.621 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.0025e-05 2024-02-05T04:54:21.621 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.621 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.621 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:21.621 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.060030+0000", 2024-02-05T04:54:21.621 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0078588890000000008 2024-02-05T04:54:21.621 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.621 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.621 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:21.621 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:06.060067+0000", 2024-02-05T04:54:21.621 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 3.7143e-05 2024-02-05T04:54:21.621 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.621 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:21.621 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.621 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.622 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.622 INFO:teuthology.orchestra.run.smithi062.stdout: "description": "osd_op(client.4634.0:794 3.3s0 3:c46da155:::smithi19639362-15:head [append 0~720896] snapc 1b=[1b,19,18,17,16,12,8,7] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-02-05T04:54:21.622 INFO:teuthology.orchestra.run.smithi062.stdout: "initiated_at": "2024-02-05T04:54:11.082842+0000", 2024-02-05T04:54:21.622 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 10.435885121, 2024-02-05T04:54:21.622 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0077717130000000004, 2024-02-05T04:54:21.622 INFO:teuthology.orchestra.run.smithi062.stdout: "type_data": { 2024-02-05T04:54:21.622 INFO:teuthology.orchestra.run.smithi062.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T04:54:21.622 INFO:teuthology.orchestra.run.smithi062.stdout: "client_info": { 2024-02-05T04:54:21.622 INFO:teuthology.orchestra.run.smithi062.stdout: "client": "client.4634", 2024-02-05T04:54:21.622 INFO:teuthology.orchestra.run.smithi062.stdout: "client_addr": "172.21.15.196:0/3033843686", 2024-02-05T04:54:21.622 INFO:teuthology.orchestra.run.smithi062.stdout: "tid": 794 2024-02-05T04:54:21.622 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.622 INFO:teuthology.orchestra.run.smithi062.stdout: "events": [ 2024-02-05T04:54:21.622 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.622 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "initiated", 2024-02-05T04:54:21.622 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:11.082842+0000", 2024-02-05T04:54:21.623 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:21.623 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.623 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.623 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "header_read", 2024-02-05T04:54:21.623 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:11.082842+0000", 2024-02-05T04:54:21.623 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0 2024-02-05T04:54:21.623 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.623 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.623 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "throttled", 2024-02-05T04:54:21.623 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:11.082845+0000", 2024-02-05T04:54:21.623 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.8930000000000001e-06 2024-02-05T04:54:21.623 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.623 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.623 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "all_read", 2024-02-05T04:54:21.624 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:11.083713+0000", 2024-02-05T04:54:21.624 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.00086801200000000004 2024-02-05T04:54:21.624 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.624 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.624 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "dispatched", 2024-02-05T04:54:21.624 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:11.083738+0000", 2024-02-05T04:54:21.624 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 2.5823999999999999e-05 2024-02-05T04:54:21.624 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.624 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.624 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "queued_for_pg", 2024-02-05T04:54:21.624 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:11.083751+0000", 2024-02-05T04:54:21.624 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 1.2843999999999999e-05 2024-02-05T04:54:21.624 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.624 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.625 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:21.625 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:11.083818+0000", 2024-02-05T04:54:21.625 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.6307000000000001e-05 2024-02-05T04:54:21.625 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.625 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.625 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "op must wait for map", 2024-02-05T04:54:21.625 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:11.083824+0000", 2024-02-05T04:54:21.625 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 6.1820000000000003e-06 2024-02-05T04:54:21.625 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.625 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.625 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "reached_pg", 2024-02-05T04:54:21.625 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:11.085345+0000", 2024-02-05T04:54:21.625 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.001521047 2024-02-05T04:54:21.625 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.625 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.626 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "started", 2024-02-05T04:54:21.626 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:11.085453+0000", 2024-02-05T04:54:21.626 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.000108039 2024-02-05T04:54:21.626 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.626 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.626 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_started", 2024-02-05T04:54:21.626 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:11.086618+0000", 2024-02-05T04:54:21.626 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0011647109999999999 2024-02-05T04:54:21.626 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.626 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.626 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "sub_op_committed", 2024-02-05T04:54:21.626 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:11.088676+0000", 2024-02-05T04:54:21.626 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.0020582230000000001 2024-02-05T04:54:21.626 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.626 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.627 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "commit_sent", 2024-02-05T04:54:21.627 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:11.090562+0000", 2024-02-05T04:54:21.627 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 0.001886176 2024-02-05T04:54:21.627 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-02-05T04:54:21.627 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-02-05T04:54:21.627 INFO:teuthology.orchestra.run.smithi062.stdout: "event": "done", 2024-02-05T04:54:21.627 INFO:teuthology.orchestra.run.smithi062.stdout: "time": "2024-02-05T04:54:11.090613+0000", 2024-02-05T04:54:21.627 INFO:teuthology.orchestra.run.smithi062.stdout: "duration": 5.1455000000000001e-05 2024-02-05T04:54:21.627 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.627 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:21.627 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.627 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-02-05T04:54:21.627 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-02-05T04:54:21.627 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-02-05T04:54:21.629 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-02-05T04:54:21.630 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0: {} 2024-02-05T04:54:21.630 INFO:teuthology.orchestra.run.smithi062.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T04:54:21.630 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1: {} 2024-02-05T04:54:21.630 INFO:teuthology.orchestra.run.smithi062.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T04:54:21.631 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2: {} 2024-02-05T04:54:21.631 INFO:teuthology.orchestra.run.smithi062.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T04:54:21.631 INFO:tasks.ceph.osd.8.smithi062.stderr:2024-02-05T04:54:21.628+0000 7f9385bf2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 34693) UID: 0 2024-02-05T04:54:21.634 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3: {} 2024-02-05T04:54:21.634 INFO:teuthology.orchestra.run.smithi062.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T04:54:21.651 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4: {} 2024-02-05T04:54:21.651 INFO:teuthology.orchestra.run.smithi062.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T04:54:21.670 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5: {} 2024-02-05T04:54:21.670 INFO:teuthology.orchestra.run.smithi062.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T04:54:21.688 INFO:teuthology.orchestra.run.smithi062.stdout:osd.6: {} 2024-02-05T04:54:21.688 INFO:teuthology.orchestra.run.smithi062.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T04:54:21.707 INFO:teuthology.orchestra.run.smithi062.stdout:osd.7: {} 2024-02-05T04:54:21.707 INFO:teuthology.orchestra.run.smithi062.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T04:54:21.724 INFO:teuthology.orchestra.run.smithi062.stdout:osd.8: {} 2024-02-05T04:54:21.724 INFO:teuthology.orchestra.run.smithi062.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-05T04:54:21.730 INFO:tasks.ceph.osd.5.smithi091.stderr:2024-02-05T04:54:21.728+0000 7f63983b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34277) UID: 0 2024-02-05T04:54:21.742 INFO:teuthology.orchestra.run.smithi062.stdout:osd.9: {} 2024-02-05T04:54:21.743 INFO:teuthology.orchestra.run.smithi062.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-05T04:54:21.743 INFO:teuthology.orchestra.run.smithi062.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:21.743 INFO:teuthology.orchestra.run.smithi062.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:21.752 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-02-05T04:54:21.760 INFO:teuthology.orchestra.run.smithi062.stdout:osd.10: {} 2024-02-05T04:54:21.760 INFO:teuthology.orchestra.run.smithi062.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-05T04:54:21.778 INFO:teuthology.orchestra.run.smithi062.stdout:osd.11: {} 2024-02-05T04:54:21.779 INFO:teuthology.orchestra.run.smithi062.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-05T04:54:21.797 INFO:teuthology.orchestra.run.smithi062.stdout:osd.12: {} 2024-02-05T04:54:21.797 INFO:teuthology.orchestra.run.smithi062.stderr:osd.12: osd_enable_op_tracker = '' 2024-02-05T04:54:21.798 INFO:teuthology.orchestra.run.smithi062.stderr:Error ENXIO: problem getting command descriptions from osd.13 2024-02-05T04:54:21.816 INFO:teuthology.orchestra.run.smithi062.stdout:osd.14: {} 2024-02-05T04:54:21.816 INFO:teuthology.orchestra.run.smithi062.stderr:osd.14: osd_enable_op_tracker = '' 2024-02-05T04:54:21.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-02-05T04:54:21.815+0000 7fef88c9c640 0 -- 172.21.15.62:0/1381051000 >> [v2:172.21.15.196:6826/3187639683,v1:172.21.15.196:6827/3187639683] conn(0x7fef8421d6a0 msgr2=0x7fef8421da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-02-05T04:54:21.831 INFO:tasks.ceph.osd.5.smithi091.stderr:2024-02-05T04:54:21.829+0000 7f63983b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34277) UID: 0 2024-02-05T04:54:21.867 INFO:teuthology.orchestra.run.smithi062.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:21.868 INFO:teuthology.orchestra.run.smithi062.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:21.876 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-02-05T04:54:21.932 INFO:tasks.ceph.osd.6.smithi150.stderr:2024-02-05T04:54:21.931+0000 7f6bea9f0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34015) UID: 0 2024-02-05T04:54:22.024 INFO:teuthology.orchestra.run.smithi062.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:22.024 INFO:teuthology.orchestra.run.smithi062.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:22.032 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-02-05T04:54:22.033 INFO:tasks.ceph.osd.3.smithi196.stderr:2024-02-05T04:54:22.031+0000 7fc8817c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34605) UID: 0 2024-02-05T04:54:22.035 INFO:teuthology.orchestra.run.smithi062.stdout:osd.15: {} 2024-02-05T04:54:22.036 INFO:teuthology.orchestra.run.smithi062.stderr:osd.15: osd_enable_op_tracker = '' 2024-02-05T04:54:22.038 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-05T04:54:22.038 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-05T04:54:22.097 INFO:teuthology.orchestra.run.smithi062.stderr:pool 'unique_pool_0' removed 2024-02-05T04:54:22.114 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-02-05T04:54:22.124 INFO:tasks.thrashosds:joining thrashosds 2024-02-05T04:54:22.124 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-02-05T04:54:22.132 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-02-05T04:54:22.146 INFO:teuthology.orchestra.run.smithi062.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:22.147 INFO:teuthology.orchestra.run.smithi062.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:22.155 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-02-05T04:54:22.302 INFO:teuthology.orchestra.run.smithi062.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:22.303 INFO:teuthology.orchestra.run.smithi062.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:22.311 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-02-05T04:54:22.418 INFO:teuthology.orchestra.run.smithi062.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:22.418 INFO:teuthology.orchestra.run.smithi062.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:22.426 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-02-05T04:54:22.543 INFO:teuthology.orchestra.run.smithi091.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:22.543 INFO:teuthology.orchestra.run.smithi091.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:22.553 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-02-05T04:54:22.659 INFO:teuthology.orchestra.run.smithi091.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:22.659 INFO:teuthology.orchestra.run.smithi091.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:22.668 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-02-05T04:54:22.774 INFO:teuthology.orchestra.run.smithi091.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:22.774 INFO:teuthology.orchestra.run.smithi091.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:22.785 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-02-05T04:54:22.896 INFO:teuthology.orchestra.run.smithi091.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:22.896 INFO:teuthology.orchestra.run.smithi091.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:22.905 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-02-05T04:54:23.011 INFO:teuthology.orchestra.run.smithi091.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:23.011 INFO:teuthology.orchestra.run.smithi091.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:23.020 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-02-05T04:54:23.132 INFO:teuthology.orchestra.run.smithi091.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:23.132 INFO:teuthology.orchestra.run.smithi091.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:23.141 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-02-05T04:54:23.141 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-02-05T04:54:23.247 INFO:teuthology.orchestra.run.smithi091.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:23.247 INFO:teuthology.orchestra.run.smithi091.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:23.255 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-02-05T04:54:23.359 INFO:teuthology.orchestra.run.smithi091.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:23.360 INFO:teuthology.orchestra.run.smithi091.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:23.368 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-02-05T04:54:23.473 INFO:teuthology.orchestra.run.smithi091.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:23.473 INFO:teuthology.orchestra.run.smithi091.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:23.482 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-02-05T04:54:23.599 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:23.600 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:23.609 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-02-05T04:54:23.715 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:23.715 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:23.724 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-02-05T04:54:23.830 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:24.103 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:24.104 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-02-05T04:54:24.105 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-02-05T04:54:24.107 INFO:teuthology.orchestra.run.smithi062.stderr:noscrub is unset 2024-02-05T04:54:24.124 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-02-05T04:54:24.212 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:24.212 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:24.221 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-02-05T04:54:24.327 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:24.327 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:24.335 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-02-05T04:54:24.441 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:24.442 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:24.450 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-02-05T04:54:24.557 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:24.557 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:24.566 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-02-05T04:54:24.671 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:24.672 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:24.680 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-02-05T04:54:24.786 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:24.786 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:24.794 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-02-05T04:54:24.900 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:24.900 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:24.908 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-02-05T04:54:25.014 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:25.014 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:25.025 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-02-05T04:54:25.109 INFO:teuthology.orchestra.run.smithi062.stderr:nodeep-scrub is unset 2024-02-05T04:54:25.131 INFO:teuthology.orchestra.run.smithi150.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:25.131 INFO:teuthology.orchestra.run.smithi150.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:25.140 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-02-05T04:54:25.260 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:25.260 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:25.268 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-02-05T04:54:25.374 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:25.375 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:25.384 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-02-05T04:54:25.489 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:25.489 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:25.499 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-02-05T04:54:25.604 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:25.604 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:25.613 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-02-05T04:54:25.717 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:25.717 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:25.726 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-02-05T04:54:25.831 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:25.831 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:25.840 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-02-05T04:54:25.946 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:25.946 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:25.954 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-02-05T04:54:26.059 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:26.059 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:26.068 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-02-05T04:54:26.158 INFO:tasks.daemonwatchdog.daemon_watchdog:OSDThrasher failed 2024-02-05T04:54:26.158 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2024-02-05T04:54:26.173 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:26.173 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:26.182 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-02-05T04:54:26.287 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:26.287 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:26.296 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-02-05T04:54:26.401 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:26.402 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:26.410 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-02-05T04:54:26.515 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T04:54:26.515 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T04:54:26.524 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-02-05T04:54:26.524 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-02-05T04:54:26.524 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-05T04:54:26.829 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-02-05T04:54:26.829 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":92,"fsid":"88c11a2d-382e-4f21-b8af-45ab9c504b93","created":"2024-02-05T04:52:55.182961+0000","modified":"2024-02-05T04:54:25.107539+0000","last_up_change":"2024-02-05T04:54:13.063833+0000","last_in_change":"2024-02-05T04:54:14.065808+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-05T04:53:07.549923+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-02-05T04:53:10.126961+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cdb503c5-a32c-4a9a-b1d9-194ab5ee48aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":1189094240},{"type":"v1","addr":"172.21.15.62:6816","nonce":1189094240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":1189094240},{"type":"v1","addr":"172.21.15.62:6819","nonce":1189094240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":1189094240},{"type":"v1","addr":"172.21.15.62:6823","nonce":1189094240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":1189094240},{"type":"v1","addr":"172.21.15.62:6821","nonce":1189094240}]},"public_addr":"172.21.15.62:6816/1189094240","cluster_addr":"172.21.15.62:6819/1189094240","heartbeat_back_addr":"172.21.15.62:6823/1189094240","heartbeat_front_addr":"172.21.15.62:6821/1189094240","state":["exists","up"]},{"osd":1,"uuid":"d7a3d394-74b2-4cb1-a754-4fe77e605ea9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6808","nonce":2878185190},{"type":"v1","addr":"172.21.15.91:6811","nonce":2878185190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6814","nonce":2878185190},{"type":"v1","addr":"172.21.15.91:6818","nonce":2878185190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6828","nonce":2878185190},{"type":"v1","addr":"172.21.15.91:6830","nonce":2878185190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6821","nonce":2878185190},{"type":"v1","addr":"172.21.15.91:6824","nonce":2878185190}]},"public_addr":"172.21.15.91:6811/2878185190","cluster_addr":"172.21.15.91:6818/2878185190","heartbeat_back_addr":"172.21.15.91:6830/2878185190","heartbeat_front_addr":"172.21.15.91:6824/2878185190","state":["exists","up"]},{"osd":2,"uuid":"d6e99efe-475f-41c1-88a3-87ce6b26ce75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":40232889},{"type":"v1","addr":"172.21.15.150:6825","nonce":40232889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":40232889},{"type":"v1","addr":"172.21.15.150:6827","nonce":40232889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":40232889},{"type":"v1","addr":"172.21.15.150:6831","nonce":40232889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":40232889},{"type":"v1","addr":"172.21.15.150:6829","nonce":40232889}]},"public_addr":"172.21.15.150:6825/40232889","cluster_addr":"172.21.15.150:6827/40232889","heartbeat_back_addr":"172.21.15.150:6831/40232889","heartbeat_front_addr":"172.21.15.150:6829/40232889","state":["exists","up"]},{"osd":3,"uuid":"3c964090-5e07-40bd-ac7c-b66e29eb25b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":3834523862},{"type":"v1","addr":"172.21.15.196:6803","nonce":3834523862}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":3834523862},{"type":"v1","addr":"172.21.15.196:6805","nonce":3834523862}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":3834523862},{"type":"v1","addr":"172.21.15.196:6809","nonce":3834523862}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":3834523862},{"type":"v1","addr":"172.21.15.196:6807","nonce":3834523862}]},"public_addr":"172.21.15.196:6803/3834523862","cluster_addr":"172.21.15.196:6805/3834523862","heartbeat_back_addr":"172.21.15.196:6809/3834523862","heartbeat_front_addr":"172.21.15.196:6807/3834523862","state":["exists","up"]},{"osd":4,"uuid":"22bd25f9-3d93-4bcb-bcfc-71feae1097f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":2244900897},{"type":"v1","addr":"172.21.15.62:6801","nonce":2244900897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":2244900897},{"type":"v1","addr":"172.21.15.62:6803","nonce":2244900897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":2244900897},{"type":"v1","addr":"172.21.15.62:6807","nonce":2244900897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":2244900897},{"type":"v1","addr":"172.21.15.62:6805","nonce":2244900897}]},"public_addr":"172.21.15.62:6801/2244900897","cluster_addr":"172.21.15.62:6803/2244900897","heartbeat_back_addr":"172.21.15.62:6807/2244900897","heartbeat_front_addr":"172.21.15.62:6805/2244900897","state":["exists","up"]},{"osd":5,"uuid":"c6ae2bd4-4a4f-4f23-8991-743ffa42ed3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6800","nonce":473596866},{"type":"v1","addr":"172.21.15.91:6801","nonce":473596866}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6802","nonce":473596866},{"type":"v1","addr":"172.21.15.91:6803","nonce":473596866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6806","nonce":473596866},{"type":"v1","addr":"172.21.15.91:6807","nonce":473596866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6804","nonce":473596866},{"type":"v1","addr":"172.21.15.91:6805","nonce":473596866}]},"public_addr":"172.21.15.91:6801/473596866","cluster_addr":"172.21.15.91:6803/473596866","heartbeat_back_addr":"172.21.15.91:6807/473596866","heartbeat_front_addr":"172.21.15.91:6805/473596866","state":["exists","up"]},{"osd":6,"uuid":"49da8b05-ffa3-4172-983e-f081d3eb9ae6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":2199648645},{"type":"v1","addr":"172.21.15.150:6801","nonce":2199648645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":2199648645},{"type":"v1","addr":"172.21.15.150:6803","nonce":2199648645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":2199648645},{"type":"v1","addr":"172.21.15.150:6807","nonce":2199648645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":2199648645},{"type":"v1","addr":"172.21.15.150:6805","nonce":2199648645}]},"public_addr":"172.21.15.150:6801/2199648645","cluster_addr":"172.21.15.150:6803/2199648645","heartbeat_back_addr":"172.21.15.150:6807/2199648645","heartbeat_front_addr":"172.21.15.150:6805/2199648645","state":["exists","up"]},{"osd":7,"uuid":"4ece26e1-2c63-4223-baaf-882cd8261ade","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":254726044},{"type":"v1","addr":"172.21.15.196:6811","nonce":254726044}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":254726044},{"type":"v1","addr":"172.21.15.196:6813","nonce":254726044}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":254726044},{"type":"v1","addr":"172.21.15.196:6820","nonce":254726044}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":254726044},{"type":"v1","addr":"172.21.15.196:6816","nonce":254726044}]},"public_addr":"172.21.15.196:6811/254726044","cluster_addr":"172.21.15.196:6813/254726044","heartbeat_back_addr":"172.21.15.196:6820/254726044","heartbeat_front_addr":"172.21.15.196:6816/254726044","state":["exists","up"]},{"osd":8,"uuid":"9ac666e9-291e-4696-a6bb-e10c9deef3f5","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":3461147157},{"type":"v1","addr":"172.21.15.62:6809","nonce":3461147157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":3461147157},{"type":"v1","addr":"172.21.15.62:6811","nonce":3461147157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":3461147157},{"type":"v1","addr":"172.21.15.62:6817","nonce":3461147157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":3461147157},{"type":"v1","addr":"172.21.15.62:6813","nonce":3461147157}]},"public_addr":"172.21.15.62:6809/3461147157","cluster_addr":"172.21.15.62:6811/3461147157","heartbeat_back_addr":"172.21.15.62:6817/3461147157","heartbeat_front_addr":"172.21.15.62:6813/3461147157","state":["exists","up"]},{"osd":9,"uuid":"7fe04cb8-ac67-46ac-b2de-1f8f60a50373","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6810","nonce":1859221562},{"type":"v1","addr":"172.21.15.91:6813","nonce":1859221562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6816","nonce":1859221562},{"type":"v1","addr":"172.21.15.91:6820","nonce":1859221562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6829","nonce":1859221562},{"type":"v1","addr":"172.21.15.91:6831","nonce":1859221562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6823","nonce":1859221562},{"type":"v1","addr":"172.21.15.91:6826","nonce":1859221562}]},"public_addr":"172.21.15.91:6813/1859221562","cluster_addr":"172.21.15.91:6820/1859221562","heartbeat_back_addr":"172.21.15.91:6831/1859221562","heartbeat_front_addr":"172.21.15.91:6826/1859221562","state":["exists","up"]},{"osd":10,"uuid":"a6b0ac25-da07-4fee-b646-a4530ce5b343","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2448697550},{"type":"v1","addr":"172.21.15.150:6809","nonce":2448697550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2448697550},{"type":"v1","addr":"172.21.15.150:6811","nonce":2448697550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":2448697550},{"type":"v1","addr":"172.21.15.150:6819","nonce":2448697550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6813","nonce":2448697550},{"type":"v1","addr":"172.21.15.150:6815","nonce":2448697550}]},"public_addr":"172.21.15.150:6809/2448697550","cluster_addr":"172.21.15.150:6811/2448697550","heartbeat_back_addr":"172.21.15.150:6819/2448697550","heartbeat_front_addr":"172.21.15.150:6815/2448697550","state":["exists","up"]},{"osd":11,"uuid":"31b22258-77da-495c-b518-9ce0297b9165","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6815","nonce":1170435852},{"type":"v1","addr":"172.21.15.196:6817","nonce":1170435852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6819","nonce":1170435852},{"type":"v1","addr":"172.21.15.196:6821","nonce":1170435852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":1170435852},{"type":"v1","addr":"172.21.15.196:6825","nonce":1170435852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":1170435852},{"type":"v1","addr":"172.21.15.196:6823","nonce":1170435852}]},"public_addr":"172.21.15.196:6817/1170435852","cluster_addr":"172.21.15.196:6821/1170435852","heartbeat_back_addr":"172.21.15.196:6825/1170435852","heartbeat_front_addr":"172.21.15.196:6823/1170435852","state":["exists","up"]},{"osd":12,"uuid":"322917eb-26ba-4439-8300-4d723bada623","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":1173398345},{"type":"v1","addr":"172.21.15.62:6825","nonce":1173398345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":1173398345},{"type":"v1","addr":"172.21.15.62:6827","nonce":1173398345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":1173398345},{"type":"v1","addr":"172.21.15.62:6831","nonce":1173398345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":1173398345},{"type":"v1","addr":"172.21.15.62:6829","nonce":1173398345}]},"public_addr":"172.21.15.62:6825/1173398345","cluster_addr":"172.21.15.62:6827/1173398345","heartbeat_back_addr":"172.21.15.62:6831/1173398345","heartbeat_front_addr":"172.21.15.62:6829/1173398345","state":["exists","up"]},{"osd":13,"uuid":"9c708044-4726-42b6-9c13-03963f956d2d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":62,"down_at":81,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6809","nonce":3656255795},{"type":"v1","addr":"172.21.15.91:6812","nonce":3656255795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6815","nonce":3656255795},{"type":"v1","addr":"172.21.15.91:6817","nonce":3656255795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6825","nonce":3656255795},{"type":"v1","addr":"172.21.15.91:6827","nonce":3656255795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6819","nonce":3656255795},{"type":"v1","addr":"172.21.15.91:6822","nonce":3656255795}]},"public_addr":"172.21.15.91:6812/3656255795","cluster_addr":"172.21.15.91:6817/3656255795","heartbeat_back_addr":"172.21.15.91:6827/3656255795","heartbeat_front_addr":"172.21.15.91:6822/3656255795","state":["exists"]},{"osd":14,"uuid":"0208144a-db9a-404c-b31e-13bc2e276b09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2102158830},{"type":"v1","addr":"172.21.15.150:6814","nonce":2102158830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":2102158830},{"type":"v1","addr":"172.21.15.150:6818","nonce":2102158830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":2102158830},{"type":"v1","addr":"172.21.15.150:6824","nonce":2102158830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":2102158830},{"type":"v1","addr":"172.21.15.150:6821","nonce":2102158830}]},"public_addr":"172.21.15.150:6814/2102158830","cluster_addr":"172.21.15.150:6818/2102158830","heartbeat_back_addr":"172.21.15.150:6824/2102158830","heartbeat_front_addr":"172.21.15.150:6821/2102158830","state":["exists","up"]},{"osd":15,"uuid":"0591e7e1-dc07-4d80-b574-9b8348635764","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":3187639683},{"type":"v1","addr":"172.21.15.196:6827","nonce":3187639683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":3187639683},{"type":"v1","addr":"172.21.15.196:6829","nonce":3187639683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":3187639683},{"type":"v1","addr":"172.21.15.196:6833","nonce":3187639683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":3187639683},{"type":"v1","addr":"172.21.15.196:6831","nonce":3187639683}]},"public_addr":"172.21.15.196:6827/3187639683","cluster_addr":"172.21.15.196:6829/3187639683","heartbeat_back_addr":"172.21.15.196:6833/3187639683","heartbeat_front_addr":"172.21.15.196:6831/3187639683","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T04:54:21.777323+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T04:54:16.107345+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T04:53:04.897353+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T04:54:24.800302+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T04:54:21.785736+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T04:53:04.777687+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T04:54:11.123847+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T04:54:16.474065+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":65536,"last_purged_snaps_scrub":"2024-02-05T04:54:20.676290+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T04:54:21.097980+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T04:54:12.049048+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T04:54:13.398408+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T04:54:05.894943+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":65536,"last_purged_snaps_scrub":"2024-02-05T04:53:04.815624+0000","dead_epoch":81},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T04:54:06.295323+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T04:54:14.351177+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":7,"length":1},{"begin":22,"length":1},{"begin":26,"length":2},{"begin":29,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-05T04:54:28.173 INFO:tasks.ceph.osd.0:Sent signal 15 2024-02-05T04:54:28.174 INFO:tasks.ceph.osd.4:Sent signal 15 2024-02-05T04:54:28.174 INFO:tasks.ceph.osd.8:Sent signal 15 2024-02-05T04:54:28.174 INFO:tasks.ceph.osd.12:Sent signal 15 2024-02-05T04:54:28.175 INFO:tasks.ceph.osd.1:Sent signal 15 2024-02-05T04:54:28.175 INFO:tasks.ceph.osd.5:Sent signal 15 2024-02-05T04:54:28.175 INFO:tasks.ceph.osd.9:Sent signal 15 2024-02-05T04:54:28.175 INFO:tasks.ceph.osd.2:Sent signal 15 2024-02-05T04:54:28.175 INFO:tasks.ceph.osd.6:Sent signal 15 2024-02-05T04:54:28.176 INFO:tasks.ceph.osd.10:Sent signal 15 2024-02-05T04:54:28.176 INFO:tasks.ceph.osd.14:Sent signal 15 2024-02-05T04:54:28.176 INFO:tasks.ceph.osd.3:Sent signal 15 2024-02-05T04:54:28.176 INFO:tasks.ceph.osd.7:Sent signal 15 2024-02-05T04:54:28.176 INFO:tasks.ceph.osd.11:Sent signal 15 2024-02-05T04:54:28.177 INFO:tasks.ceph.osd.15:Sent signal 15 2024-02-05T04:54:28.177 INFO:tasks.ceph.mon.a:Sent signal 15 2024-02-05T04:54:28.177 INFO:tasks.ceph.mon.b:Sent signal 15 2024-02-05T04:54:28.177 INFO:tasks.ceph.mon.c:Sent signal 15 2024-02-05T04:54:28.177 INFO:tasks.ceph.mgr.y:Sent signal 15 2024-02-05T04:54:28.178 INFO:tasks.ceph.mgr.x:Sent signal 15 2024-02-05T04:54:28.179 INFO:tasks.ceph.osd.3.smithi196.stderr:2024-02-05T04:54:28.176+0000 7fc8817c4640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34605) UID: 0 2024-02-05T04:54:28.179 INFO:tasks.ceph.osd.3.smithi196.stderr:2024-02-05T04:54:28.176+0000 7fc8817c4640 -1 osd.3 92 *** Got signal Terminated *** 2024-02-05T04:54:28.179 INFO:tasks.ceph.osd.3.smithi196.stderr:2024-02-05T04:54:28.176+0000 7fc8817c4640 -1 osd.3 92 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-05T04:54:28.179 INFO:tasks.ceph.osd.2.smithi150.stderr:2024-02-05T04:54:28.175+0000 7f7337bf9640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 34014) UID: 0 2024-02-05T04:54:28.179 INFO:tasks.ceph.osd.2.smithi150.stderr:2024-02-05T04:54:28.175+0000 7f7337bf9640 -1 osd.2 92 *** Got signal Terminated *** 2024-02-05T04:54:28.179 INFO:tasks.ceph.osd.2.smithi150.stderr:2024-02-05T04:54:28.175+0000 7f7337bf9640 -1 osd.2 92 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-05T04:54:28.180 INFO:tasks.ceph.osd.1.smithi091.stderr:2024-02-05T04:54:28.174+0000 7ff9dc283640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34275) UID: 0 2024-02-05T04:54:28.180 INFO:tasks.ceph.osd.1.smithi091.stderr:2024-02-05T04:54:28.174+0000 7ff9dc283640 -1 osd.1 92 *** Got signal Terminated *** 2024-02-05T04:54:28.180 INFO:tasks.ceph.osd.1.smithi091.stderr:2024-02-05T04:54:28.174+0000 7ff9dc283640 -1 osd.1 92 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-05T04:54:28.180 INFO:tasks.ceph.osd.0.smithi062.stderr:2024-02-05T04:54:28.173+0000 7fece8e03640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34688) UID: 0 2024-02-05T04:54:28.180 INFO:tasks.ceph.osd.0.smithi062.stderr:2024-02-05T04:54:28.173+0000 7fece8e03640 -1 osd.0 92 *** Got signal Terminated *** 2024-02-05T04:54:28.180 INFO:tasks.ceph.osd.0.smithi062.stderr:2024-02-05T04:54:28.173+0000 7fece8e03640 -1 osd.0 92 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-05T04:54:28.181 INFO:tasks.ceph.osd.7.smithi196.stderr:2024-02-05T04:54:28.176+0000 7fb281419640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34606) UID: 0 2024-02-05T04:54:28.181 INFO:tasks.ceph.osd.7.smithi196.stderr:2024-02-05T04:54:28.176+0000 7fb281419640 -1 osd.7 92 *** Got signal Terminated *** 2024-02-05T04:54:28.181 INFO:tasks.ceph.osd.7.smithi196.stderr:2024-02-05T04:54:28.176+0000 7fb281419640 -1 osd.7 92 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-05T04:54:28.181 INFO:tasks.ceph.osd.5.smithi091.stderr:2024-02-05T04:54:28.174+0000 7f63983b6640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34277) UID: 0 2024-02-05T04:54:28.181 INFO:tasks.ceph.osd.5.smithi091.stderr:2024-02-05T04:54:28.174+0000 7f63983b6640 -1 osd.5 92 *** Got signal Terminated *** 2024-02-05T04:54:28.181 INFO:tasks.ceph.osd.5.smithi091.stderr:2024-02-05T04:54:28.174+0000 7f63983b6640 -1 osd.5 92 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-05T04:54:28.182 INFO:tasks.ceph.osd.11.smithi196.stderr:2024-02-05T04:54:28.176+0000 7f85a8d86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 34609) UID: 0 2024-02-05T04:54:28.182 INFO:tasks.ceph.osd.11.smithi196.stderr:2024-02-05T04:54:28.176+0000 7f85a8d86640 -1 osd.11 92 *** Got signal Terminated *** 2024-02-05T04:54:28.182 INFO:tasks.ceph.osd.11.smithi196.stderr:2024-02-05T04:54:28.176+0000 7f85a8d86640 -1 osd.11 92 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-05T04:54:28.182 INFO:tasks.ceph.osd.6.smithi150.stderr:2024-02-05T04:54:28.175+0000 7f6bea9f0640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34015) UID: 0 2024-02-05T04:54:28.182 INFO:tasks.ceph.osd.6.smithi150.stderr:2024-02-05T04:54:28.175+0000 7f6bea9f0640 -1 osd.6 92 *** Got signal Terminated *** 2024-02-05T04:54:28.182 INFO:tasks.ceph.osd.6.smithi150.stderr:2024-02-05T04:54:28.175+0000 7f6bea9f0640 -1 osd.6 92 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-05T04:54:28.182 INFO:tasks.ceph.osd.9.smithi091.stderr:2024-02-05T04:54:28.174+0000 7efc18827640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 34279) UID: 0 2024-02-05T04:54:28.183 INFO:tasks.ceph.osd.9.smithi091.stderr:2024-02-05T04:54:28.174+0000 7efc18827640 -1 osd.9 92 *** Got signal Terminated *** 2024-02-05T04:54:28.183 INFO:tasks.ceph.osd.9.smithi091.stderr:2024-02-05T04:54:28.174+0000 7efc18827640 -1 osd.9 92 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-05T04:54:28.183 INFO:tasks.ceph.osd.4.smithi062.stderr:2024-02-05T04:54:28.174+0000 7f708c308640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34689) UID: 0 2024-02-05T04:54:28.183 INFO:tasks.ceph.osd.4.smithi062.stderr:2024-02-05T04:54:28.174+0000 7f708c308640 -1 osd.4 92 *** Got signal Terminated *** 2024-02-05T04:54:28.183 INFO:tasks.ceph.osd.4.smithi062.stderr:2024-02-05T04:54:28.174+0000 7f708c308640 -1 osd.4 92 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-05T04:54:28.183 INFO:tasks.ceph.osd.15.smithi196.stderr:2024-02-05T04:54:28.176+0000 7f1facc45640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 34611) UID: 0 2024-02-05T04:54:28.183 INFO:tasks.ceph.osd.15.smithi196.stderr:2024-02-05T04:54:28.176+0000 7f1facc45640 -1 osd.15 92 *** Got signal Terminated *** 2024-02-05T04:54:28.183 INFO:tasks.ceph.osd.15.smithi196.stderr:2024-02-05T04:54:28.176+0000 7f1facc45640 -1 osd.15 92 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-05T04:54:28.183 INFO:tasks.ceph.osd.10.smithi150.stderr:2024-02-05T04:54:28.175+0000 7f3075a4d640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 34018) UID: 0 2024-02-05T04:54:28.184 INFO:tasks.ceph.osd.10.smithi150.stderr:2024-02-05T04:54:28.175+0000 7f3075a4d640 -1 osd.10 92 *** Got signal Terminated *** 2024-02-05T04:54:28.184 INFO:tasks.ceph.osd.10.smithi150.stderr:2024-02-05T04:54:28.175+0000 7f3075a4d640 -1 osd.10 92 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-05T04:54:28.184 INFO:tasks.ceph.mon.b.smithi091.stderr:2024-02-05T04:54:28.176+0000 7f1b33df2640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster ceph -i b (PID: 34094) UID: 0 2024-02-05T04:54:28.184 INFO:tasks.ceph.mon.b.smithi091.stderr:2024-02-05T04:54:28.176+0000 7f1b33df2640 -1 mon.b@1(peon) e1 *** Got Signal Terminated *** 2024-02-05T04:54:28.184 INFO:tasks.ceph.osd.14.smithi150.stderr:2024-02-05T04:54:28.175+0000 7f5f81083640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 34020) UID: 0 2024-02-05T04:54:28.184 INFO:tasks.ceph.osd.14.smithi150.stderr:2024-02-05T04:54:28.175+0000 7f5f81083640 -1 osd.14 92 *** Got signal Terminated *** 2024-02-05T04:54:28.185 INFO:tasks.ceph.osd.14.smithi150.stderr:2024-02-05T04:54:28.175+0000 7f5f81083640 -1 osd.14 92 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-05T04:54:28.185 INFO:tasks.ceph.osd.8.smithi062.stderr:2024-02-05T04:54:28.174+0000 7f9385bf2640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 34693) UID: 0 2024-02-05T04:54:28.185 INFO:tasks.ceph.osd.8.smithi062.stderr:2024-02-05T04:54:28.174+0000 7f9385bf2640 -1 osd.8 92 *** Got signal Terminated *** 2024-02-05T04:54:28.185 INFO:tasks.ceph.osd.8.smithi062.stderr:2024-02-05T04:54:28.174+0000 7f9385bf2640 -1 osd.8 92 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-05T04:54:28.185 INFO:tasks.ceph.mon.c.smithi150.stderr:2024-02-05T04:54:28.177+0000 7ffbd72c3640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster ceph -i c (PID: 33833) UID: 0 2024-02-05T04:54:28.185 INFO:tasks.ceph.mon.c.smithi150.stderr:2024-02-05T04:54:28.177+0000 7ffbd72c3640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2024-02-05T04:54:28.186 INFO:tasks.ceph.osd.12.smithi062.stderr:2024-02-05T04:54:28.174+0000 7f5e5847e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 34695) UID: 0 2024-02-05T04:54:28.186 INFO:tasks.ceph.osd.12.smithi062.stderr:2024-02-05T04:54:28.174+0000 7f5e5847e640 -1 osd.12 92 *** Got signal Terminated *** 2024-02-05T04:54:28.186 INFO:tasks.ceph.osd.12.smithi062.stderr:2024-02-05T04:54:28.174+0000 7f5e5847e640 -1 osd.12 92 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-05T04:54:28.186 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-02-05T04:54:28.177+0000 7fe604a86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster ceph -i a (PID: 34413) UID: 0 2024-02-05T04:54:28.186 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-02-05T04:54:28.177+0000 7fe604a86640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-02-05T04:54:28.379 INFO:tasks.ceph.mgr.y.smithi062.stderr:daemon-helper: command crashed with signal 15 2024-02-05T04:54:29.842 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-05T04:56:29.881 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-02-05T04:56:29.883 ERROR:teuthology.run_tasks:Manager failed: thrashosds Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/thrashosds.py", line 215, in task cluster_manager.wait_for_all_osds_up() File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/ceph_manager.py", line 2808, in wait_for_all_osds_up while not self.are_all_osds_up(): File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/ceph_manager.py", line 2798, in are_all_osds_up x = self.get_osd_dump() File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/ceph_manager.py", line 2561, in get_osd_dump return self.get_osd_dump_json()['osds'] File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/ceph_manager.py", line 2553, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/ceph_manager.py", line 1627, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/ceph_manager.py", line 1618, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi062 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' 2024-02-05T04:56:29.886 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-02-05T04:56:29.898 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-02-05T04:56:29.898 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-05T04:58:29.936 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-02-05T04:58:29.937 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/thrashosds.py", line 215, in task cluster_manager.wait_for_all_osds_up() File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/ceph_manager.py", line 2808, in wait_for_all_osds_up while not self.are_all_osds_up(): File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/ceph_manager.py", line 2798, in are_all_osds_up x = self.get_osd_dump() File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/ceph_manager.py", line 2561, in get_osd_dump return self.get_osd_dump_json()['osds'] File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/ceph_manager.py", line 2553, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/ceph_manager.py", line 1627, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/ceph_manager.py", line 1618, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi062 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/ceph_manager.py", line 2774, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/ceph_manager.py", line 2606, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/ceph_manager.py", line 2372, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/ceph_manager.py", line 1627, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/ceph_manager.py", line 1618, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi062 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2024-02-05T04:58:29.939 INFO:teuthology.misc:Shutting down mds daemons... 2024-02-05T04:58:29.939 INFO:teuthology.misc:Shutting down osd daemons... 2024-02-05T04:58:29.939 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-02-05T04:58:29.939 INFO:teuthology.orchestra.run:waiting for 300 2024-02-05T04:58:29.939 INFO:tasks.ceph.osd.0:Stopped 2024-02-05T04:58:29.940 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-02-05T04:58:29.940 INFO:teuthology.orchestra.run:waiting for 300 2024-02-05T04:58:29.940 INFO:tasks.ceph.osd.4:Stopped 2024-02-05T04:58:29.940 DEBUG:tasks.ceph.osd.8:waiting for process to exit 2024-02-05T04:58:29.940 INFO:teuthology.orchestra.run:waiting for 300 2024-02-05T04:58:29.940 INFO:tasks.ceph.osd.8:Stopped 2024-02-05T04:58:29.940 DEBUG:tasks.ceph.osd.12:waiting for process to exit 2024-02-05T04:58:29.940 INFO:teuthology.orchestra.run:waiting for 300 2024-02-05T04:58:29.941 INFO:tasks.ceph.osd.12:Stopped 2024-02-05T04:58:29.941 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-02-05T04:58:29.941 INFO:teuthology.orchestra.run:waiting for 300 2024-02-05T04:58:29.941 INFO:tasks.ceph.osd.1:Stopped 2024-02-05T04:58:29.941 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-02-05T04:58:29.941 INFO:teuthology.orchestra.run:waiting for 300 2024-02-05T04:58:29.941 INFO:tasks.ceph.osd.5:Stopped 2024-02-05T04:58:29.941 DEBUG:tasks.ceph.osd.9:waiting for process to exit 2024-02-05T04:58:29.941 INFO:teuthology.orchestra.run:waiting for 300 2024-02-05T04:58:29.942 INFO:tasks.ceph.osd.9:Stopped 2024-02-05T04:58:29.942 ERROR:tasks.ceph.osd.13:tried to stop a non-running daemon 2024-02-05T04:58:29.942 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-02-05T04:58:29.942 INFO:teuthology.orchestra.run:waiting for 300 2024-02-05T04:58:29.942 INFO:tasks.ceph.osd.2:Stopped 2024-02-05T04:58:29.942 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-02-05T04:58:29.942 INFO:teuthology.orchestra.run:waiting for 300 2024-02-05T04:58:29.942 INFO:tasks.ceph.osd.6:Stopped 2024-02-05T04:58:29.942 DEBUG:tasks.ceph.osd.10:waiting for process to exit 2024-02-05T04:58:29.942 INFO:teuthology.orchestra.run:waiting for 300 2024-02-05T04:58:29.943 INFO:tasks.ceph.osd.10:Stopped 2024-02-05T04:58:29.943 DEBUG:tasks.ceph.osd.14:waiting for process to exit 2024-02-05T04:58:29.943 INFO:teuthology.orchestra.run:waiting for 300 2024-02-05T04:58:29.943 INFO:tasks.ceph.osd.14:Stopped 2024-02-05T04:58:29.943 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-02-05T04:58:29.943 INFO:teuthology.orchestra.run:waiting for 300 2024-02-05T04:58:29.943 INFO:tasks.ceph.osd.3:Stopped 2024-02-05T04:58:29.943 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-02-05T04:58:29.943 INFO:teuthology.orchestra.run:waiting for 300 2024-02-05T04:58:29.944 INFO:tasks.ceph.osd.7:Stopped 2024-02-05T04:58:29.944 DEBUG:tasks.ceph.osd.11:waiting for process to exit 2024-02-05T04:58:29.944 INFO:teuthology.orchestra.run:waiting for 300 2024-02-05T04:58:29.944 INFO:tasks.ceph.osd.11:Stopped 2024-02-05T04:58:29.944 DEBUG:tasks.ceph.osd.15:waiting for process to exit 2024-02-05T04:58:29.944 INFO:teuthology.orchestra.run:waiting for 300 2024-02-05T04:58:29.945 INFO:tasks.ceph.osd.15:Stopped 2024-02-05T04:58:29.945 INFO:teuthology.misc:Shutting down mgr daemons... 2024-02-05T04:58:29.945 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-02-05T04:58:29.945 INFO:teuthology.orchestra.run:waiting for 300 2024-02-05T04:58:29.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T04:58:29.945 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/thrashosds.py", line 215, in task cluster_manager.wait_for_all_osds_up() File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/ceph_manager.py", line 2808, in wait_for_all_osds_up while not self.are_all_osds_up(): File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/ceph_manager.py", line 2798, in are_all_osds_up x = self.get_osd_dump() File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/ceph_manager.py", line 2561, in get_osd_dump return self.get_osd_dump_json()['osds'] File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/ceph_manager.py", line 2553, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/ceph_manager.py", line 1627, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/ceph_manager.py", line 1618, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi062 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/ceph_manager.py", line 2774, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/ceph_manager.py", line 2606, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/ceph_manager.py", line 2372, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/ceph_manager.py", line 1627, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/ceph_manager.py", line 1618, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi062 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi062 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y' 2024-02-05T04:58:29.946 INFO:tasks.ceph.mgr.y:Stopped 2024-02-05T04:58:29.946 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-02-05T04:58:29.946 INFO:teuthology.orchestra.run:waiting for 300 2024-02-05T04:58:29.946 INFO:tasks.ceph.mgr.x:Stopped 2024-02-05T04:58:29.946 INFO:teuthology.misc:Shutting down mon daemons... 2024-02-05T04:58:29.946 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-02-05T04:58:29.946 INFO:teuthology.orchestra.run:waiting for 300 2024-02-05T04:58:29.946 INFO:tasks.ceph.mon.a:Stopped 2024-02-05T04:58:29.947 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-02-05T04:58:29.947 INFO:teuthology.orchestra.run:waiting for 300 2024-02-05T04:58:29.947 INFO:tasks.ceph.mon.b:Stopped 2024-02-05T04:58:29.947 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-02-05T04:58:29.947 INFO:teuthology.orchestra.run:waiting for 300 2024-02-05T04:58:29.947 INFO:tasks.ceph.mon.c:Stopped 2024-02-05T04:58:29.947 INFO:tasks.ceph:Checking cluster log for badness... 2024-02-05T04:58:29.948 DEBUG:teuthology.orchestra.run.smithi062:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'slow request' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | head -n 1 2024-02-05T04:58:29.988 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi062.front.sepia.ceph.com 2024-02-05T04:58:29.989 DEBUG:teuthology.orchestra.run.smithi062:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-02-05T04:58:30.102 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi062.front.sepia.ceph.com 2024-02-05T04:58:30.103 DEBUG:teuthology.orchestra.run.smithi062:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-02-05T04:58:30.151 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-8 on ubuntu@smithi062.front.sepia.ceph.com 2024-02-05T04:58:30.152 DEBUG:teuthology.orchestra.run.smithi062:> sync && sudo umount -f /var/lib/ceph/osd/ceph-8 2024-02-05T04:58:30.319 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-12 on ubuntu@smithi062.front.sepia.ceph.com 2024-02-05T04:58:30.319 DEBUG:teuthology.orchestra.run.smithi062:> sync && sudo umount -f /var/lib/ceph/osd/ceph-12 2024-02-05T04:58:30.379 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi091.front.sepia.ceph.com 2024-02-05T04:58:30.379 DEBUG:teuthology.orchestra.run.smithi091:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-02-05T04:58:30.438 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi091.front.sepia.ceph.com 2024-02-05T04:58:30.438 DEBUG:teuthology.orchestra.run.smithi091:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-02-05T04:58:30.485 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-9 on ubuntu@smithi091.front.sepia.ceph.com 2024-02-05T04:58:30.486 DEBUG:teuthology.orchestra.run.smithi091:> sync && sudo umount -f /var/lib/ceph/osd/ceph-9 2024-02-05T04:58:30.519 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-13 on ubuntu@smithi091.front.sepia.ceph.com 2024-02-05T04:58:30.519 DEBUG:teuthology.orchestra.run.smithi091:> sync && sudo umount -f /var/lib/ceph/osd/ceph-13 2024-02-05T04:58:30.636 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi150.front.sepia.ceph.com 2024-02-05T04:58:30.637 DEBUG:teuthology.orchestra.run.smithi150:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-02-05T04:58:30.703 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi150.front.sepia.ceph.com 2024-02-05T04:58:30.703 DEBUG:teuthology.orchestra.run.smithi150:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-02-05T04:58:30.770 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-10 on ubuntu@smithi150.front.sepia.ceph.com 2024-02-05T04:58:30.770 DEBUG:teuthology.orchestra.run.smithi150:> sync && sudo umount -f /var/lib/ceph/osd/ceph-10 2024-02-05T04:58:30.835 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-14 on ubuntu@smithi150.front.sepia.ceph.com 2024-02-05T04:58:30.835 DEBUG:teuthology.orchestra.run.smithi150:> sync && sudo umount -f /var/lib/ceph/osd/ceph-14 2024-02-05T04:58:30.893 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi196.front.sepia.ceph.com 2024-02-05T04:58:30.893 DEBUG:teuthology.orchestra.run.smithi196:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-02-05T04:58:31.000 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi196.front.sepia.ceph.com 2024-02-05T04:58:31.000 DEBUG:teuthology.orchestra.run.smithi196:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-02-05T04:58:31.056 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-11 on ubuntu@smithi196.front.sepia.ceph.com 2024-02-05T04:58:31.056 DEBUG:teuthology.orchestra.run.smithi196:> sync && sudo umount -f /var/lib/ceph/osd/ceph-11 2024-02-05T04:58:31.125 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-15 on ubuntu@smithi196.front.sepia.ceph.com 2024-02-05T04:58:31.125 DEBUG:teuthology.orchestra.run.smithi196:> sync && sudo umount -f /var/lib/ceph/osd/ceph-15 2024-02-05T04:58:31.180 INFO:tasks.ceph:Archiving mon data... 2024-02-05T04:58:31.182 DEBUG:teuthology.misc:Transferring archived files from smithi062:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/nmordech-2024-01-31_07:15:30-rados-wip-nitzan-max-pglog-size-distro-default-smithi/7539667/data/mon.a.tgz 2024-02-05T04:58:31.182 DEBUG:teuthology.orchestra.run.smithi062:> mktemp 2024-02-05T04:58:31.201 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/tmp.kfGfCenhZE 2024-02-05T04:58:31.201 DEBUG:teuthology.orchestra.run.smithi062:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.kfGfCenhZE 2024-02-05T04:58:31.363 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 0666 /tmp/tmp.kfGfCenhZE 2024-02-05T04:58:31.450 DEBUG:teuthology.orchestra.remote:smithi062:/tmp/tmp.kfGfCenhZE is 335KB 2024-02-05T04:58:31.548 DEBUG:teuthology.orchestra.run.smithi062:> rm -fr /tmp/tmp.kfGfCenhZE 2024-02-05T04:58:31.562 DEBUG:teuthology.misc:Transferring archived files from smithi091:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/nmordech-2024-01-31_07:15:30-rados-wip-nitzan-max-pglog-size-distro-default-smithi/7539667/data/mon.b.tgz 2024-02-05T04:58:31.563 DEBUG:teuthology.orchestra.run.smithi091:> mktemp 2024-02-05T04:58:31.581 INFO:teuthology.orchestra.run.smithi091.stdout:/tmp/tmp.rN6Nd7EiwU 2024-02-05T04:58:31.581 DEBUG:teuthology.orchestra.run.smithi091:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.rN6Nd7EiwU 2024-02-05T04:58:31.760 DEBUG:teuthology.orchestra.run.smithi091:> sudo chmod 0666 /tmp/tmp.rN6Nd7EiwU 2024-02-05T04:58:31.846 DEBUG:teuthology.orchestra.remote:smithi091:/tmp/tmp.rN6Nd7EiwU is 474KB 2024-02-05T04:58:31.943 DEBUG:teuthology.orchestra.run.smithi091:> rm -fr /tmp/tmp.rN6Nd7EiwU 2024-02-05T04:58:31.957 DEBUG:teuthology.misc:Transferring archived files from smithi150:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/nmordech-2024-01-31_07:15:30-rados-wip-nitzan-max-pglog-size-distro-default-smithi/7539667/data/mon.c.tgz 2024-02-05T04:58:31.957 DEBUG:teuthology.orchestra.run.smithi150:> mktemp 2024-02-05T04:58:31.975 INFO:teuthology.orchestra.run.smithi150.stdout:/tmp/tmp.hX9qUg6H46 2024-02-05T04:58:31.975 DEBUG:teuthology.orchestra.run.smithi150:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.hX9qUg6H46 2024-02-05T04:58:32.153 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 0666 /tmp/tmp.hX9qUg6H46 2024-02-05T04:58:32.238 DEBUG:teuthology.orchestra.remote:smithi150:/tmp/tmp.hX9qUg6H46 is 474KB 2024-02-05T04:58:32.336 DEBUG:teuthology.orchestra.run.smithi150:> rm -fr /tmp/tmp.hX9qUg6H46 2024-02-05T04:58:32.351 INFO:tasks.ceph:Cleaning ceph cluster... 2024-02-05T04:58:32.351 DEBUG:teuthology.orchestra.run.smithi062:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-02-05T04:58:32.355 DEBUG:teuthology.orchestra.run.smithi091:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-02-05T04:58:32.357 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-02-05T04:58:32.392 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-02-05T04:58:32.441 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-02-05T04:58:32.456 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-02-05T04:58:32.472 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-02-05T04:58:32.489 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-02-05T04:58:32.489 INFO:tasks.ceph:Archiving crash dumps... 2024-02-05T04:58:32.491 DEBUG:teuthology.misc:Transferring archived files from smithi062:/var/lib/ceph/crash to /home/teuthworker/archive/nmordech-2024-01-31_07:15:30-rados-wip-nitzan-max-pglog-size-distro-default-smithi/7539667/remote/smithi062/crash 2024-02-05T04:58:32.492 DEBUG:teuthology.orchestra.run.smithi062:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-02-05T04:58:32.523 DEBUG:teuthology.misc:Transferring archived files from smithi091:/var/lib/ceph/crash to /home/teuthworker/archive/nmordech-2024-01-31_07:15:30-rados-wip-nitzan-max-pglog-size-distro-default-smithi/7539667/remote/smithi091/crash 2024-02-05T04:58:32.524 DEBUG:teuthology.orchestra.run.smithi091:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-02-05T04:58:32.556 DEBUG:teuthology.misc:Transferring archived files from smithi150:/var/lib/ceph/crash to /home/teuthworker/archive/nmordech-2024-01-31_07:15:30-rados-wip-nitzan-max-pglog-size-distro-default-smithi/7539667/remote/smithi150/crash 2024-02-05T04:58:32.557 DEBUG:teuthology.orchestra.run.smithi150:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-02-05T04:58:32.583 DEBUG:teuthology.misc:Transferring archived files from smithi196:/var/lib/ceph/crash to /home/teuthworker/archive/nmordech-2024-01-31_07:15:30-rados-wip-nitzan-max-pglog-size-distro-default-smithi/7539667/remote/smithi196/crash 2024-02-05T04:58:32.584 DEBUG:teuthology.orchestra.run.smithi196:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-02-05T04:58:32.627 INFO:tasks.ceph:Compressing logs... 2024-02-05T04:58:32.627 DEBUG:teuthology.orchestra.run.smithi062:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-05T04:58:32.630 DEBUG:teuthology.orchestra.run.smithi091:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-05T04:58:32.636 DEBUG:teuthology.orchestra.run.smithi150:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-05T04:58:32.638 DEBUG:teuthology.orchestra.run.smithi196:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-05T04:58:32.656 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-02-05T04:58:32.656 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-02-05T04:58:32.660 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-02-05T04:58:32.660 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph.audit.log: /var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.13.log 2024-02-05T04:58:32.660 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/objectstore_tool.40658.log 2024-02-05T04:58:32.661 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-osd.13.log: gzip -5 --verbose -- /var/log/ceph/objectstore_tool.41421.log 2024-02-05T04:58:32.661 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/objectstore_tool.40658.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-02-05T04:58:32.661 INFO:teuthology.orchestra.run.smithi091.stderr: 89.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-02-05T04:58:32.661 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/objectstore_tool.41421.log: gzip -5 --verbose -- /var/log/ceph/objectstore_tool.43127.log 2024-02-05T04:58:32.661 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-02-05T04:58:32.661 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/objectstore_tool.43127.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.9.log 2024-02-05T04:58:32.661 INFO:teuthology.orchestra.run.smithi091.stderr: 83.3% -- replaced with /var/log/ceph/ceph.log.gz/var/log/ceph/ceph-mon.b.log: 2024-02-05T04:58:32.661 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/objectstore_tool.42364.log 2024-02-05T04:58:32.671 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-02-05T04:58:32.671 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.10.log 2024-02-05T04:58:32.671 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-02-05T04:58:32.671 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.14.log 2024-02-05T04:58:32.671 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-02-05T04:58:32.671 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-02-05T04:58:32.671 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-osd.14.log: /var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-02-05T04:58:32.672 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-osd.2.log: 89.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-02-05T04:58:32.672 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-mon.c.log: 83.3% -- replaced with /var/log/ceph/ceph.log.gz 2024-02-05T04:58:32.689 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-osd.9.log: /var/log/ceph/objectstore_tool.42364.log: 93.9% -- replaced with /var/log/ceph/objectstore_tool.41421.log.gz 2024-02-05T04:58:32.689 INFO:teuthology.orchestra.run.smithi091.stderr: 93.4% -- replaced with /var/log/ceph/objectstore_tool.43127.log.gz 2024-02-05T04:58:32.707 INFO:teuthology.orchestra.run.smithi091.stderr: 93.7% -- replaced with /var/log/ceph/objectstore_tool.42364.log.gz 2024-02-05T04:58:32.723 INFO:teuthology.orchestra.run.smithi196.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33944.log 2024-02-05T04:58:32.723 INFO:teuthology.orchestra.run.smithi196.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34104.log 2024-02-05T04:58:32.723 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-client.admin.33944.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34224.log 2024-02-05T04:58:32.723 INFO:teuthology.orchestra.run.smithi196.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33944.log.gz 2024-02-05T04:58:32.724 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-client.admin.34104.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.15.log 2024-02-05T04:58:32.724 INFO:teuthology.orchestra.run.smithi196.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34104.log.gz 2024-02-05T04:58:32.724 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-client.admin.34224.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33824.log 2024-02-05T04:58:32.724 INFO:teuthology.orchestra.run.smithi196.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34224.log.gz 2024-02-05T04:58:32.724 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-osd.15.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34024.log 2024-02-05T04:58:32.724 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-client.admin.33824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33984.log 2024-02-05T04:58:32.724 INFO:teuthology.orchestra.run.smithi196.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33824.log.gz 2024-02-05T04:58:32.724 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-client.admin.34024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33864.log 2024-02-05T04:58:32.724 INFO:teuthology.orchestra.run.smithi196.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34024.log.gz 2024-02-05T04:58:32.724 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-client.admin.33984.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.39362.log 2024-02-05T04:58:32.724 INFO:teuthology.orchestra.run.smithi196.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33984.log.gz 2024-02-05T04:58:32.724 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-client.admin.33864.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33904.log 2024-02-05T04:58:32.724 INFO:teuthology.orchestra.run.smithi196.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33864.log.gz 2024-02-05T04:58:32.725 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-client.0.39362.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.11.log 2024-02-05T04:58:32.725 INFO:teuthology.orchestra.run.smithi196.stderr: 74.9% -- replaced with /var/log/ceph/ceph-client.0.39362.log.gz 2024-02-05T04:58:32.725 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-client.admin.33904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34271.log 2024-02-05T04:58:32.725 INFO:teuthology.orchestra.run.smithi196.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33904.log.gz 2024-02-05T04:58:32.725 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-02-05T04:58:32.725 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-client.admin.34271.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-02-05T04:58:32.725 INFO:teuthology.orchestra.run.smithi196.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34271.log.gz 2024-02-05T04:58:32.725 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34315.log 2024-02-05T04:58:32.725 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34355.log 2024-02-05T04:58:32.725 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-client.admin.34315.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34144.log 2024-02-05T04:58:32.725 INFO:teuthology.orchestra.run.smithi196.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34315.log.gz 2024-02-05T04:58:32.725 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-client.admin.34355.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-02-05T04:58:32.725 INFO:teuthology.orchestra.run.smithi196.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34355.log.gz 2024-02-05T04:58:32.725 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-client.admin.34144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34064.log 2024-02-05T04:58:32.725 INFO:teuthology.orchestra.run.smithi196.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34144.log.gz 2024-02-05T04:58:32.726 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34184.log 2024-02-05T04:58:32.726 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-client.admin.34064.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34064.log.gz 2024-02-05T04:58:32.726 INFO:teuthology.orchestra.run.smithi196.stderr: --verbose -- /var/log/ceph/ceph-client.admin.34514.log 2024-02-05T04:58:32.726 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-client.admin.34184.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34184.log.gz 2024-02-05T04:58:32.726 INFO:teuthology.orchestra.run.smithi196.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34395.log 2024-02-05T04:58:32.726 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-client.admin.34514.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34514.log.gz 2024-02-05T04:58:32.726 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-client.admin.34395.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34395.log.gz 2024-02-05T04:58:32.743 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40305.log 2024-02-05T04:58:32.743 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48581.log 2024-02-05T04:58:32.744 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47122.log 2024-02-05T04:58:32.744 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.40305.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40305.log.gz 2024-02-05T04:58:32.744 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.48581.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39705.log 2024-02-05T04:58:32.744 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48581.log.gz 2024-02-05T04:58:32.744 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.47122.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47677.log 2024-02-05T04:58:32.744 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47122.log.gz 2024-02-05T04:58:32.744 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.39705.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39705.log.gz 2024-02-05T04:58:32.744 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-02-05T04:58:32.744 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.47677.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43191.log 2024-02-05T04:58:32.744 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47677.log.gz 2024-02-05T04:58:32.744 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42615.log 2024-02-05T04:58:32.744 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.43191.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39768.log 2024-02-05T04:58:32.744 INFO:teuthology.orchestra.run.smithi062.stderr: 24.3% -- replaced with /var/log/ceph/ceph-client.admin.43191.log.gz 2024-02-05T04:58:32.744 INFO:teuthology.orchestra.run.smithi062.stderr:gzip/var/log/ceph/ceph-client.admin.42615.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.43607.log 2024-02-05T04:58:32.745 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42615.log.gz 2024-02-05T04:58:32.745 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.39768.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43431.log 2024-02-05T04:58:32.745 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39768.log.gz 2024-02-05T04:58:32.745 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.43607.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43607.log.gz 2024-02-05T04:58:32.745 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40959.log 2024-02-05T04:58:32.745 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.43431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44310.log 2024-02-05T04:58:32.745 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43431.log.gz 2024-02-05T04:58:32.745 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.40959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34868.log 2024-02-05T04:58:32.745 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40959.log.gz 2024-02-05T04:58:32.745 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.44310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40311.log 2024-02-05T04:58:32.745 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44310.log.gz 2024-02-05T04:58:32.745 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.34868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47081.log 2024-02-05T04:58:32.745 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34868.log.gz 2024-02-05T04:58:32.745 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.40311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49186.log 2024-02-05T04:58:32.746 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40311.log.gz 2024-02-05T04:58:32.746 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.47081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40913.log 2024-02-05T04:58:32.746 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47081.log.gz 2024-02-05T04:58:32.746 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.49186.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45376.log 2024-02-05T04:58:32.746 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49186.log.gz 2024-02-05T04:58:32.746 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.40913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47918.log 2024-02-05T04:58:32.746 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40913.log.gz 2024-02-05T04:58:32.746 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.45376.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45449.log 2024-02-05T04:58:32.746 INFO:teuthology.orchestra.run.smithi062.stderr: 24.6% -- replaced with /var/log/ceph/ceph-client.admin.45376.log.gz 2024-02-05T04:58:32.746 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.47918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39827.log 2024-02-05T04:58:32.746 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47918.log.gz 2024-02-05T04:58:32.746 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.45449.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40745.log 2024-02-05T04:58:32.746 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45449.log.gz 2024-02-05T04:58:32.746 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.39827.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47297.log 2024-02-05T04:58:32.747 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39827.log.gz 2024-02-05T04:58:32.747 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.40745.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47858.log 2024-02-05T04:58:32.747 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40745.log.gz 2024-02-05T04:58:32.747 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.47297.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41375.log 2024-02-05T04:58:32.747 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47297.log.gz 2024-02-05T04:58:32.747 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.47858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46695.log 2024-02-05T04:58:32.747 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47858.log.gz 2024-02-05T04:58:32.747 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.41375.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41375.log.gz 2024-02-05T04:58:32.747 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49102.log 2024-02-05T04:58:32.747 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.46695.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43737.log 2024-02-05T04:58:32.747 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46695.log.gz 2024-02-05T04:58:32.747 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.49102.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44351.log 2024-02-05T04:58:32.747 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49102.log.gz 2024-02-05T04:58:32.747 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.43737.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.12.log 2024-02-05T04:58:32.747 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43737.log.gz 2024-02-05T04:58:32.748 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.44351.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39969.log 2024-02-05T04:58:32.748 INFO:teuthology.orchestra.run.smithi062.stderr: 24.2% -- replaced with /var/log/ceph/ceph-client.admin.44351.log.gz 2024-02-05T04:58:32.748 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-osd.12.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40961.log 2024-02-05T04:58:32.748 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.39969.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41649.log 2024-02-05T04:58:32.748 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39969.log.gz 2024-02-05T04:58:32.748 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.40961.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40961.log.gz 2024-02-05T04:58:32.748 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48683.log 2024-02-05T04:58:32.748 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.41649.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42301.log 2024-02-05T04:58:32.748 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41649.log.gz 2024-02-05T04:58:32.748 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.48683.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48683.log.gz 2024-02-05T04:58:32.748 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34720.log 2024-02-05T04:58:32.748 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.42301.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46003.log 2024-02-05T04:58:32.748 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42301.log.gz 2024-02-05T04:58:32.748 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.34720.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34720.log.gz 2024-02-05T04:58:32.749 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46840.log 2024-02-05T04:58:32.749 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.46003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41405.log 2024-02-05T04:58:32.749 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46003.log.gz 2024-02-05T04:58:32.749 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.46840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48766.log 2024-02-05T04:58:32.749 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46840.log.gz 2024-02-05T04:58:32.749 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.41405.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.41405.log.gz 2024-02-05T04:58:32.749 INFO:teuthology.orchestra.run.smithi062.stderr: /var/log/ceph/ceph-client.admin.43347.log 2024-02-05T04:58:32.749 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.48766.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47017.log 2024-02-05T04:58:32.749 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48766.log.gz 2024-02-05T04:58:32.749 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.43347.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43347.log.gz 2024-02-05T04:58:32.749 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49060.log 2024-02-05T04:58:32.749 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.47017.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43608.log 2024-02-05T04:58:32.749 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47017.log.gz 2024-02-05T04:58:32.749 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.49060.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49060.log.gz -5 2024-02-05T04:58:32.749 INFO:teuthology.orchestra.run.smithi062.stderr: --verbose -- /var/log/ceph/ceph-client.admin.47960.log 2024-02-05T04:58:32.749 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.43608.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42792.log 2024-02-05T04:58:32.750 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43608.log.gz 2024-02-05T04:58:32.750 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.47960.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47960.log.gz 2024-02-05T04:58:32.750 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39927.log 2024-02-05T04:58:32.750 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.42792.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43050.log 2024-02-05T04:58:32.750 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42792.log.gz 2024-02-05T04:58:32.750 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.39927.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39927.log.gz 2024-02-05T04:58:32.750 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44268.log 2024-02-05T04:58:32.750 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.43050.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47502.log 2024-02-05T04:58:32.750 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43050.log.gz 2024-02-05T04:58:32.750 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.44268.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44268.log.gz 2024-02-05T04:58:32.750 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42018.log 2024-02-05T04:58:32.750 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.47502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43670.log 2024-02-05T04:58:32.750 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47502.log.gz 2024-02-05T04:58:32.750 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.42018.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46359.log 2024-02-05T04:58:32.751 INFO:teuthology.orchestra.run.smithi062.stderr: 23.8% -- replaced with /var/log/ceph/ceph-client.admin.42018.log.gz 2024-02-05T04:58:32.751 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.43670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40904.log 2024-02-05T04:58:32.751 INFO:teuthology.orchestra.run.smithi062.stderr: 24.5% -- replaced with /var/log/ceph/ceph-client.admin.43670.log.gz 2024-02-05T04:58:32.751 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.46359.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46359.log.gz 2024-02-05T04:58:32.751 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42614.log 2024-02-05T04:58:32.751 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-02-05T04:58:32.751 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.40904.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40904.log.gz 2024-02-05T04:58:32.751 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.42614.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44069.log 2024-02-05T04:58:32.751 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42614.log.gz 2024-02-05T04:58:32.751 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42019.log 2024-02-05T04:58:32.751 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.44069.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46902.log 2024-02-05T04:58:32.751 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44069.log.gz 2024-02-05T04:58:32.751 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.42019.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42019.log.gz 2024-02-05T04:58:32.751 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42021.log 2024-02-05T04:58:32.751 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41691.log 2024-02-05T04:58:32.752 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.46902.log: 24.8% -- replaced with /var/log/ceph/ceph-client.admin.46902.log.gz 2024-02-05T04:58:32.752 INFO:teuthology.orchestra.run.smithi062.stderr: 89.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-02-05T04:58:32.752 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.42021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41501.log 2024-02-05T04:58:32.752 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42021.log.gz 2024-02-05T04:58:32.752 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.41691.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28590.log 2024-02-05T04:58:32.752 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41691.log.gz 2024-02-05T04:58:32.752 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.41501.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41501.log.gz 2024-02-05T04:58:32.752 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40267.log 2024-02-05T04:58:32.752 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph.tmp-client.admin.28590.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41607.log 2024-02-05T04:58:32.752 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.28590.log.gz 2024-02-05T04:58:32.752 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.40267.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40267.log.gz 2024-02-05T04:58:32.752 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41879.log 2024-02-05T04:58:32.752 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.41607.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-02-05T04:58:32.752 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41607.log.gz 2024-02-05T04:58:32.752 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.41879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46242.log 2024-02-05T04:58:32.753 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41879.log.gz 2024-02-05T04:58:32.753 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46673.log 2024-02-05T04:58:32.753 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48124.log 2024-02-05T04:58:32.753 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.46242.log: 24.7% -- replaced with /var/log/ceph/ceph-client.admin.46242.log.gz 2024-02-05T04:58:32.753 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.46673.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46673.log.gz 2024-02-05T04:58:32.753 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34452.log 2024-02-05T04:58:32.753 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41481.log 2024-02-05T04:58:32.753 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.48124.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48124.log.gz 2024-02-05T04:58:32.753 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.34452.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34452.log.gz 2024-02-05T04:58:32.753 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43297.log 2024-02-05T04:58:32.753 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45648.log 2024-02-05T04:58:32.753 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.41481.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41481.log.gz 2024-02-05T04:58:32.753 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.43297.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43297.log.gz 2024-02-05T04:58:32.753 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40985.log 2024-02-05T04:58:32.754 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49216.log 2024-02-05T04:58:32.754 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.45648.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45648.log.gz 2024-02-05T04:58:32.754 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.40985.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42866.log 2024-02-05T04:58:32.754 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40985.log.gz 2024-02-05T04:58:32.754 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.8.log 2024-02-05T04:58:32.754 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.49216.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49216.log.gz 2024-02-05T04:58:32.754 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.42866.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42866.log.gz 2024-02-05T04:58:32.754 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45530.log 2024-02-05T04:58:32.754 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40302.log 2024-02-05T04:58:32.754 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-osd.8.log: /var/log/ceph/ceph-client.admin.45530.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40280.log 2024-02-05T04:58:32.754 INFO:teuthology.orchestra.run.smithi062.stderr: 24.4% -- replaced with /var/log/ceph/ceph-client.admin.45530.log.gz 2024-02-05T04:58:32.754 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.40302.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45491.log 2024-02-05T04:58:32.754 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40302.log.gz 2024-02-05T04:58:32.754 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42466.log 2024-02-05T04:58:32.755 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.40280.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40280.log.gz 2024-02-05T04:58:32.755 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.45491.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46421.log 2024-02-05T04:58:32.755 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45491.log.gz 2024-02-05T04:58:32.755 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.42466.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42751.log 2024-02-05T04:58:32.755 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42466.log.gz 2024-02-05T04:58:32.755 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48621.log 2024-02-05T04:58:32.755 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.46421.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46421.log.gz 2024-02-05T04:58:32.755 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.42751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40976.log 2024-02-05T04:58:32.755 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42751.log.gz 2024-02-05T04:58:32.755 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.48621.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44397.log 2024-02-05T04:58:32.755 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48621.log.gz 2024-02-05T04:58:32.755 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.40976.log: -- /var/log/ceph/ceph-client.admin.39885.log 2024-02-05T04:58:32.755 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40976.log.gz 2024-02-05T04:58:32.755 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.44397.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41565.log 2024-02-05T04:58:32.755 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44397.log.gz 2024-02-05T04:58:32.756 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.39885.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44686.log 2024-02-05T04:58:32.756 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39885.log.gz 2024-02-05T04:58:32.756 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46044.log 2024-02-05T04:58:32.756 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.41565.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41565.log.gz 2024-02-05T04:58:32.756 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.44686.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46221.log 2024-02-05T04:58:32.756 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44686.log.gz 2024-02-05T04:58:32.756 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.46044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48002.log 2024-02-05T04:58:32.756 INFO:teuthology.orchestra.run.smithi062.stderr: 24.4% -- replaced with /var/log/ceph/ceph-client.admin.46044.log.gz 2024-02-05T04:58:32.756 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43367.log 2024-02-05T04:58:32.756 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.46221.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46221.log.gz 2024-02-05T04:58:32.756 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.48002.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41355.log 2024-02-05T04:58:32.756 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48002.log.gz 2024-02-05T04:58:32.756 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.43367.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43367.log.gz 2024-02-05T04:58:32.756 INFO:teuthology.orchestra.run.smithi062.stderr: --verbose -- /var/log/ceph/ceph-client.admin.45020.log 2024-02-05T04:58:32.756 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.41355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40277.log 2024-02-05T04:58:32.757 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41355.log.gz 2024-02-05T04:58:32.757 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.45020.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40308.log 2024-02-05T04:58:32.757 INFO:teuthology.orchestra.run.smithi062.stderr: 24.8% -- replaced with /var/log/ceph/ceph-client.admin.45020.log.gz 2024-02-05T04:58:32.757 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.40277.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40277.log.gz 2024-02-05T04:58:32.757 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42709.log 2024-02-05T04:58:32.757 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40729.log 2024-02-05T04:58:32.757 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.40308.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40308.log.gz 2024-02-05T04:58:32.757 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.42709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46160.log 0.0% 2024-02-05T04:58:32.757 INFO:teuthology.orchestra.run.smithi062.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42709.log.gz 2024-02-05T04:58:32.757 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.40729.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40729.log.gz 2024-02-05T04:58:32.757 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40824.log 2024-02-05T04:58:32.757 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40298.log 2024-02-05T04:58:32.757 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.46160.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46160.log.gz 2024-02-05T04:58:32.757 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.40824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42022.log 2024-02-05T04:58:32.758 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40824.log.gz 2024-02-05T04:58:32.758 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.40298.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40298.log.gz 2024-02-05T04:58:32.758 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48510.log 2024-02-05T04:58:32.758 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45592.log 2024-02-05T04:58:32.758 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.42022.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42022.log.gz 2024-02-05T04:58:32.758 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.48510.log: gzip -5 25.7% -- replaced with /var/log/ceph/ceph-client.admin.48510.log.gz 2024-02-05T04:58:32.758 INFO:teuthology.orchestra.run.smithi062.stderr: --verbose -- /var/log/ceph/ceph-client.admin.40941.log 2024-02-05T04:58:32.758 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.45592.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44864.log 2024-02-05T04:58:32.758 INFO:teuthology.orchestra.run.smithi062.stderr: -- replaced with /var/log/ceph/ceph-client.admin.45592.log.gz 2024-02-05T04:58:32.758 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.40941.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40941.log.gz 2024-02-05T04:58:32.758 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40276.log 2024-02-05T04:58:32.758 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49144.log 2024-02-05T04:58:32.758 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.44864.log: 24.8%/var/log/ceph/ceph-client.admin.40276.log: -- replaced with /var/log/ceph/ceph-client.admin.44864.log.gz 2024-02-05T04:58:32.758 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40276.log.gz 2024-02-05T04:58:32.758 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44150.log 2024-02-05T04:58:32.759 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.49144.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49144.log.gz 2024-02-05T04:58:32.759 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47801.log 2024-02-05T04:58:32.759 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40779.log 2024-02-05T04:58:32.759 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.44150.log: 24.1% -- replaced with /var/log/ceph/ceph-client.admin.44150.log.gz 2024-02-05T04:58:32.759 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.47801.log: gzip -5 23.9% -- replaced with /var/log/ceph/ceph-client.admin.47801.log.gz 2024-02-05T04:58:32.759 INFO:teuthology.orchestra.run.smithi062.stderr: --verbose -- /var/log/ceph/ceph-client.admin.40879.log 2024-02-05T04:58:32.759 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.40779.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45689.log 2024-02-05T04:58:32.759 INFO:teuthology.orchestra.run.smithi062.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40779.log.gz 2024-02-05T04:58:32.759 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.40879.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40879.log.gz 2024-02-05T04:58:32.759 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-02-05T04:58:32.759 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43471.log 2024-02-05T04:58:32.759 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.45689.log: 24.8% -- replaced with /var/log/ceph/ceph-client.admin.45689.log.gz/var/log/ceph/ceph-mgr.y.log: 2024-02-05T04:58:32.759 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40965.log 2024-02-05T04:58:32.759 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.43471.log: gzip -5 24.4% -- replaced with /var/log/ceph/ceph-client.admin.43471.log.gz 2024-02-05T04:58:32.759 INFO:teuthology.orchestra.run.smithi062.stderr: --verbose -- /var/log/ceph/ceph-client.admin.42886.log 2024-02-05T04:58:32.760 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.40965.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40965.log.gz 2024-02-05T04:58:32.760 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46882.log 2024-02-05T04:58:32.760 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.42886.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42886.log.gz 2024-02-05T04:58:32.760 INFO:teuthology.orchestra.run.smithi062.stderr: --verbose -- /var/log/ceph/ceph-osd.0.log 2024-02-05T04:58:32.760 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.46882.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46882.log.gz 2024-02-05T04:58:32.760 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44089.log 2024-02-05T04:58:32.760 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44979.log 2024-02-05T04:58:32.760 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.44089.log: 28.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45094.log 2024-02-05T04:58:32.760 INFO:teuthology.orchestra.run.smithi062.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44089.log.gz 2024-02-05T04:58:32.760 INFO:teuthology.orchestra.run.smithi062.stderr: 92.2% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-02-05T04:58:32.760 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.44979.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44979.log.gz 2024-02-05T04:58:32.760 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47188.log 2024-02-05T04:58:32.760 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.45094.log: gzip 0.0% -5 --verbose -- /var/log/ceph/ceph-client.admin.40973.log 2024-02-05T04:58:32.760 INFO:teuthology.orchestra.run.smithi062.stderr: -- replaced with /var/log/ceph/ceph-client.admin.45094.log.gz 2024-02-05T04:58:32.761 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.47188.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47188.log.gz 2024-02-05T04:58:32.761 INFO:teuthology.orchestra.run.smithi062.stderr: --verbose -- /var/log/ceph/ceph-client.admin.47759.log 2024-02-05T04:58:32.761 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.40973.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40973.log.gz 2024-02-05T04:58:32.761 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41733.log 2024-02-05T04:58:32.761 INFO:teuthology.orchestra.run.smithi062.stderr:gzip/var/log/ceph/ceph-client.admin.47759.log: -5 --verbose -- 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47759.log.gz 2024-02-05T04:58:32.761 INFO:teuthology.orchestra.run.smithi062.stderr: /var/log/ceph/ceph-client.admin.48397.log 2024-02-05T04:58:32.761 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.41733.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41733.log.gz 2024-02-05T04:58:32.761 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-02-05T04:58:32.761 INFO:teuthology.orchestra.run.smithi062.stderr:gzip/var/log/ceph/ceph-client.admin.48397.log: -5 --verbose 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48397.log.gz 2024-02-05T04:58:32.761 INFO:teuthology.orchestra.run.smithi062.stderr: -- /var/log/ceph/ceph-client.admin.44937.log 2024-02-05T04:58:32.761 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40966.log 2024-02-05T04:58:32.761 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.44937.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44937.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47718.log 2024-02-05T04:58:32.761 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-02-05T04:58:32.761 INFO:teuthology.orchestra.run.smithi062.stderr: 83.3% -- replaced with /var/log/ceph/ceph.log.gz 2024-02-05T04:58:32.761 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.40966.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46997.log 2024-02-05T04:58:32.762 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40966.log.gz 2024-02-05T04:58:32.762 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.47718.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48468.log 2024-02-05T04:58:32.762 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47718.log.gz 2024-02-05T04:58:32.762 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.46997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42507.log 2024-02-05T04:58:32.762 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46997.log.gz 2024-02-05T04:58:32.762 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46522.log 2024-02-05T04:58:32.762 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.48468.log: /var/log/ceph/ceph-client.admin.42507.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48468.log.gz 2024-02-05T04:58:32.762 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40279.log 2024-02-05T04:58:32.762 INFO:teuthology.orchestra.run.smithi062.stderr: 24.1% -- replaced with /var/log/ceph/ceph-client.admin.42507.log.gz 2024-02-05T04:58:32.762 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46116.log 2024-02-05T04:58:32.762 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.46522.log: 24.9% -- replaced with /var/log/ceph/ceph-client.admin.46522.log.gz 2024-02-05T04:58:32.762 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43927.log 2024-02-05T04:58:32.762 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.40279.log: /var/log/ceph/ceph-client.admin.46116.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44467.log 2024-02-05T04:58:32.762 INFO:teuthology.orchestra.run.smithi062.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40279.log.gz 2024-02-05T04:58:32.762 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46116.log.gz 2024-02-05T04:58:32.763 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.43927.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48044.log 2024-02-05T04:58:32.763 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43927.log.gz 2024-02-05T04:58:32.763 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.44467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44589.log 2024-02-05T04:58:32.763 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44467.log.gz 2024-02-05T04:58:32.763 INFO:teuthology.orchestra.run.smithi062.stderr:gzip/var/log/ceph/ceph-client.admin.48044.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.42300.log 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48044.log.gz 2024-02-05T04:58:32.763 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-02-05T04:58:32.763 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.44589.log: 24.4% -- replaced with /var/log/ceph/ceph-client.admin.44589.log.gz 2024-02-05T04:58:32.763 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40304.log 2024-02-05T04:58:32.763 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.42300.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42300.log.gz 2024-02-05T04:58:32.763 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44226.log 2024-02-05T04:58:32.763 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.40304.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40304.log.gz 2024-02-05T04:58:32.763 INFO:teuthology.orchestra.run.smithi062.stderr: --verbose -- /var/log/ceph/ceph-client.admin.41817.log 2024-02-05T04:58:32.763 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.44226.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44226.log.gz 2024-02-05T04:58:32.764 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42572.log 2024-02-05T04:58:32.764 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.41817.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41817.log.gz 2024-02-05T04:58:32.764 INFO:teuthology.orchestra.run.smithi062.stderr: --verbose -- /var/log/ceph/ceph-client.admin.40303.log 2024-02-05T04:58:32.764 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.42572.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42572.log.gz 2024-02-05T04:58:32.764 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40294.log 2024-02-05T04:58:32.764 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.40303.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40293.log 2024-02-05T04:58:32.764 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40303.log.gz 2024-02-05T04:58:32.764 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.40294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41775.log 2024-02-05T04:58:32.764 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40294.log.gz 2024-02-05T04:58:32.764 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43546.log 2024-02-05T04:58:32.764 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.40293.log: /var/log/ceph/ceph-client.admin.41775.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40290.log 2024-02-05T04:58:32.764 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40293.log.gz 2024-02-05T04:58:32.764 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41775.log.gz 2024-02-05T04:58:32.765 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.43546.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43546.log.gz 2024-02-05T04:58:32.765 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48868.log 2024-02-05T04:58:32.765 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46222.log/var/log/ceph/ceph-client.admin.40290.log: 2024-02-05T04:58:32.765 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0%/var/log/ceph/ceph-client.admin.48868.log: -- replaced with /var/log/ceph/ceph-client.admin.40290.log.gz 2024-02-05T04:58:32.765 INFO:teuthology.orchestra.run.smithi062.stderr: 24.5%gzip -5 -- replaced with /var/log/ceph/ceph-client.admin.48868.log.gz --verbose 2024-02-05T04:58:32.765 INFO:teuthology.orchestra.run.smithi062.stderr: -- /var/log/ceph/ceph-client.admin.47582.log 2024-02-05T04:58:32.765 INFO:teuthology.orchestra.run.smithi062.stderr:gzip/var/log/ceph/ceph-client.admin.46222.log: -5 --verbose 0.0% -- /var/log/ceph/ceph-client.admin.40288.log 2024-02-05T04:58:32.765 INFO:teuthology.orchestra.run.smithi062.stderr: -- replaced with /var/log/ceph/ceph-client.admin.46222.log.gz 2024-02-05T04:58:32.765 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.47582.log: gzip -5 24.4% -- replaced with /var/log/ceph/ceph-client.admin.47582.log.gz 2024-02-05T04:58:32.765 INFO:teuthology.orchestra.run.smithi062.stderr: --verbose -- /var/log/ceph/ceph-client.admin.40979.log 2024-02-05T04:58:32.765 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.40288.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40288.log.gz 2024-02-05T04:58:32.765 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47298.log 2024-02-05T04:58:32.765 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.40979.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40979.log.gz 2024-02-05T04:58:32.766 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.47298.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47298.log.gz 2024-02-05T04:58:32.766 INFO:teuthology.orchestra.run.smithi150.stderr: 91.0% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-02-05T04:58:32.766 INFO:teuthology.orchestra.run.smithi196.stderr: 92.2% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-02-05T04:58:32.767 INFO:teuthology.orchestra.run.smithi091.stderr: 94.5% -- replaced with /var/log/ceph/objectstore_tool.40658.log.gz 2024-02-05T04:58:32.767 INFO:teuthology.orchestra.run.smithi091.stderr: 91.1% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-02-05T04:58:32.852 INFO:teuthology.orchestra.run.smithi062.stderr: 91.1% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-02-05T04:58:32.980 INFO:teuthology.orchestra.run.smithi091.stderr: 91.8% -- replaced with /var/log/ceph/ceph-osd.9.log.gz 2024-02-05T04:58:33.127 INFO:teuthology.orchestra.run.smithi150.stderr: 91.8% -- replaced with /var/log/ceph/ceph-osd.14.log.gz 2024-02-05T04:58:33.152 INFO:teuthology.orchestra.run.smithi196.stderr: 92.0% -- replaced with /var/log/ceph/ceph-osd.15.log.gz 2024-02-05T04:58:33.158 INFO:teuthology.orchestra.run.smithi062.stderr: 91.8% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-02-05T04:58:33.277 INFO:teuthology.orchestra.run.smithi091.stderr: 92.0% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-02-05T04:58:33.493 INFO:teuthology.orchestra.run.smithi062.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.8.log.gz 2024-02-05T04:58:33.496 INFO:teuthology.orchestra.run.smithi091.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-02-05T04:58:33.508 INFO:teuthology.orchestra.run.smithi150.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.10.log.gz 2024-02-05T04:58:33.532 INFO:teuthology.orchestra.run.smithi196.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-02-05T04:58:33.556 INFO:teuthology.orchestra.run.smithi062.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-02-05T04:58:33.559 INFO:teuthology.orchestra.run.smithi150.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-02-05T04:58:33.595 INFO:teuthology.orchestra.run.smithi196.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-02-05T04:58:33.619 INFO:teuthology.orchestra.run.smithi150.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-02-05T04:58:33.621 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-02-05T04:58:33.621 INFO:teuthology.orchestra.run.smithi150.stderr:real 0m0.970s 2024-02-05T04:58:33.621 INFO:teuthology.orchestra.run.smithi150.stderr:user 0m3.148s 2024-02-05T04:58:33.621 INFO:teuthology.orchestra.run.smithi150.stderr:sys 0m0.137s 2024-02-05T04:58:33.669 INFO:teuthology.orchestra.run.smithi062.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.12.log.gz 2024-02-05T04:58:33.670 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-02-05T04:58:33.670 INFO:teuthology.orchestra.run.smithi062.stderr:real 0m1.026s 2024-02-05T04:58:33.670 INFO:teuthology.orchestra.run.smithi062.stderr:user 0m3.301s 2024-02-05T04:58:33.671 INFO:teuthology.orchestra.run.smithi062.stderr:sys 0m0.195s 2024-02-05T04:58:33.671 INFO:teuthology.orchestra.run.smithi196.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.11.log.gz 2024-02-05T04:58:33.672 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-02-05T04:58:33.673 INFO:teuthology.orchestra.run.smithi196.stderr:real 0m0.983s 2024-02-05T04:58:33.673 INFO:teuthology.orchestra.run.smithi196.stderr:user 0m3.073s 2024-02-05T04:58:33.673 INFO:teuthology.orchestra.run.smithi196.stderr:sys 0m0.140s 2024-02-05T04:58:33.680 INFO:teuthology.orchestra.run.smithi091.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.13.log.gz 2024-02-05T04:58:33.682 INFO:teuthology.orchestra.run.smithi091.stderr: 2024-02-05T04:58:33.682 INFO:teuthology.orchestra.run.smithi091.stderr:real 0m1.038s 2024-02-05T04:58:33.682 INFO:teuthology.orchestra.run.smithi091.stderr:user 0m2.985s 2024-02-05T04:58:33.682 INFO:teuthology.orchestra.run.smithi091.stderr:sys 0m0.115s 2024-02-05T04:58:33.683 INFO:tasks.ceph:Archiving logs... 2024-02-05T04:58:33.683 DEBUG:teuthology.misc:Transferring archived files from smithi062:/var/log/ceph to /home/teuthworker/archive/nmordech-2024-01-31_07:15:30-rados-wip-nitzan-max-pglog-size-distro-default-smithi/7539667/remote/smithi062/log 2024-02-05T04:58:33.684 DEBUG:teuthology.orchestra.run.smithi062:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-05T04:58:34.333 DEBUG:teuthology.misc:Transferring archived files from smithi091:/var/log/ceph to /home/teuthworker/archive/nmordech-2024-01-31_07:15:30-rados-wip-nitzan-max-pglog-size-distro-default-smithi/7539667/remote/smithi091/log 2024-02-05T04:58:34.333 DEBUG:teuthology.orchestra.run.smithi091:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-05T04:58:34.866 DEBUG:teuthology.misc:Transferring archived files from smithi150:/var/log/ceph to /home/teuthworker/archive/nmordech-2024-01-31_07:15:30-rados-wip-nitzan-max-pglog-size-distro-default-smithi/7539667/remote/smithi150/log 2024-02-05T04:58:34.866 DEBUG:teuthology.orchestra.run.smithi150:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-05T04:58:35.424 DEBUG:teuthology.misc:Transferring archived files from smithi196:/var/log/ceph to /home/teuthworker/archive/nmordech-2024-01-31_07:15:30-rados-wip-nitzan-max-pglog-size-distro-default-smithi/7539667/remote/smithi196/log 2024-02-05T04:58:35.425 DEBUG:teuthology.orchestra.run.smithi196:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-05T04:58:35.953 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/thrashosds.py", line 215, in task cluster_manager.wait_for_all_osds_up() File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/ceph_manager.py", line 2808, in wait_for_all_osds_up while not self.are_all_osds_up(): File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/ceph_manager.py", line 2798, in are_all_osds_up x = self.get_osd_dump() File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/ceph_manager.py", line 2561, in get_osd_dump return self.get_osd_dump_json()['osds'] File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/ceph_manager.py", line 2553, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/ceph_manager.py", line 1627, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/ceph_manager.py", line 1618, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi062 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/ceph_manager.py", line 2774, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/ceph_manager.py", line 2606, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/ceph_manager.py", line 2372, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/ceph_manager.py", line 1627, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/ceph_manager.py", line 1618, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi062 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2024-02-05T04:58:35.954 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-02-05T04:58:35.964 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/thrashosds.py", line 215, in task cluster_manager.wait_for_all_osds_up() File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/ceph_manager.py", line 2808, in wait_for_all_osds_up while not self.are_all_osds_up(): File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/ceph_manager.py", line 2798, in are_all_osds_up x = self.get_osd_dump() File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/ceph_manager.py", line 2561, in get_osd_dump return self.get_osd_dump_json()['osds'] File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/ceph_manager.py", line 2553, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/ceph_manager.py", line 1627, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/ceph_manager.py", line 1618, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi062 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' 2024-02-05T04:58:35.964 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-02-05T04:58:35.964 DEBUG:teuthology.orchestra.run.smithi062:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-05T04:58:35.968 DEBUG:teuthology.orchestra.run.smithi091:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-05T04:58:35.970 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-05T04:58:35.973 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-05T04:58:36.103 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-02-05T04:58:36.113 INFO:teuthology.task.clock:Checking final clock skew... 2024-02-05T04:58:36.113 DEBUG:teuthology.orchestra.run.smithi062:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-05T04:58:36.115 DEBUG:teuthology.orchestra.run.smithi091:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-05T04:58:36.118 DEBUG:teuthology.orchestra.run.smithi150:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-05T04:58:36.120 DEBUG:teuthology.orchestra.run.smithi196:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-05T04:58:36.130 INFO:teuthology.orchestra.run.smithi062.stderr:bash: line 1: ntpq: command not found 2024-02-05T04:58:36.132 INFO:teuthology.orchestra.run.smithi091.stderr:bash: line 1: ntpq: command not found 2024-02-05T04:58:36.135 INFO:teuthology.orchestra.run.smithi062.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-05T04:58:36.135 INFO:teuthology.orchestra.run.smithi062.stdout:=============================================================================== 2024-02-05T04:58:36.135 INFO:teuthology.orchestra.run.smithi062.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 7 +1111us[+1111us] +/- 122ms 2024-02-05T04:58:36.135 INFO:teuthology.orchestra.run.smithi062.stdout:^* hv02.front.sepia.ceph.com 3 6 377 9 -1407us[-1378us] +/- 65ms 2024-02-05T04:58:36.135 INFO:teuthology.orchestra.run.smithi062.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 9 +781us[ +781us] +/- 63ms 2024-02-05T04:58:36.135 INFO:teuthology.orchestra.run.smithi062.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-02-05T04:58:36.136 INFO:teuthology.orchestra.run.smithi196.stderr:bash: line 1: ntpq: command not found 2024-02-05T04:58:36.136 INFO:teuthology.orchestra.run.smithi150.stderr:bash: line 1: ntpq: command not found 2024-02-05T04:58:36.136 INFO:teuthology.orchestra.run.smithi091.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-05T04:58:36.137 INFO:teuthology.orchestra.run.smithi091.stdout:=============================================================================== 2024-02-05T04:58:36.137 INFO:teuthology.orchestra.run.smithi091.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 9 +1155us[+1133us] +/- 122ms 2024-02-05T04:58:36.137 INFO:teuthology.orchestra.run.smithi091.stdout:^* hv02.front.sepia.ceph.com 3 6 377 6 -1334us[-1356us] +/- 65ms 2024-02-05T04:58:36.137 INFO:teuthology.orchestra.run.smithi091.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 7 +821us[ +798us] +/- 63ms 2024-02-05T04:58:36.137 INFO:teuthology.orchestra.run.smithi091.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-02-05T04:58:36.139 INFO:teuthology.orchestra.run.smithi150.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-05T04:58:36.139 INFO:teuthology.orchestra.run.smithi150.stdout:=============================================================================== 2024-02-05T04:58:36.139 INFO:teuthology.orchestra.run.smithi150.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 7 +1156us[+1167us] +/- 122ms 2024-02-05T04:58:36.139 INFO:teuthology.orchestra.run.smithi150.stdout:^* hv02.front.sepia.ceph.com 3 6 377 6 -1367us[-1355us] +/- 65ms 2024-02-05T04:58:36.139 INFO:teuthology.orchestra.run.smithi150.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 9 +828us[ +839us] +/- 63ms 2024-02-05T04:58:36.139 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-02-05T04:58:36.141 INFO:teuthology.orchestra.run.smithi196.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-05T04:58:36.141 INFO:teuthology.orchestra.run.smithi196.stdout:=============================================================================== 2024-02-05T04:58:36.141 INFO:teuthology.orchestra.run.smithi196.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 8 +1130us[+1157us] +/- 122ms 2024-02-05T04:58:36.141 INFO:teuthology.orchestra.run.smithi196.stdout:^* hv02.front.sepia.ceph.com 3 6 377 7 -1399us[-1372us] +/- 65ms 2024-02-05T04:58:36.141 INFO:teuthology.orchestra.run.smithi196.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 7 +786us[ +786us] +/- 63ms 2024-02-05T04:58:36.141 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-02-05T04:58:36.141 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-02-05T04:58:36.204 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-02-05T04:58:36.204 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-02-05T04:58:36.213 DEBUG:teuthology.orchestra.run.smithi062:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-05T04:58:36.291 DEBUG:teuthology.orchestra.run.smithi091:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-05T04:58:36.363 DEBUG:teuthology.orchestra.run.smithi150:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-05T04:58:36.436 DEBUG:teuthology.orchestra.run.smithi196:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-05T04:58:36.509 DEBUG:teuthology.orchestra.run.smithi062:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-05T04:58:36.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T04:58:36.544 DEBUG:teuthology.orchestra.run.smithi091:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-05T04:58:36.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T04:58:36.576 DEBUG:teuthology.orchestra.run.smithi150:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-05T04:58:36.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T04:58:36.609 DEBUG:teuthology.orchestra.run.smithi196:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-05T04:58:36.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T04:58:36.642 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-02-05T04:58:36.652 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-02-05T04:58:36.684 INFO:teuthology.task.internal:Duration was 850.680428 seconds 2024-02-05T04:58:36.684 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-02-05T04:58:36.694 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-02-05T04:58:36.694 DEBUG:teuthology.orchestra.run.smithi062:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-05T04:58:36.696 DEBUG:teuthology.orchestra.run.smithi091:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-05T04:58:36.698 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-05T04:58:36.701 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-05T04:58:36.743 INFO:teuthology.orchestra.run.smithi091.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-05T04:58:36.743 INFO:teuthology.orchestra.run.smithi062.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-05T04:58:36.744 INFO:teuthology.orchestra.run.smithi150.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-05T04:58:36.747 INFO:teuthology.orchestra.run.smithi196.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-05T04:58:37.198 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-02-05T04:58:37.198 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi062.front.sepia.ceph.com 2024-02-05T04:58:37.198 DEBUG:teuthology.orchestra.run.smithi062:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-05T04:58:37.222 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi091.front.sepia.ceph.com 2024-02-05T04:58:37.222 DEBUG:teuthology.orchestra.run.smithi091:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-05T04:58:37.248 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi150.front.sepia.ceph.com 2024-02-05T04:58:37.248 DEBUG:teuthology.orchestra.run.smithi150:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-05T04:58:37.278 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi196.front.sepia.ceph.com 2024-02-05T04:58:37.278 DEBUG:teuthology.orchestra.run.smithi196:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-05T04:58:37.307 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-02-05T04:58:37.308 DEBUG:teuthology.orchestra.run.smithi062:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-05T04:58:37.311 DEBUG:teuthology.orchestra.run.smithi091:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-05T04:58:37.313 DEBUG:teuthology.orchestra.run.smithi150:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-05T04:58:37.321 DEBUG:teuthology.orchestra.run.smithi196:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-05T04:58:37.378 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-02-05T04:58:37.378 DEBUG:teuthology.orchestra.run.smithi062:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-05T04:58:37.382 DEBUG:teuthology.orchestra.run.smithi091:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-05T04:58:37.385 DEBUG:teuthology.orchestra.run.smithi150:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-05T04:58:37.394 DEBUG:teuthology.orchestra.run.smithi196:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-05T04:58:37.520 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-02-05T04:58:37.531 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-02-05T04:58:37.531 DEBUG:teuthology.orchestra.run.smithi062:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-05T04:58:37.560 DEBUG:teuthology.orchestra.run.smithi091:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-05T04:58:37.587 DEBUG:teuthology.orchestra.run.smithi150:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-05T04:58:37.615 DEBUG:teuthology.orchestra.run.smithi196:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-05T04:58:37.643 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-02-05T04:58:37.654 DEBUG:teuthology.orchestra.run.smithi062:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-05T04:58:37.656 DEBUG:teuthology.orchestra.run.smithi091:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-05T04:58:37.659 DEBUG:teuthology.orchestra.run.smithi150:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-05T04:58:37.661 DEBUG:teuthology.orchestra.run.smithi196:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-05T04:58:37.680 INFO:teuthology.orchestra.run.smithi062.stdout:kernel.core_pattern = core 2024-02-05T04:58:37.682 INFO:teuthology.orchestra.run.smithi091.stdout:kernel.core_pattern = core 2024-02-05T04:58:37.684 INFO:teuthology.orchestra.run.smithi150.stdout:kernel.core_pattern = core 2024-02-05T04:58:37.708 INFO:teuthology.orchestra.run.smithi196.stdout:kernel.core_pattern = core 2024-02-05T04:58:37.735 DEBUG:teuthology.orchestra.run.smithi062:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-05T04:58:37.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T04:58:37.763 DEBUG:teuthology.orchestra.run.smithi091:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-05T04:58:37.777 WARNING:teuthology.task.internal:Found coredumps on ubuntu@smithi091.front.sepia.ceph.com, flagging run as failed 2024-02-05T04:58:37.777 DEBUG:teuthology.orchestra.run.smithi150:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-05T04:58:37.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T04:58:37.792 DEBUG:teuthology.orchestra.run.smithi196:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-05T04:58:37.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T04:58:37.806 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-02-05T04:58:37.816 INFO:teuthology.task.internal:Transferring archived files... 2024-02-05T04:58:37.817 DEBUG:teuthology.misc:Transferring archived files from smithi062:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/nmordech-2024-01-31_07:15:30-rados-wip-nitzan-max-pglog-size-distro-default-smithi/7539667/remote/smithi062 2024-02-05T04:58:37.817 DEBUG:teuthology.orchestra.run.smithi062:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-05T04:58:37.854 DEBUG:teuthology.misc:Transferring archived files from smithi091:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/nmordech-2024-01-31_07:15:30-rados-wip-nitzan-max-pglog-size-distro-default-smithi/7539667/remote/smithi091 2024-02-05T04:58:37.855 DEBUG:teuthology.orchestra.run.smithi091:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-05T04:58:45.353 INFO:teuthology.task.internal:Transferring binaries for coredumps... 2024-02-05T04:58:45.421 INFO:teuthology.task.internal: core looks like: /home/teuthworker/archive/nmordech-2024-01-31_07:15:30-rados-wip-nitzan-max-pglog-size-distro-default-smithi/7539667/remote/smithi091/coredump/1707108860.43127.core.gz: gzip compressed data, was "1707108860.43127.core", last modified: Mon Feb 5 04:58:37 2024, max compression, original size modulo 2^32 234889216 2024-02-05T04:58:45.422 INFO:teuthology.task.internal:core is compressed, try accessing gzip file ... 2024-02-05T04:58:46.371 INFO:teuthology.task.internal: core looks like: /tmp/tmp5kqbgnt_: ELF 64-bit LSB core file, x86-64, version 1 (SYSV), SVR4-style, from 'ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-13 --j', real uid: 0, effective uid: 0, real gid: 0, effective gid: 0, execfn: '/bin/ceph-objectstore-tool', platform: 'x86_64' 2024-02-05T04:58:46.398 INFO:teuthology.task.internal: dump_program: ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-13 --j 2024-02-05T04:58:46.399 DEBUG:teuthology.orchestra.run.smithi091:> which 'ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-13 --j' 2024-02-05T04:58:46.417 INFO:teuthology.orchestra.run.smithi091.stderr:/usr/bin/which: unrecognized option '--err-to-stderr' 2024-02-05T04:58:46.417 INFO:teuthology.orchestra.run.smithi091.stderr:/usr/bin/which: unrecognized option '--data-path' 2024-02-05T04:58:46.417 INFO:teuthology.orchestra.run.smithi091.stderr:/usr/bin/which: unrecognized option '--j' 2024-02-05T04:58:46.418 INFO:teuthology.orchestra.run.smithi091.stdout:/usr/bin/ceph-objectstore-tool 2024-02-05T04:58:46.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T04:58:46.419 INFO:teuthology.orchestra.run.smithi091.stderr:/usr/bin/which: no ceph-13 in (/var/lib/ceph/osd) 2024-02-05T04:58:46.419 ERROR:teuthology.run_tasks:Manager failed: internal.archive Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/internal/__init__.py", line 392, in archive yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/thrashosds.py", line 215, in task cluster_manager.wait_for_all_osds_up() File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/ceph_manager.py", line 2808, in wait_for_all_osds_up while not self.are_all_osds_up(): File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/ceph_manager.py", line 2798, in are_all_osds_up x = self.get_osd_dump() File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/ceph_manager.py", line 2561, in get_osd_dump return self.get_osd_dump_json()['osds'] File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/ceph_manager.py", line 2553, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/ceph_manager.py", line 1627, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/ceph_manager.py", line 1618, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi062 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/internal/__init__.py", line 420, in archive fetch_binaries_for_coredumps(path, rem) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/internal/__init__.py", line 345, in fetch_binaries_for_coredumps remote_path = remote.sh(['which', dump_program]).rstrip() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi091 with status 1: "which 'ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-13 --j'" 2024-02-05T04:58:46.420 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-02-05T04:58:46.432 INFO:teuthology.task.internal:Not uploading archives. 2024-02-05T04:58:46.432 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-02-05T04:58:46.443 INFO:teuthology.task.internal:Tidying up after the test... 2024-02-05T04:58:46.443 DEBUG:teuthology.orchestra.run.smithi062:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-05T04:58:46.446 DEBUG:teuthology.orchestra.run.smithi091:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-05T04:58:46.461 INFO:teuthology.orchestra.run.smithi062.stdout: 262166 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Feb 5 04:58 /home/ubuntu/cephtest 2024-02-05T04:58:46.461 INFO:teuthology.orchestra.run.smithi062.stdout: 262167 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Feb 5 04:58 /home/ubuntu/cephtest/archive 2024-02-05T04:58:46.461 INFO:teuthology.orchestra.run.smithi062.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 5 04:58 /home/ubuntu/cephtest/archive/syslog 2024-02-05T04:58:46.461 INFO:teuthology.orchestra.run.smithi062.stdout: 262223 4 -rw-rw-rw- 1 ubuntu ubuntu 29 Feb 5 04:44 /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2024-02-05T04:58:46.461 INFO:teuthology.orchestra.run.smithi062.stdout: 265294 4 -rw-rw-rw- 1 ubuntu ubuntu 29 Feb 5 04:44 /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2024-02-05T04:58:46.461 INFO:teuthology.orchestra.run.smithi062.stdout: 265293 184 -rw-r--r-- 1 ubuntu ubuntu 184815 Feb 5 04:58 /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-05T04:58:46.461 INFO:teuthology.orchestra.run.smithi062.stdout: 402633 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 5 04:58 /home/ubuntu/cephtest/archive/audit 2024-02-05T04:58:46.462 INFO:teuthology.orchestra.run.smithi062.stdout: 402912 136 -rw------- 1 ubuntu ubuntu 138211 Feb 5 04:58 /home/ubuntu/cephtest/archive/audit/audit.log.gz 2024-02-05T04:58:46.462 DEBUG:teuthology.orchestra.run.smithi150:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-05T04:58:46.463 INFO:teuthology.orchestra.run.smithi062.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-02-05T04:58:46.464 DEBUG:teuthology.orchestra.run.smithi196:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-05T04:58:46.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T04:58:46.469 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/thrashosds.py", line 215, in task cluster_manager.wait_for_all_osds_up() File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/ceph_manager.py", line 2808, in wait_for_all_osds_up while not self.are_all_osds_up(): File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/ceph_manager.py", line 2798, in are_all_osds_up x = self.get_osd_dump() File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/ceph_manager.py", line 2561, in get_osd_dump return self.get_osd_dump_json()['osds'] File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/ceph_manager.py", line 2553, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/ceph_manager.py", line 1627, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/ceph_manager.py", line 1618, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi062 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi062 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-02-05T04:58:46.470 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-02-05T04:58:46.481 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-02-05T04:58:46.494 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi062 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' 2024-02-05T04:58:46.494 INFO:teuthology.nuke:Checking targets against current locks 2024-02-05T04:58:46.499 INFO:teuthology.orchestra.run.smithi150.stdout: 262166 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Feb 5 04:58 /home/ubuntu/cephtest 2024-02-05T04:58:46.499 INFO:teuthology.orchestra.run.smithi150.stdout: 262167 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Feb 5 04:58 /home/ubuntu/cephtest/archive 2024-02-05T04:58:46.499 INFO:teuthology.orchestra.run.smithi150.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 5 04:58 /home/ubuntu/cephtest/archive/syslog 2024-02-05T04:58:46.499 INFO:teuthology.orchestra.run.smithi150.stdout: 262223 4 -rw-rw-rw- 1 ubuntu ubuntu 29 Feb 5 04:44 /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2024-02-05T04:58:46.499 INFO:teuthology.orchestra.run.smithi150.stdout: 265294 4 -rw-rw-rw- 1 ubuntu ubuntu 29 Feb 5 04:44 /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2024-02-05T04:58:46.499 INFO:teuthology.orchestra.run.smithi150.stdout: 265293 180 -rw-r--r-- 1 ubuntu ubuntu 181226 Feb 5 04:58 /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-05T04:58:46.499 INFO:teuthology.orchestra.run.smithi150.stdout: 402680 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 5 04:58 /home/ubuntu/cephtest/archive/audit 2024-02-05T04:58:46.499 INFO:teuthology.orchestra.run.smithi150.stdout: 402682 124 -rw------- 1 ubuntu ubuntu 123779 Feb 5 04:58 /home/ubuntu/cephtest/archive/audit/audit.log.gz 2024-02-05T04:58:46.500 INFO:teuthology.orchestra.run.smithi091.stdout: 262166 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Feb 5 04:58 /home/ubuntu/cephtest 2024-02-05T04:58:46.500 INFO:teuthology.orchestra.run.smithi091.stdout: 262167 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Feb 5 04:58 /home/ubuntu/cephtest/archive 2024-02-05T04:58:46.500 INFO:teuthology.orchestra.run.smithi091.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 5 04:58 /home/ubuntu/cephtest/archive/syslog 2024-02-05T04:58:46.500 INFO:teuthology.orchestra.run.smithi091.stdout: 262223 4 -rw-rw-rw- 1 ubuntu ubuntu 29 Feb 5 04:44 /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2024-02-05T04:58:46.500 INFO:teuthology.orchestra.run.smithi091.stdout: 265294 4 -rw-rw-rw- 1 ubuntu ubuntu 29 Feb 5 04:44 /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2024-02-05T04:58:46.500 INFO:teuthology.orchestra.run.smithi091.stdout: 265293 180 -rw-r--r-- 1 ubuntu ubuntu 181395 Feb 5 04:58 /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-05T04:58:46.500 INFO:teuthology.orchestra.run.smithi091.stdout: 402680 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 5 04:58 /home/ubuntu/cephtest/archive/audit 2024-02-05T04:58:46.500 INFO:teuthology.orchestra.run.smithi091.stdout: 402682 124 -rw------- 1 ubuntu ubuntu 123268 Feb 5 04:58 /home/ubuntu/cephtest/archive/audit/audit.log.gz 2024-02-05T04:58:46.500 INFO:teuthology.orchestra.run.smithi091.stdout: 262168 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 5 04:54 /home/ubuntu/cephtest/archive/coredump 2024-02-05T04:58:46.500 INFO:teuthology.orchestra.run.smithi091.stdout: 265388 20784 -rw------- 1 root root 234889216 Feb 5 04:54 /home/ubuntu/cephtest/archive/coredump/1707108860.43127.core 2024-02-05T04:58:46.501 INFO:teuthology.orchestra.run.smithi196.stdout: 262166 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Feb 5 04:58 /home/ubuntu/cephtest 2024-02-05T04:58:46.501 INFO:teuthology.orchestra.run.smithi196.stdout: 262167 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Feb 5 04:58 /home/ubuntu/cephtest/archive 2024-02-05T04:58:46.501 INFO:teuthology.orchestra.run.smithi196.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 5 04:58 /home/ubuntu/cephtest/archive/syslog 2024-02-05T04:58:46.501 INFO:teuthology.orchestra.run.smithi196.stdout: 262223 4 -rw-rw-rw- 1 ubuntu ubuntu 29 Feb 5 04:44 /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2024-02-05T04:58:46.501 INFO:teuthology.orchestra.run.smithi196.stdout: 265294 4 -rw-rw-rw- 1 ubuntu ubuntu 29 Feb 5 04:44 /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2024-02-05T04:58:46.501 INFO:teuthology.orchestra.run.smithi196.stdout: 265293 180 -rw-r--r-- 1 ubuntu ubuntu 180736 Feb 5 04:58 /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-05T04:58:46.501 INFO:teuthology.orchestra.run.smithi196.stdout: 402680 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 5 04:58 /home/ubuntu/cephtest/archive/audit 2024-02-05T04:58:46.501 INFO:teuthology.orchestra.run.smithi196.stdout: 402959 120 -rw------- 1 ubuntu ubuntu 121095 Feb 5 04:58 /home/ubuntu/cephtest/archive/audit/audit.log.gz 2024-02-05T04:58:46.503 INFO:teuthology.orchestra.run.smithi150.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-02-05T04:58:46.503 INFO:teuthology.orchestra.run.smithi091.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-02-05T04:58:46.503 INFO:teuthology.orchestra.run.smithi196.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-02-05T04:58:46.525 DEBUG:teuthology.nuke:shortname: smithi062 2024-02-05T04:58:46.525 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-05T04:58:46.545 DEBUG:teuthology.nuke:shortname: smithi091 2024-02-05T04:58:46.545 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-05T04:58:46.553 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi062.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/nmordech-2024-01-31_07:15:30-rados-wip-nitzan-max-pglog-size-distro-default-smithi/7539667', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-05 04:34:49.111665', 'locked_by': 'scheduled_nmordech@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLrCXX9FGrvpb2YY//9WUXmiMbUrI6W2c1RxGH7NTG8iNneJEkvdg41pWixfxXaNkgRiaumItanuQ+E6EfZtsuQ='} 2024-02-05T04:58:46.568 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi091.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/nmordech-2024-01-31_07:15:30-rados-wip-nitzan-max-pglog-size-distro-default-smithi/7539667', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-05 04:34:49.106744', 'locked_by': 'scheduled_nmordech@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKACiH2/ZV+aC13a4lw3tv6KdmbuvnXSsmKFzugeTLDBQHhxcPZCUScqyjU02uLKZMsAiDzBiR/NEjXhmt//V1Q='} 2024-02-05T04:58:46.572 DEBUG:teuthology.nuke:shortname: smithi150 2024-02-05T04:58:46.572 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-05T04:58:46.594 DEBUG:teuthology.nuke:shortname: smithi196 2024-02-05T04:58:46.595 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-05T04:58:46.604 INFO:teuthology.orchestra.console.smithi062:Power off 2024-02-05T04:58:46.604 DEBUG:teuthology.orchestra.console.smithi062:pexpect command: ipmitool -H smithi062.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-05T04:58:46.618 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi150.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/nmordech-2024-01-31_07:15:30-rados-wip-nitzan-max-pglog-size-distro-default-smithi/7539667', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-05 04:34:49.113890', 'locked_by': 'scheduled_nmordech@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDkiF69tNRQpFmCKo4Q2yleP7Jc1Pgc2YXFX0TcAZ3/oJIAxt4oc0AJkq+b2psJRwqrnXI9Z/uxNWwNPz7ZO4Fo='} 2024-02-05T04:58:46.627 DEBUG:teuthology.orchestra.console.smithi062:power off output: Chassis Power Control: Down/Off 2024-02-05T04:58:46.628 DEBUG:teuthology.orchestra.console.smithi062:pexpect command: ipmitool -H smithi062.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-05T04:58:46.639 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi196.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/nmordech-2024-01-31_07:15:30-rados-wip-nitzan-max-pglog-size-distro-default-smithi/7539667', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-05 04:34:49.112812', 'locked_by': 'scheduled_nmordech@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOOMfC5VqynyKF0c2ZrwRiL5Aavhag9Mt0amEBga9yehMArhs/XBirD15Vfk+fhSedG28dlp3w/xd36KPqL4jPg='} 2024-02-05T04:58:46.650 INFO:teuthology.orchestra.console.smithi091:Power off 2024-02-05T04:58:46.651 DEBUG:teuthology.orchestra.console.smithi091:pexpect command: ipmitool -H smithi091.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-05T04:58:46.665 DEBUG:teuthology.orchestra.console.smithi062:check power output: Chassis Power is on 2024-02-05T04:58:46.675 DEBUG:teuthology.orchestra.console.smithi091:power off output: Chassis Power Control: Down/Off 2024-02-05T04:58:46.675 DEBUG:teuthology.orchestra.console.smithi091:pexpect command: ipmitool -H smithi091.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-05T04:58:46.692 INFO:teuthology.orchestra.console.smithi150:Power off 2024-02-05T04:58:46.692 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-05T04:58:46.702 DEBUG:teuthology.orchestra.console.smithi091:check power output: Chassis Power is on 2024-02-05T04:58:46.720 DEBUG:teuthology.orchestra.console.smithi150:power off output: Chassis Power Control: Down/Off 2024-02-05T04:58:46.720 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-05T04:58:46.744 INFO:teuthology.orchestra.console.smithi196:Power off 2024-02-05T04:58:46.745 DEBUG:teuthology.orchestra.console.smithi196:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-05T04:58:46.754 DEBUG:teuthology.orchestra.console.smithi150:check power output: Chassis Power is on 2024-02-05T04:58:46.767 DEBUG:teuthology.orchestra.console.smithi196:power off output: Chassis Power Control: Down/Off 2024-02-05T04:58:46.768 DEBUG:teuthology.orchestra.console.smithi196:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-05T04:58:46.790 DEBUG:teuthology.orchestra.console.smithi196:check power output: Chassis Power is on 2024-02-05T04:58:50.666 DEBUG:teuthology.orchestra.console.smithi062:pexpect command: ipmitool -H smithi062.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-05T04:58:50.703 DEBUG:teuthology.orchestra.console.smithi091:pexpect command: ipmitool -H smithi091.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-05T04:58:50.756 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-05T04:58:50.777 DEBUG:teuthology.orchestra.console.smithi062:check power output: Chassis Power is on 2024-02-05T04:58:50.791 DEBUG:teuthology.orchestra.console.smithi196:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-05T04:58:50.814 DEBUG:teuthology.orchestra.console.smithi091:check power output: Chassis Power is on 2024-02-05T04:58:50.867 DEBUG:teuthology.orchestra.console.smithi150:check power output: Chassis Power is on 2024-02-05T04:58:50.902 DEBUG:teuthology.orchestra.console.smithi196:check power output: Chassis Power is on 2024-02-05T04:58:54.780 DEBUG:teuthology.orchestra.console.smithi062:pexpect command: ipmitool -H smithi062.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-05T04:58:54.815 DEBUG:teuthology.orchestra.console.smithi091:pexpect command: ipmitool -H smithi091.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-05T04:58:54.868 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-05T04:58:54.891 DEBUG:teuthology.orchestra.console.smithi062:check power output: Chassis Power is off 2024-02-05T04:58:54.902 DEBUG:teuthology.orchestra.console.smithi196:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-05T04:58:54.927 DEBUG:teuthology.orchestra.console.smithi091:check power output: Chassis Power is on 2024-02-05T04:58:54.979 DEBUG:teuthology.orchestra.console.smithi150:check power output: Chassis Power is on 2024-02-05T04:58:54.991 INFO:teuthology.orchestra.console.smithi062:Power off completed 2024-02-05T04:58:55.012 DEBUG:teuthology.orchestra.console.smithi196:check power output: Chassis Power is off 2024-02-05T04:58:55.113 INFO:teuthology.orchestra.console.smithi196:Power off completed 2024-02-05T04:58:58.929 DEBUG:teuthology.orchestra.console.smithi091:pexpect command: ipmitool -H smithi091.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-05T04:58:58.980 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-05T04:58:59.041 DEBUG:teuthology.orchestra.console.smithi091:check power output: Chassis Power is off 2024-02-05T04:58:59.092 DEBUG:teuthology.orchestra.console.smithi150:check power output: Chassis Power is off 2024-02-05T04:58:59.142 INFO:teuthology.orchestra.console.smithi091:Power off completed 2024-02-05T04:58:59.193 INFO:teuthology.orchestra.console.smithi150:Power off completed 2024-02-05T04:58:59.298 INFO:teuthology.run:Summary data: description: rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} duration: 850.6804280281067 failure_reason: 'Command failed on smithi062 with status 124: ''sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json''' flavor: default owner: scheduled_nmordech@teuthology status: fail success: false 2024-02-05T04:58:59.299 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-05T04:58:59.383 INFO:teuthology.run:FAIL